Unicode libraries - Fun with Unicode (introspection, lookup, sets, guards, transforms...)

News Announcing
,
3

Added two helpful functions in version 0.2.0 that:

  • Generates compiled patterns for speedy String.split/3 and String.replace/3:
  • Generates list of code point ranges that can be used in nimble_parsec.

Generating compiled patterns for String matching

String.split/3 and String.replace/3 allow for patterns and compiled patterns to be used with compiled patterns being the more performant approach. Unicode Set supports the generation of patterns and compiled patterns:

iex> pattern = Unicode.Set.compile_pattern "[[:digit:]]"
{:ac, #Reference<0.2128975411.2135031811.52710>}
iex> list = String.split("abc1def2ghi3jkl", pattern)
["abc", "def", "ghi", "jkl"]

Generating NimbleParsec ranges

The parser generator nimble_parsec allows a list of codepoint ranges as parameters to several combinators. Unicode Set can generate such ranges:

iex> Unicode.Set.utf8_char("[[^abcd][mnb]]")
[{:not, 97}, {:not, 98}, {:not, 99}, {:not, 100}, 98, 109, 110]

This can be used as shown in the following example:

defmodule MyCombinators do
  import NimbleParsec

  @digit_list Unicode.Set.utf8_char("[[:digit:]]")
  def unicode_digit do
    utf8_char(@digit_list)
    |> label("a digit in any Unicode script")
  end
end
3 Likes