Unicode libraries - Fun with Unicode (introspection, lookup, sets, guards, transforms...)

News Announcing
,
32

Lots of solid updates to several the Unicode libraries today. Overall they compile faster, run faster and are more conformant to their respective standards.

unicode 2.0.0

A major release of the base library.

  • ~10x faster lookups, an order of magnitude faster compilation: Category/script/property lookups now use binary search over compact range tables instead of huge generated guard clauses.

  • unicode_guards library is folded in. The separate unicode_guards package is no longer needed and will be retired, The guards (is_upper/1, is_lower/1, is_digit/1, is_whitespace/1, the quotation-mark guards and more) now ship with unicode.

  • Correct derived categories: :Assigned, :Graph, :Visible and :Printable are now computed from the current character database rather than stale static tables. :Printable now matches String.printable?/1 (it previously excluded most of the BMP, including Arabic, CJK and Hangul), and :Assigned picked up ~16k codepoints.

  • Unicode.CharacterName.to_codepoint/1 resolves a character name to its codepoint (loose matching), backed by a compact sorted blob.

  • Fixed UTF‑16/UTF‑32 validation in Unicode.replace_invalid/3, which previously crashed on any input.

unicode_set 1.7.0

A large correctness pass on UnicodeSet parsing and regex generation, closing many gaps against ICU/TR35. Tested against both standards suggest the implementation is now properly conforming.

Highlights:

  • the full set of escapes (\a–\v, \xH, \u{…}/\x{…} including astral, octal, \cX), single-quote quoting,
  • \N{name} resolution (via unicode 2.0),
  • correct handling of string members and string ranges when compiled to a regex,
  • negated sets containing string members,
  • De Morgan reduction for unions of complements ([[^a][^b]]),
  • the Is/In property prefixes,
  • and digit-bearing block names, and left-to-right set-operation precedence.

unicode_transform 1.1.0

The pure-Elixir CLDR transform engine now conforms to 99.99% of the official CLDR transform test data, up from ~81% — roughly twenty root-cause fixes across the parser, compiler, engine and resolver. The remaining gaps a real spec ambiguity issues in a few tests in 4 of the 290 transforms.

It also ships a strict CLDR conformance suite driven by ~297k vendored test cases — one test per transform — so the engine’s conformance is measured and guarded on every run.

5 Likes