# \#unicode **URL:** https://forum.elixirforum.com/tag/unicode/24.md [Latest](https://forum.elixirforum.com/latest.md) · [Categories](https://forum.elixirforum.com/categories.md) · [Tags](https://forum.elixirforum.com/tags.md) --- ## [Unicode libraries - Fun with Unicode (introspection, lookup, sets, guards, transforms...)](https://forum.elixirforum.com/t/unicode-libraries-fun-with-unicode-introspection-lookup-sets-guards-transforms/27063)
**Author:** [@kip](https://forum.elixirforum.com/u/kip)\ **Replies:** 32\ **Last updated:** [September 20, 2026, 1:30am UTC](https://forum.elixirforum.com/t/unicode-libraries-fun-with-unicode-introspection-lookup-sets-guards-transforms/27063 "2026-09-20T01:30:42Z")
Following on from my CLDR lbraries I started work on Unicode transforms. But like everything related to CLDR there is a lot of yak-shaving and rabbit-hole travelling required. The net result is a bunch of new libraries … --- ## [Cldr Messages - Localised message formatting following the MF2 spec with opt-in NIF](https://forum.elixirforum.com/t/cldr-messages-localised-message-formatting-following-the-mf2-spec-with-opt-in-nif/74701)
**Author:** [@kip](https://forum.elixirforum.com/u/kip)\ **Replies:** 3\ **Last updated:** [March 18, 2026, 8:33am UTC](https://forum.elixirforum.com/t/cldr-messages-localised-message-formatting-following-the-mf2-spec-with-opt-in-nif/74701 "2026-03-18T08:33:19Z")
Capping of a hectic few days is the announcement of ex\_cldr\_messages version 2.0. This significant update adds support Unicode MessageFormat 2 (MF2) as well as the existing ICU Message Format for Elixir, integrated with… --- ## [Cldr Collation - language-aware string sorting and comparison with opt-in NIF](https://forum.elixirforum.com/t/cldr-collation-language-aware-string-sorting-and-comparison-with-opt-in-nif/74680)
**Author:** [@kip](https://forum.elixirforum.com/u/kip)\ **Replies:** 0\ **Last updated:** [March 16, 2026, 9:28pm UTC](https://forum.elixirforum.com/t/cldr-collation-language-aware-string-sorting-and-comparison-with-opt-in-nif/74680 "2026-03-16T21:28:32Z")
Announcing ex\_cldr\_collation version 1.0 ex\_cldr\_collation is an Elixir implementation of the Unicode Collation Algorithm (UCA) as extended by CLDR, providing language-aware string sorting and comparison. An opt-in NIF i… --- ## [Unicode Transform - transliterate text between scripts, un-accenting and case mapping](https://forum.elixirforum.com/t/unicode-transform-transliterate-text-between-scripts-un-accenting-and-case-mapping/74678)
**Author:** [@kip](https://forum.elixirforum.com/u/kip)\ **Replies:** 0\ **Last updated:** [March 16, 2026, 7:15pm UTC](https://forum.elixirforum.com/t/unicode-transform-transliterate-text-between-scripts-un-accenting-and-case-mapping/74678 "2026-03-16T19:15:33Z")
unicode\_transform has reached the 1.0 milestone after a complete rewrite delivering a library which transliterates text between scripts, applies normalization and case mappings, and executes arbitrary CLDR transform rule… --- ## [Fun with OTP 28's new \`:re\` module - which version of Unicode is used by OTP 27/28?](https://forum.elixirforum.com/t/fun-with-otp-28s-new-re-module-which-version-of-unicode-is-used-by-otp-27-28/70143)
**Author:** [@kip](https://forum.elixirforum.com/u/kip)\ **Replies:** 2\ **Last updated:** [March 26, 2025, 2:30pm UTC](https://forum.elixirforum.com/t/fun-with-otp-28s-new-re-module-which-version-of-unicode-is-used-by-otp-27-28/70143 "2025-03-26T14:30:58Z")
I’m working on making sure all the ex\_cldr\_\* and unicode\_\* libraries are ready for OTP 28. I know that one of the key changes is that OTP 28 has a new :re module (hooray!). What I can’t find out is what version of Unicod… --- ## [Converting hex chars to unicode equivalent](https://forum.elixirforum.com/t/converting-hex-chars-to-unicode-equivalent/63370)
**Author:** [@cblavier](https://forum.elixirforum.com/u/cblavier)\ **Replies:** 5\ **Last updated:** [May 7, 2024, 10:46am UTC](https://forum.elixirforum.com/t/converting-hex-chars-to-unicode-equivalent/63370 "2024-05-07T10:46:41Z")
Hey there :waving\_hand:, I fetch some HTML content with Req and extract plain text with Floki. The retrieved text contains hex characters such as "Bient\\xf4t" where I would expect "Bientôt" or the unicode equivalent "B… --- ## [Is there a way to escape escape characters i.e. turn \`"\\u2003"\` =\> \`"\\\\u2003"\`?](https://forum.elixirforum.com/t/is-there-a-way-to-escape-escape-characters-i-e-turn-u2003-u2003/59759)
**Author:** [@mruoss](https://forum.elixirforum.com/u/mruoss)\ **Replies:** 16\ **Last updated:** [November 21, 2023, 12:04pm UTC](https://forum.elixirforum.com/t/is-there-a-way-to-escape-escape-characters-i-e-turn-u2003-u2003/59759 "2023-11-21T12:04:45Z")
Hey, does anybody know if it is possible to escape escape characters like \\r or \\u2003? Given a string containing such characters, I’d like to turn them into an escaped form as follows: iex\> Magic.escape("\\r\\u2003") "\\\\… --- ## [How to remove invalid unicode from an API response: Jason.DecodeError unexpected sequence "\\\\udc51"](https://forum.elixirforum.com/t/how-to-remove-invalid-unicode-from-an-api-response-jason-decodeerror-unexpected-sequence-udc51/56504)
**Author:** [@rnice01](https://forum.elixirforum.com/u/rnice01)\ **Replies:** 5\ **Last updated:** [June 14, 2023, 8:38pm UTC](https://forum.elixirforum.com/t/how-to-remove-invalid-unicode-from-an-api-response-jason-decodeerror-unexpected-sequence-udc51/56504 "2023-06-14T20:38:04Z")
Hello! We are working with responses from an API that contain HTML from websites. Some of these websites contain unicode code points that Jason is unable to decode. We have a Python script that runs if this fails and ca… --- ## [Library to split string on unicode word boundaries](https://forum.elixirforum.com/t/library-to-split-string-on-unicode-word-boundaries/50421)
**Author:** [@aochagavia](https://forum.elixirforum.com/u/aochagavia)\ **Replies:** 2\ **Last updated:** [September 21, 2022, 4:35pm UTC](https://forum.elixirforum.com/t/library-to-split-string-on-unicode-word-boundaries/50421 "2022-09-21T16:35:09Z")
I am looking for a way to split utf8 strings into chunks that have some logical cohesion (e.g. avoid splitting in the middle of a word or, worse, in the middle of a grapheme cluster). The unicode standard annex 29 define… --- ## [Possible bug in String.split/1's handling of Unicode whitespace?](https://forum.elixirforum.com/t/possible-bug-in-string-split-1s-handling-of-unicode-whitespace/50437)
**Author:** [@dogweather](https://forum.elixirforum.com/u/dogweather)\ **Replies:** 5\ **Last updated:** [September 22, 2022, 5:44am UTC](https://forum.elixirforum.com/t/possible-bug-in-string-split-1s-handling-of-unicode-whitespace/50437 "2022-09-22T05:44:30Z")
According to the docs, String.split/1… Divides a string into substrings at each Unicode whitespace occurrence… (emphasis mine) But I found that it doesn’t do this for U+00A0 “NO-BREAK SPACE” (UTF-8 \<\<194, 160\>\>). Ma… --- ## [How to sanitize string input containing \<\<194, 160\>\> and maybe other junk?](https://forum.elixirforum.com/t/how-to-sanitize-string-input-containing-194-160-and-maybe-other-junk/50434)
**Author:** [@dogweather](https://forum.elixirforum.com/u/dogweather)\ **Replies:** 1\ **Last updated:** [September 22, 2022, 2:07am UTC](https://forum.elixirforum.com/t/how-to-sanitize-string-input-containing-194-160-and-maybe-other-junk/50434 "2022-09-22T02:07:14Z")
I’m parsing a lot of Government web pages which were produced on ancient Microsoft systems. Like the .aspx Oregon Revised Statutes homepage. Or awkward not-really-html files like this “windows-1252” converted using MS Wo… --- ## [Applying :crypto and Jason.encode on Unicode characters](https://forum.elixirforum.com/t/applying-crypto-and-jason-encode-on-unicode-characters/45411)
**Author:** [@chemist](https://forum.elixirforum.com/u/chemist)\ **Replies:** 7\ **Last updated:** [January 21, 2022, 5:37pm UTC](https://forum.elixirforum.com/t/applying-crypto-and-jason-encode-on-unicode-characters/45411 "2022-01-21T17:37:48Z")
Spent quite a bit of time pondering this as I am unable to get the desired output similar to the PHP function that im trying to duplicate: plaintext = \["\","'bar'","\\"baz\\"","&blong&"\] |\> Jason.encode! key = "secret… --- ## [Fun with unicode chardata](https://forum.elixirforum.com/t/fun-with-unicode-chardata/42666)
**Author:** [@derek-zhou](https://forum.elixirforum.com/u/derek-zhou)\ **Replies:** 2\ **Last updated:** [September 28, 2021, 3:25pm UTC](https://forum.elixirforum.com/t/fun-with-unicode-chardata/42666 "2021-09-28T15:25:38Z")
Just figured out a tricky bug, which I’d like to share with you. I read binary data from a port, and I know the program on the other side is sending nothing but utf-8 strings. However, the list of binary data I received … --- ## [How to make unicode characters?](https://forum.elixirforum.com/t/how-to-make-unicode-characters/38792)
**Author:** [@vrod](https://forum.elixirforum.com/u/vrod)\ **Replies:** 2\ **Last updated:** [April 7, 2021, 7:16pm UTC](https://forum.elixirforum.com/t/how-to-make-unicode-characters/38792 "2021-04-07T19:16:35Z")
I understand that \\u can help make unicode characters like "\\u00E4" becomes "ä". How can I take a variable like "00E4" and convert it to "ä"? I cannot do "\\u#{var}". I ithink first convert from hex to integer maybe? --- ## [Ex\_confusables - an Unicode tr39 confusable detection and skeleton implementation](https://forum.elixirforum.com/t/ex-confusables-an-unicode-tr39-confusable-detection-and-skeleton-implementation/38607)
**Author:** [@ColaCheng](https://forum.elixirforum.com/u/ColaCheng)\ **Replies:** 7\ **Last updated:** [April 1, 2021, 11:05pm UTC](https://forum.elixirforum.com/t/ex-confusables-an-unicode-tr39-confusable-detection-and-skeleton-implementation/38607 "2021-04-01T23:05:00Z")
Hi all, I have made an unicode tr39 confusable detection and skeleton implementation, ex\_confusables This library is able to compare two strings if they are visually confusable as described in Unicode® Technical Standa… --- ## [Unicode for emoji - string concatenation](https://forum.elixirforum.com/t/unicode-for-emoji-string-concatenation/32235)
**Author:** [@shotleybuilder](https://forum.elixirforum.com/u/shotleybuilder)\ **Replies:** 12\ **Last updated:** [June 10, 2020, 10:17am UTC](https://forum.elixirforum.com/t/unicode-for-emoji-string-concatenation/32235 "2020-06-10T10:17:07Z")
Hi, I’m not understanding the behaviour when including unicode inside a string which is then string concatenated. The resulting string is not known when streamed into a file. Here’s something that works: "❔ foobar" -\>… --- ## [Ucwidth - A small library to detect the display width in columns for Unicode texts](https://forum.elixirforum.com/t/ucwidth-a-small-library-to-detect-the-display-width-in-columns-for-unicode-texts/30665)
**Author:** [@qhwa](https://forum.elixirforum.com/u/qhwa)\ **Replies:** 0\ **Last updated:** [April 16, 2020, 2:58am UTC](https://forum.elixirforum.com/t/ucwidth-a-small-library-to-detect-the-display-width-in-columns-for-unicode-texts/30665 "2020-04-16T02:58:41Z")
Hi everyone! I just published a small library, Ucwidth to help to detect the display width for any Unicode characters or texts. I think it might be useful if you’re working on something related to text typographing. Li… --- ## [Sort strings without taking into account the accents](https://forum.elixirforum.com/t/sort-strings-without-taking-into-account-the-accents/29790)
**Author:** [@thojanssens1](https://forum.elixirforum.com/u/thojanssens1)\ **Replies:** 9\ **Last updated:** [March 13, 2020, 8:15am UTC](https://forum.elixirforum.com/t/sort-strings-without-taking-into-account-the-accents/29790 "2020-03-13T08:15:24Z")
I want to sort a list of countries. Unfortunately, the sort function will set “Åland Islands” as last element, because it starts with an accent. I would like to sort without taking into account the accents, so that “Ålan… --- ## [Capturing UTF-8 code point, then concatenate the corresponding character; covert code point to utf-8 binary?](https://forum.elixirforum.com/t/capturing-utf-8-code-point-then-concatenate-the-corresponding-character-covert-code-point-to-utf-8-binary/29774)
**Author:** [@fireproofsocks](https://forum.elixirforum.com/u/fireproofsocks)\ **Replies:** 2\ **Last updated:** [March 11, 2020, 4:56pm UTC](https://forum.elixirforum.com/t/capturing-utf-8-code-point-then-concatenate-the-corresponding-character-covert-code-point-to-utf-8-binary/29774 "2020-03-11T16:56:59Z")
I think I’ve been staring at pattern matches so long I’ve started to go blind. What I’d like to do is to inspect the code point for a character and then concatenate the character onto an accumulator. iex\> acc = "" iex\> … --- ## [Bitstring with codepoints of non-ASCII characters](https://forum.elixirforum.com/t/bitstring-with-codepoints-of-non-ascii-characters/29620)
**Author:** [@nikody](https://forum.elixirforum.com/u/nikody)\ **Replies:** 5\ **Last updated:** [March 5, 2020, 9:51am UTC](https://forum.elixirforum.com/t/bitstring-with-codepoints-of-non-ascii-characters/29620 "2020-03-05T09:51:56Z")
I stumbled upon something that baffles me while working with PDF files. The problem arose due to the non-ASCII characters âãÏÓ on the second line of the PDFs. I started with a base64 encoded PDF whose first two lines wh… --- ## [Is there way to tell if a given character (i.e. code point) is printable?](https://forum.elixirforum.com/t/is-there-way-to-tell-if-a-given-character-i-e-code-point-is-printable/29493)
**Author:** [@fireproofsocks](https://forum.elixirforum.com/u/fireproofsocks)\ **Replies:** 1\ **Last updated:** [February 29, 2020, 1:16am UTC](https://forum.elixirforum.com/t/is-there-way-to-tell-if-a-given-character-i-e-code-point-is-printable/29493 "2020-02-29T01:16:50Z")
Is there any way to tell if a given character (e.g. ä) or a code point (e.g. 228) represents a printable character (and not, say, a formatting control or something non-printable)? --- ## [Error when zipping files whose names contain unicode codepoints above 255](https://forum.elixirforum.com/t/error-when-zipping-files-whose-names-contain-unicode-codepoints-above-255/23643)
**Author:** [@mischov](https://forum.elixirforum.com/u/mischov)\ **Replies:** 5\ **Last updated:** [December 30, 2019, 4:19pm UTC](https://forum.elixirforum.com/t/error-when-zipping-files-whose-names-contain-unicode-codepoints-above-255/23643 "2019-12-30T16:19:30Z")
I think I may have found an error in :zip related to zipping files whose names contain unicode codepoints above 255, but I would appreciate it if somebody was interested in double checking and helping get it fixed. Recr… --- ## [Regular expressions in Elixir with unicode characters](https://forum.elixirforum.com/t/regular-expressions-in-elixir-with-unicode-characters/18048)
**Author:** [@abtrapp](https://forum.elixirforum.com/u/abtrapp)\ **Replies:** 28\ **Last updated:** [March 10, 2019, 8:35pm UTC](https://forum.elixirforum.com/t/regular-expressions-in-elixir-with-unicode-characters/18048 "2019-03-10T20:35:24Z")
Hi! Can somebody please explain this: iex(9)\> Regex.match?(~r/\\p{Lu}/, "Ö") true iex(10)\> Regex.match?(~r/\\p{Lu}/, "ö") true iex(11)\> Regex.match?(~r/\\p{Lu}/, "o") false iex(12)\> Regex.match?(~r/\\p{Lu}/, "O") true Why… --- ## [Unicode-savvy \`letter\_or\_digit?\` function?](https://forum.elixirforum.com/t/unicode-savvy-letter-or-digit-function/20254)
**Author:** [@scouten](https://forum.elixirforum.com/u/scouten)\ **Replies:** 10\ **Last updated:** [February 24, 2019, 8:17am UTC](https://forum.elixirforum.com/t/unicode-savvy-letter-or-digit-function/20254 "2019-02-24T08:17:42Z")
I’m parsing a file format by recursing on a charlist. One of the challenges I have in parsing this format is determining whether the current Unicode codepoint is a letter or digit. So far I’ve written this: # HELP: This… --- ## [Regex punctuation matchers](https://forum.elixirforum.com/t/regex-punctuation-matchers/16842)
**Author:** [@mudasobwa](https://forum.elixirforum.com/u/mudasobwa)\ **Replies:** 12\ **Last updated:** [September 26, 2018, 7:21am UTC](https://forum.elixirforum.com/t/regex-punctuation-matchers/16842 "2018-09-26T07:21:30Z")
AFAIU, Elixir delegates regular expression evaluation to Erlang. Erlang claims to support Unicode general category matchers. “ is declared in Unicode spec as LEFT DOUBLE QUOTATION MARK under General punctuation. Both R… --- ## [Unicode output from the debugger](https://forum.elixirforum.com/t/unicode-output-from-the-debugger/16716)
**Author:** [@locopati](https://forum.elixirforum.com/u/locopati)\ **Replies:** 5\ **Last updated:** [September 21, 2018, 7:28am UTC](https://forum.elixirforum.com/t/unicode-output-from-the-debugger/16716 "2018-09-21T07:28:35Z")
I was using the Erlang trace debugger to see how my string parsing code was being called. The strings are Unicode and so the output was like this… (\<0.101.0\>) returned from ‘Elixir.Tinker’:handle/2 → \[\<\<226,191,178\>\>, … --- ## [Regex and unicode returning wrong indexes](https://forum.elixirforum.com/t/regex-and-unicode-returning-wrong-indexes/15080)
**Author:** [@adrian](https://forum.elixirforum.com/u/adrian)\ **Replies:** 7\ **Last updated:** [July 11, 2018, 2:19pm UTC](https://forum.elixirforum.com/t/regex-and-unicode-returning-wrong-indexes/15080 "2018-07-11T14:19:00Z")
Hello, I need to find the position of a match using Regex.run on a text with unicode characters. This is an example: iex\> Regex.run(~r/x/u, "áéíóúx", return: :index) \[{10, 1}\] When te expected output should be: \[{5, … --- ## [Unidecode implementation for Elixir](https://forum.elixirforum.com/t/unidecode-implementation-for-elixir/5953)
**Author:** [@cevado](https://forum.elixirforum.com/u/cevado)\ **Replies:** 2\ **Last updated:** [June 12, 2017, 1:49pm UTC](https://forum.elixirforum.com/t/unidecode-implementation-for-elixir/5953 "2017-06-12T13:49:21Z")
I just published unidecode package to hex: https://github.com/fcevado/unidecode Unidecode was initially a perl module to transliterate unicode characters to ascii, then it became a widespread library throughout several… --- ## [Icon fonts vs SVG icons](https://forum.elixirforum.com/t/icon-fonts-vs-svg-icons/749)
**Author:** [@Qqwy](https://forum.elixirforum.com/u/Qqwy)\ **Replies:** 3\ **Last updated:** [May 31, 2016, 4:23pm UTC](https://forum.elixirforum.com/t/icon-fonts-vs-svg-icons/749 "2016-05-31T16:23:13Z")
Continuing the discussion from Phoenix Blog Posts: Why is this the case? The reasons I found so far through the internet to pick SVGs over icon fonts are: Many fonts are bloated. When the code-points for the icons a…