# Library to split string on unicode word boundaries

**URL:** <https://forum.elixirforum.com/t/library-to-split-string-on-unicode-word-boundaries/50421>\
**Category:** Questions\
**Tags:** unicode, strings, rust\
**Created:** [September 21, 2022, 1:14pm UTC](https://forum.elixirforum.com/t/library-to-split-string-on-unicode-word-boundaries/50421 "2022-09-21T13:14:03Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\
**Post date:** [September 21, 2022, 4:35pm UTC](https://forum.elixirforum.com/t/library-to-split-string-on-unicode-word-boundaries/50421/3 "2022-09-21T16:35:09Z")

</div>

[unicode\_string](https://hex.pm/packages/unicode_string) implements the [Unicode text segmentation specification](https://unicode.org/reports/tr29/) and for your case specifically the [Unicode word break algorithm](https://unicode.org/reports/tr29/#Word_Boundaries). The current version is based upon [Unicode 15](https://unicode.org/versions/Unicode15.0.0) released on September 13, 2022.

See [Unicode.String.split/2](https://hexdocs.pm/unicode_string/Unicode.String.html#split/2).

TR 29 provides specific break data for several but not all locales. Some locales have data only for line break suppressions (ie abbreviations like `i.e.` that do not signal a line break) and some include data for word breaks too, like the [“ja” locale](https://github.com/unicode-org/cldr/blob/main/common/segments/ja.xml#L20-L31).

Otherwise the [default break rules](https://github.com/unicode-org/cldr/blob/main/common/segments/root.xml) are used and worth a look to see what is going on under the hood. Its not as straight forward as you might think. All of these rules are used to [define functions at compile time](https://github.com/elixir-unicode/unicode_string/blob/master/lib/unicode/segment.ex) to optimise performance.

```elixir
iex> Unicode.String.Segment.known_locales
["de", "el", "en", "en-US", "en-US-POSIX", "es", "fi", "fr", "it", "ja", "pt",
 "root", "ru", "sv", "zh", "zh-Hant"]

```

Here are your examples, using `Unicode.String.split/2`:

```elixir
# "en" uses the default word break rules
iex> Unicode.String.split "the quick brown fox", trim: true
["the", "quick", "brown", "fox"]
# There are word break rules specific to "zh"
iex> Unicode.String.split "这只是一些随机的文本", locale: "zh"
["这", "只", "是", "一", "些", "随", "机", "的", "文", "本"]

```

Japanese does have specific word break rules so here’s an example. Notice that there is no whitespace separation. Word breaks in Japanese are determined by other means.

```elixir
iex> Unicode.String.split "注ちゅう文もんの多おおい料りょう理り店", locale: "ja"
["注", "ちゅう", "文", "もんの", "多", "おおい", "料", "りょう",
 "理", "り", "店"]

```

---

_[View the full topic](https://forum.elixirforum.com/t/library-to-split-string-on-unicode-word-boundaries/50421)._