# Text - a text analysis library **URL:** \ **Category:** Announcing\ **Tags:** library, text\ **Created:** [June 15, 2020, 7:34am UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363 "2020-06-15T07:34:56Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\ **Post date:** [June 15, 2020, 7:34am UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/1 "2020-06-15T07:34:56Z")
I’ll shortly be launching [Text](https://github.com/kipcole9/text), a nascent text analysis library. #### Current functionality In this early version (not ready for prime time) it includes: 1. Word counting 2. N-gram generation 3. Language detection (of about 250 languages with pluggable vocabularies and pluggable correlation models) 4. An English inflector (singular to plural) using a non-regex algorithmic approach #### Future functionality - A language stemmer - as soon as I finishing writing the [snowball compiler](https://snowballstem.org) - Parts of speech tagger #### Collaboration encouraged - Contributions in all areas are most welcome - Non-english speakers who would like to contribute to non-english inflectors are particularly welcome #### Next steps After some polishing this weekend I will publish a version to hex. ---
**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\ **Post date:** [June 21, 2020, 10:43pm UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/2 "2020-06-21T22:43:49Z")
Based on the positive feedback it seems this project has merit. I didn’t get finished all the work I planned for the weekend but I’m clearing a backlog so I can give this project some greater attention. The inflector is finished (nouns, pronouns, verbs). I’ve started work on adding the Metaphone 2 algorithm but its a slow slog because there is no description of the algorithm I can find - just imperative code. And then some more testing and verification on language detection. All in all, likely a one week delay. ---
**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\ **Post date:** [June 22, 2020, 6:12am UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/3 "2020-06-22T06:12:47Z")
Thought I’d share the near term roadmap in a little more detail. Feedback is most definitely welcome on the capabilities you would find most useful. Or any areas you’d like to contribute to. #### Step 1: Language recognition Most natural language processing is language dependent. So identifying the source language is important. The primary way of identifying languages is to split the text into n-grams and then perform various statistical analysis of the source text versus the same analysis of a standard corpora in multiple different languages. The [Universal Declaration of Human Rights](http://research.ics.aalto.fi/cog/data/udhr/) is a standard text published in a lot of languages so this is the corpora I’m using. There are different ways to correlate source text versus a corpora. I am primarily using the algorithms in [Language Identification from Text Using N-gram Based Cumulative Frequency Addition](http://www.csis.pace.edu/~ctappert/srd2004/paper12.pdf). This is the due now for delivery on 28th June. #### Step 2: Text segmentation No matter what analysis is required, segmenting the text into grapheme clusters, words and sentences is required. This is very language dependent. Elixir’s `String.graphemes/1` implements the [Unicode segmentation algorithm for grapheme clusters](https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries) so thats taken care of. Elixir’s `String.split/1` implements the [Unicode segmentation algorithm for words](https://unicode.org/reports/tr29/#Word_Boundaries). `String.split/1` is great for a default case but its not sufficient for language-specific segmentation. And we still need sentence segmentation too. Therefore I am implementing the [CLDR Segmentation rules](https://unicode-org.github.io/cldr/ldml/tr35-general.html#Segmentations) which provide language-specific customisation for text segmentation. This is another rules parser (I think so far I have implemented 8 different rules parsers and “compilers” in various parts of the [ex\_cldr](https://github.com/elixir-cldr) project). The text segmentation algorithms will be implemented as part of the [unicode\_string](https://github.com/elixir-unicode/unicode-string) library. #### Step 3: Parts of Speech Tagging Now we have segments of text we can proceed to understanding what is being expressed. The starting point for this is called “parts of speech tagging”. Because I want a good native Elixir implementation that supports a wide variety of languages with good (but not necessarily the absolutely best) tagging I’m using [A Rule-based Part-of-Speech and Morphological Tagging Toolkit](http://rdrpostagger.sourceforge.net) which provides a fully trained corpora for ~90 languages using the open source data maintained in the [Universal Dependencies](https://universaldependencies.org) [treebanks](https://en.wikipedia.org/wiki/Treebank). The trained models are maintained in the [RDRPOSTagger project](https://github.com/datquocnguyen/RDRPOSTagger) which also defines a [rules engine](https://github.com/datquocnguyen/RDRPOSTagger/blob/master/Models/MORPH/French.RDR) that I will implement in Elixir. Another parser/compiler 🙂 #### Step 4: Sentiment Analysis Now that we have a grammatical breakdown of the target source we can start to identify meaning. Wikipedia says: > A basic task in sentiment analysis is classifying the _polarity_ of a given text at the document, sentence, or feature/aspect level—whether the expressed opinion in a document, a sentence or an entity feature/aspect is positive, negative, or neutral. Advanced, “beyond polarity” sentiment classification looks, for instance, at emotional states such as “angry”, “sad”, and “happy”. The implementation approach is not yet defined and feedback and suggestions are warmly welcomed. #### Step 5 To be determined. It will take a 2-4 months to get through the first 4 steps so thats plenty of time for feedback and collaboration 🙂 ---
**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\ **Post date:** [June 29, 2020, 12:50am UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/4 "2020-06-29T00:50:35Z")
[text version 2.0](https://hex.pm/pacakges/text/0.2.0) has been published today, just on schedule. In addition the library [text\_corpus\_udhr](https://hex.pm/packages/text_corpus_udhr) is also published today - it provides a corpus to support natural language detection. #### Language Detection `text` contains 3 language classifiers to aid in natural language detection. However it does not include any corpora; these are contained in separate libraries. The available classifiers are: - `Text.Language.Classifier.CommulativeFrequency` - `Text.Language.Classifier.NaiveBayesian` - `Text.Language.Classifier.RankOrder` Additional classifiers can be added by defining a module that implements the `Text.Language.Classifier` behaviour. The library [text\_corpus\_udhr](https://hex.pm/packages/text_corpus_udhr) implements the `Text.Corpus` behaviour for the [United National Declaration of Human Rights](https://en.wikipedia.org/wiki/Universal_Declaration_of_Human_Rights) which is available for download in 423 languages from [Unicode](https://unicode.org/udhr/). Examples: ```elixir iex> Text.Language.detect "this is some english language thing" {:ok, "en"} # Options include `:corpus`, `:vocabulary` and `:classifier` iex> Text.Language.detect "this is some english language thing", corpus: Text.Corpus.Udhr, vocabulary: Text.Vocabulary.Udhr.Quadgram {:ok, "en"} ``` #### Word Counting `text` contains an implementation of word counting that is oriented towards large streams of words rather than discrete strings. Input to `Text.Word.word_count/2` can be a `String.t`, `File.Stream.t` or `Flow.t` allowing flexible streaming of text. #### English Pluralization `text` includes an inflector for the English language that takes an approach based upon [An Algorithmic Approach to English Pluralization](http://users.monash.edu/~damian/papers/HTML/Plurals.html). See the module `Text.Inflect.En` and the functions: - `Text.Inflect.En.pluralize/2` - `Text.Inflect.En.pluralize_noun/2` - `Text.Inflect.En.pluralize_verb/1` - `Text.Inflect.En.pluralize_adjective/1` #### N-Gram generation The `Text.Ngram` module supports efficient generation of n-grams of length `2` to `7`. See `Text.Ngram.ngram/2`. #### Language detection accuracy Detection accuracy is reliable at text lengths of 150 characters or more, reasonable at 100 characters and may not be considered acceptable at shorter lengths. The results are consistent for the range of tested languages with German being a clear exception where the results are unacceptable for now. - English - Greek - Russian - Spanish - Finnish - French - Icelandic - Italian - Japanese - Simplified Chinese Further details are contained in the [github repo](https://github.com/kipcole9/text_corpus_udhr) in the `analysis` directory. #### English language with Naive Bayesian classifier `Text.Language.detect/2` with `classifier: Text.Classifier.NaiveBayesian` and three different vocabularies. | Text Length | Udhr.Bigram | Udhr.Multigram | Udhr.Quadgram | | --- | --- | --- | --- | | 50 | 95.6% | 92.7% | 95.6% | | 100 | 99.9% | 99.5% | 98.8% | | 150 | 100.0% | 100.0% | 99.3% | | 300 | 100.0% | 100.0% | 100.0% | #### Accuracy for German language detection German is an exception to the consistent accuracy of most languages and the results are poor. Further analysis is required to understand the underlying cause. | Text Length | Udhr.Bigram | Udhr.Multigram | Udhr.Quadgram | | --- | --- | --- | --- | | 50 | 45.6% | 38.0% | 27.2% | | 100 | 64.4% | 47.2% | 42.3% | | 150 | 71.6% | 57.6% | 51.4% | | 300 | 78.7% | 47.2% | 56.9% | ---
**Author:** ![josefrichter](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/josefrichter/32/19529_2.png) [@josefrichter](https://forum.elixirforum.com/u/josefrichter)\ **Post date:** [June 29, 2020, 8:09am UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/5 "2020-06-29T08:09:43Z")
> [@kip](#): > > - Non-english speakers who would like to contribute to non-english inflectors are particularly welcome How can I do that? ---
**Author:** ![ondrej-tucek](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ondrej-tucek/32/11566_2.png) [@ondrej-tucek](https://forum.elixirforum.com/u/ondrej-tucek)\ **Post date:** [June 29, 2020, 8:59am UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/6 "2020-06-29T08:59:21Z")
I would say that you have to create PR in [kipcole9/text\_corpus\_udhr](https://github.com/kipcole9/text_corpus_udhr) where you put a file, e.g. `corpus/udhr/udhr_cze.txt` with translation of [udhr\_eng.txt](https://github.com/kipcole9/text_corpus_udhr/blob/master/corpus/udhr/udhr_eng.txt). ---
**Author:** ![josefrichter](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/josefrichter/32/19529_2.png) [@josefrichter](https://forum.elixirforum.com/u/josefrichter)\ **Post date:** [June 29, 2020, 9:11am UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/7 "2020-06-29T09:11:23Z")
Seems to be there already, right? [text\_corpus\_udhr/corpus/udhr/udhr\_ces.txt at master · kipcole9-legacy/text\_corpus\_udhr · GitHub](https://github.com/kipcole9/text_corpus_udhr/blob/master/corpus/udhr/udhr_ces.txt) ---
**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\ **Post date:** [June 29, 2020, 9:27am UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/8 "2020-06-29T09:27:20Z")
Thanks for the interest! Language detection should work at an acceptable level for ~200 languages using the UDHR corpus. Of course you can also contribute additional corpora in a library of your own making as long as it has a module that implements the `Text.Corpus` behaviour. Inflection - specifically pluralisation - is on a per-language basis. If you would like to contribute an inflector then a PR with a module called `Text.Inflect.` that implements a function called `pluralize/2` would be “all” thats required. I will define a `Text.Inflection` behaviour in the `0.3.0` release to make this more clearly defined. ---
**Author:** ![ondrej-tucek](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ondrej-tucek/32/11566_2.png) [@ondrej-tucek](https://forum.elixirforum.com/u/ondrej-tucek)\ **Post date:** [June 29, 2020, 10:49am UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/9 "2020-06-29T10:49:48Z")
Mea culpa, I was looking for that file but obviously not successfully… ---
**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\ **Post date:** [June 29, 2020, 10:52am UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/10 "2020-06-29T10:52:33Z")
Nothing to apologise for. I need to add a _Contributing_ section to the docs - thankis for the prompt to do so 🙂 ---
**Author:** ![smolcatgirl](https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/s/c5a1d2/32.png) [@smolcatgirl](https://forum.elixirforum.com/u/smolcatgirl)\ **Post date:** [June 29, 2020, 7:15pm UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/11 "2020-06-29T19:15:52Z")
I think this is cool but i dont have a usecase for it. Keep up the good work 👍 ---
**Author:** ![sorentwo](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/sorentwo/32/37360_2.png) [@sorentwo](https://forum.elixirforum.com/u/sorentwo)\ **Post date:** [June 29, 2020, 10:28pm UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/12 "2020-06-29T22:28:08Z")
I (we, at dscout) definitely have a usecase for nearly all of this work . I hope to contribute in the future, and would love to support the effort financially if you decide to make that possible 💛. ---
**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\ **Post date:** [June 29, 2020, 11:35pm UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/13 "2020-06-29T23:35:34Z")
Just a little fun addition over coffee this morning - [deriving a CLDR locale from natural language](https://github.com/elixir-cldr/cldr-text). I’ll publish it to hex after I add some tests. #### Examples ```elixir iex> Cldr.Text.locale_from_text "this is some text that I think will be English" {:ok, %Cldr.LanguageTag{ backend: MyApp.Cldr, canonical_locale_name: "en-Latn-US", cldr_locale_name: "en", extensions: %{}, gettext_locale_name: nil, language: "en", language_subtags: [], language_variant: nil, locale: %{}, private_use: [], rbnf_locale_name: "en", requested_locale_name: "en", script: "Latn", territory: :US, transform: %{} }} iex> german_text = "Wir wohnen in einem kleinen Haus mit einem Garten. Dort können die Kinder ein bisschen spielen. Unser Sohn kommt bald in die Schule, unsere Tochter geht noch eine Zeit lang in den Kindergarten. Meine Kinder sind am Nachmittag zu Hause. So arbeite ich nur halbtags." iex> Cldr.Text.locale_from_text german_text {:ok, %Cldr.LanguageTag{ backend: MyApp.Cldr, canonical_locale_name: "de-Latn-DE-1901", cldr_locale_name: "de", extensions: %{}, gettext_locale_name: nil, language: "de", language_subtags: [], language_variant: "1901", locale: %{}, private_use: [], rbnf_locale_name: "de", requested_locale_name: "de-1901", script: "Latn", territory: :DE, transform: %{} }} ``` ---
**Author:** ![rengel](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/rengel/32/20073_2.png) [@rengel](https://forum.elixirforum.com/u/rengel)\ **Post date:** [July 31, 2020, 6:48am UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/14 "2020-07-31T06:48:07Z")
Just stumbled upon this post. In case you didn’t konw: > **[Client Challenge](https://pypi.org/project/Metaphone/)** ---
**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\ **Post date:** [July 31, 2020, 6:56am UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/15 "2020-07-31T06:56:25Z")
Thanks much for the link. I’m a bit challenged reading these imperative implementations for two reasons: (a) such ugly code compared to using pattern matching for most of it as one would in Elixir and (b) as a result, I just want the rules. Megaphone I can find them, but not double. Maybe I’ll do a basic Metaphone implementation first and at least move forward … ---
**Author:** ![tfwright](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/tfwright/32/20299_2.png) [@tfwright](https://forum.elixirforum.com/u/tfwright)\ **Post date:** [August 6, 2020, 8:20pm UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/16 "2020-08-06T20:20:43Z")
I’m really interested in using this library in a project of mine, in particular to generate something similar to “word clouds” where common _significant_ words are highlighted. Is that something you are planning on supporting? Please let me know if there’s any part I can help out with. ---
**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\ **Post date:** [April 29, 2026, 8:41am UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/17 "2026-04-29T08:41:05Z")
> I’ll shortly be launching [Text](https://github.com/kipcole9/text), a nascent text analysis library. Well, instead of _shortly_, what I should have said is _in about 6 years_. But fear not: ![Its-not-dead-its-resting-meme-generator-e1597175206791](https://forum.elixirforum.com/uploads/default/original/3X/f/5/f593d49908c02e4aff7d5084478c961edb4cc7ba.jpeg) _See the [YouTube video](https://www.youtube.com/watch?v=4vuW6tQ0218) if the meme isn’t familiar to you_. A lot has change in the NLP world since 2020 and the new, modernised and thoroughly renovated [text 0.3.0](https://hex.pm/packages/text/0.3.0) is now available on Hex. ## Features ### Language identification `Text.Language.Classifier.Fasttext` is a pure-Elixir port of [lid.176](https://fasttext.cc/docs/en/language-identification.html), validated bit-for-bit against fastText’s reference. 176 languages, ~100 µs per prediction with EXLA. ### Sentiment, POS, NER Multilingual sentiment via bundled [AFINN](http://corpustext.com/reference/sentiment_afinn.html) lexicons (default) or [XLM-RoBERTa](https://huggingface.co/docs/transformers/en/model_doc/xlm-roberta) through Bumblebee (optional). Part-of-speech tagging and multilingual named-entity recognition via Bumblebee, with `mix text.download_models` to pre-fetch the weights at deploy time. ### Word clouds A new `Text.WordCloud` module with six scoring backends — [YAKE!](http://yake.inesctec.pt) by default, plus frequency, RAKE, TextRank, TF-IDF, and a KeyBERT variant for users who configure Bumblebee. `Text.WordCloud.Layout` does Wordle-style spiral packing (with `:radial` and `:spiral` orientation modes for sunburst and vortex looks), and `Text.WordCloud.SVG` produces renderer-agnostic output that plays nicely with `Color.Palette` for coordinated tonal-scale colour ramps. You can see some examples in [the guide](https://hexdocs.pm/text/word_clouds.html). ### Fundamentals - String distance and similarity (Levenshtein, Damerau-Levenshtein, Jaro-Winkler, Jaccard, cosine, …) - Phonetic encoding (Soundex, Metaphone), - Unicode-aware segmentation, slug generation, TF-IDF and BM25 search, - Collocation extraction (PMI, log-likelihood), and - Keyword-in-context concordance. Also includes bundled stopwords for ~60 languages from stopwords-iso, and optional Snowball stemming via `:text_stemmer` to consolidate morphological variants in word clouds. ### Optional ML Heavy ML deps (`:bumblebee`, `:exla`, KeyBERT) are all optional. Without them, the package still does most of what it does — just without the neural-quality ceiling. Same goes for `:color` (SVG palettes), `:localize` (CLDR locale resolution), and `:text_stemmer`. ---
**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\ **Post date:** [April 29, 2026, 8:52am UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/18 "2026-04-29T08:52:02Z")
There are two companion libraries also published today: - [Snowball](https://hex.pm/packages/snowball) that implements the [Snowball language](https://snowballstem.org) as a cross compiler to Elixir. Its a fun language with its roots in [SNOBOL](https://www.snobol5.com/intro1.htm) but was designed by [Porter](https://en.wikipedia.org/wiki/Martin_Porter) specifically to support the implementation of language stemmers. This is not a general purpose language or compiler - there are no conveniences at all. Just a mix task to take `.sbl` files and cross-compile them to Elixir. - [text\_stemmer](https://hex.pm/packages/text_stemmer) which implements all [37 stemming algorithms](https://snowballstem.org/algorithms/) - validated against their respective conformance suites. [text\_stemmer](https://hex.pm/packages/text_stemmer) is an optional dependency for [text](https://hex.pm/packages/text) which can be used to tune word clouds. These two are 100% LLM generated. It took 4 complete days for Claude (Sonnet) to implement. It’s a classic LLM opportunity. Formal specification, canonical implementations, complete conformance testing suites. Prompt the LLM and then go work on [something else](https://forum.elixirforum.com/t/what-do-you-do-while-waiting-for-it-claude-code/75143). ---
**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\ **Post date:** [April 30, 2026, 10:57pm UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/19 "2026-04-30T22:57:10Z")
I’ve published [Text 0.4.0](https://hex.pm/packages/text/0.4.0) today with seven new NLP modules (all native Elixir, no NIF or ML). They cover the kinds of preprocessing you might reach for once your sentiment / classification / search pipeline outgrows `String.split/1`. This release represents a largely feature complete text library from my perspective. Happy to take feature suggestions though. A few of the more immediately useful additions in this release: ### `Text.Clean` — pipeline-style normalisation Whitespace, control characters, smart quotes, mojibake, NFC/NFKC. Composable; defaults are sensible. ```elixir iex> Text.Clean.clean("

it’s cool

") "it's cool" iex> Text.Clean.collapse_whitespace(" hello \tworld \n") "hello world" ``` ### `Text.Truecase` — restore casing for ALL-CAPS or lowercased text POS-aware heuristics for proper nouns, acronyms, and sentence starts. Useful when an upstream system has destroyed the casing (chat logs, OCR, screaming customer feedback). ```elixir iex> Text.Truecase.truecase("THE QUICK BROWN FOX JUMPS OVER NEW YORK") "The quick brown fox jumps over New York" iex> Text.Truecase.truecase("nasa launched apollo 11 in july 1969.") "NASA launched Apollo 11 in July 1969." # Add domain-specific terms once at boot Text.Truecase.add_terms(["GraphQL", "Phoenix"]) Text.Truecase.truecase("we use phoenix and graphql") #=> "we use Phoenix and GraphQL" ``` ### `Text.Emoji` — detection, stripping, counting, conversion Backed by the `:unicode` package’s emoji property tables, so it recognises every codepoint flagged emoji in the current Unicode release — no shipped JSON. ```elixir iex> Text.Emoji.count("Loved it 🤩 read it twice 📚📚") 3 iex> Text.Emoji.demojize("ship it 🚀") "ship it :rocket:" iex> Text.Emoji.emojize("ship it :rocket:") "ship it 🚀" ``` ### `Text.Hyphenation` — Knuth–Liang TeX-pattern hyphenation Ships en-US patterns baked in (~5 000). Other languages load from any standard `hyph-*.tex` file. ```elixir iex> Text.Hyphenation.hyphenate("hyphenation") "hy-phen-ation" iex> Text.Hyphenation.count("supercalifragilisticexpialidocious") 9 # Load German patterns once; thereafter all calls are fast Text.Hyphenation.load_language(:de, path: "hyph-de-1996.tex") Text.Hyphenation.hyphenate("Bundesausbildungsförderungsgesetz", language: :de) #=> "Bun-des-aus-bil-dungs-för-de-rungs-ge-setz" ``` ### `Text.PII` — detect & redact common identifiers Phone, email, credit-card-shaped digits, IBANs, IPv4/IPv6, US SSN. Pattern-based — fast and deterministic. The right tool for _“please don’t paste this into the LLM”_ preflight; pair with a stricter checker if you need legal-grade accuracy. ```elixir iex> Text.PII.detect("Email me at jane@example.com or call (415) 555-0142.") [%{type: :email, value: "jane@example.com", offset: 12, length: 16}, %{type: :phone, value: "(415) 555-0142", offset: 37, length: 14}] iex> Text.PII.redact("Card 4111-1111-1111-1111 expires 12/29") "Card [CREDIT_CARD] expires 12/29" ``` ### `Text.Spell` — Norvig-style spelling suggestions Edit-distance candidates ranked by frequency in `Text.WordFreq` (the 30,000-word English frequency table that also ships in 0.4.0). ```elixir iex> Text.Spell.correct("speling") "spelling" iex> Text.Spell.candidates("teh") |> Enum.take(3) [%{word: "the", distance: 1, frequency: 6_187_267}, %{word: "tech", distance: 1, frequency: 49_320}, %{word: "ten", distance: 1, frequency: 21_117}] ``` ### `Text.Summarize` — extractive summarisation via TextRank Sentence-graph TextRank with configurable similarity (`:cosine` or `:jaccard`) and target length. ```elixir article = """ The new bridge, opened on Tuesday, connects the two halves of the city for the first time in decades. Engineers worked three winters to anchor the central pier on the riverbed. Residents who used to take a 40-minute ferry now make the trip in five. The mayor said the project came in 2 % under budget, a rarity for civic work of this scale. """ iex> Text.Summarize.summarize(article, sentences: 2) "The new bridge, opened on Tuesday, connects the two halves of the city for the first time in decades. Residents who used to take a 40-minute ferry now make the trip in five." ``` ---
**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\ **Post date:** [May 4, 2026, 2:55am UTC](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363/20 "2026-05-04T02:55:33Z")
I’ve published a couple of updates in the last week. I think that pretty much all I need in a text analysis library but I’m more than happy to take feature requests. ## [0.6.0] — 2026-05-04 ### Added - `Text.Extract` — twitter-text-quality URL and email extraction with full UTS #46 IDNA, IANA TLD validation, and UTR #39 single-script defence against homograph attacks. Public API is `urls/2`, `emails/2`, `all/2`, `split/2`, and `autolink/2`; options include `:require_scheme`, `:tld_mode`, `:eai`, `:strict_idn`, and `:twitter_quirks`. - `Text.Extract.split/2` — splits text into an interleaved list of plain-string fragments and validated entity maps, byte-for-byte round-trippable to the original. The building block for custom rendering of extracted URLs/emails into anchors, mentions, badges, or link-preview cards. - `Text.Extract.autolink/2` — wraps URLs and emails in HTML `` anchors, returning `Phoenix.HTML.safe()` for drop-in Phoenix template use. Display text preserves the original Unicode (`bücher.de`); the `href` uses Punycode (`xn--bcher-kva.de`). - `mix text.download_tlds` — refreshes the bundled IANA TLD list at `priv/extract/tlds.txt`. `--diff` previews added/removed entries; `--force` overwrites unconditionally. - `Text.WordCloud.to_d3_cloud/2` — adapts `terms/2` output into the `[%{text, size}, …]` shape consumed by [d3-cloud](https://github.com/jasondavies/d3-cloud). Supports `:linear` (default) and `:sqrt` sizing; shares the `:font_size_range` vocabulary with `Text.WordCloud.Layout`. ## [0.5.0] — 2026-05-02 ### Added - `Text.Phonetic.NYSIIS` — New York State Identification and Intelligence System phonetic encoding (Taft, 1970). Designed as a Soundex successor for English personal-name matching; produces pronounceable letter codes rather than digits and is more discriminating than Soundex on common name variations. - `Text.Phonetic.Cologne` — Kölner Phonetik (Postel, 1969), the German-language counterpart to Soundex. Optimized for German spelling variants — `Müller` / `Mueller` / `Muller` and `Meyer` / `Mayer` / `Maier` / `Meier` collapse to single codes. - `Text.Phonetic.DoubleMetaphone` — Lawrence Philips’ Double Metaphone (2000), the de-facto standard for fuzzy English-name matching with non-Anglo origins. Returns a `{primary, alternate}` code pair so the same Anglicised name can match across multiple plausible pronunciations (e.g. `Smith` ↔ `Schmidt`, `Catherine` ↔ `Katherine`). Handles Germanic, Italian, Spanish, French, Greek, and Slavic patterns. - `match?/2` (and `match?/3` where options apply) on every `Text.Phonetic.*` module for direct equality comparison without manual `encode/2 == encode/2` boilerplate. `Text.Phonetic.DoubleMetaphone.match?/3` checks all four primary/alternate combinations. - `Text.Clean.unaccent/1` — strip diacritics and fold non-decomposable Latin letters (`Þ` → `Th`, `ß` → `ss`, `Æ` → `AE`, `ł` → `l`, `đ` → `d`) by delegating to `Unicode.Transform.LatinAscii.transform/1`. Also exposed as the `:unaccent` option on `Text.Clean.clean/2`. - `Text.Distance` gains four set-based similarity metrics over character n-grams: `jaccard/3`, `sorensen_dice/3`, `tanimoto/3` (alias for `jaccard/3`), and `cosine/3`. All accept an `:n` option for configurable shingle size (default 2). Operate at the grapheme level for Unicode correctness. - `Text.Inflect.En.singularize/2` and `Text.Inflect.En.singularize_noun/2` — invert the existing pluralizer. Combines reverse lookup of Conway’s irregular tables, explicit suffix rules for unambiguous English plural forms (`-ies`, `-shes`/`-ches`/`-xes`/`-zes`/`-sses`), small whitelists for Greek-derived `-is`/`-es` plurals (`analyses → analysis`) and English `-us` plurals (`geniuses → genius`), and a `pluralize/2` round-trip search to validate other candidates. - `Text.Readability.dale_chall/2` and `Text.Readability.spache/2` — the two classic word-list readability indices, backed by bundled easy-words lists in `priv/readability/` (Dale-Chall 2,949 words, Spache 1,063 words; both sourced from the MIT-licensed `py-readability-metrics` distribution of the public-domain originals). `statistics/2` now also returns `:difficult_words` and `:unfamiliar_words` counts. - `Text.Hyphenation` bundles six additional language packs: `de-1996`, `fr`, `es`, `it`, `nl`, `pt`. All loaded at compile time with zero I/O, joining the existing `en-us` pack. Source: hyph-utf8 upstream; per-file licenses (MIT/X11/BSD/LPPL) are preserved in each `.tex` header. - `Text.WordFreq` bundles six additional frequency tables at the same top-30,000 cap as English: `de`, `fr`, `es`, `it`, `nl`, `pt`. Source: Hermit Dave’s MIT-licensed [FrequencyWords](https://github.com/hermitdave/FrequencyWords) OpenSubtitles 2018 corpus. - `Text.Emoji.sentiment/1` and `Text.Emoji.text_sentiment/1` — per-emoji and aggregate sentiment scoring backed by the bundled Emoji Sentiment Ranking v1.0 (Kralj Novak et al., 2015; ~750 emoji with negative/neutral/positive proportions and an aggregate score in `[-1.0, 1.0]`). Aggregate scoring is occurrence-weighted to match the original paper. [Next page](https://forum.elixirforum.com/t/text-a-text-analysis-library/32363.md?page=2)