Hi, i have been working on a code that counts words on a file. I mostly used standard libraries from Elixir, then tried to run some tests to see how it responds. I tried a simple text with few words and something absolutely ireal with 500k lines of text.. And i tought it’s probably becouse of not us…

[image] jgarrido: &String.split(&1, ~r/(*UTF)\W/, trim: true) You probably want to :binary.copy/1 individual chunks, to avoid keeping refs to the containing binary forever. The earlier you copy, the earlier the upper refs can be GC’d Also, words |> Strem.map/2 |> Enum.to_list can be written …

String.to_atom(word) Here is your answer, as @NobbZ says, try to avoid this. The atoms on the beam are not collected by the garbage collector, and that might be the cause of your memory increasing even after the code finished

A general note about using Stream functions: the benefit of streaming is not holding the whole file in memory. Calling Enum.to_list on the stream totally eliminates that benefit. For instance: readFile |> File.stream! |> Stream.flat_map(&String.split(...)) |> Stream.map(&String.downcase/1) |> Enum…

This still has the references problem, as no explicit copy of the split substrings happens.

Looks like String.downcase reassembles the string from individual codepoints: lib/elixir/unicode/unicode.ex 18c44e2e4 end def downcase(<<byte, rest::bits>>, acc, mode) do if byte >= ?A and byte <= ?Z do downcase(re…

Thanks all for the answers! I updated the code with some of your advices. It runs so efficiently now! Here you can see it: defmodule RepeatedWords do def word_count(readFile) do readFile |> File.stream! |> sanitize |> count |> order end defp sanitize(words) do …

Elixir Programming Language Forum

Standard Libraries vs Tail-Recursion

Questions & Help Questions

NobbZ March 18, 2022, 6:02pm 7

Indeed that should do it.