I’ve spun out a streaming bzip2 codec as part of the work to support Elixir processing of Wikipedia dumps.
Example usage:
Mix.install([:bzip2])
:inets.start()
:ssl.start()
tmpfile = '/tmp/articles.xml.bz2'
url = 'https://dumps.wikimedia.org/testwiki/20220501/testwiki-20220501-pages-articles.xml.bz2'
{:ok, :saved_to_file} = :httpc.request(:get, {url, []}, [], [stream: tmpfile])
File.stream!(to_string(tmpfile), [], 900 * 1024)
|> Bzip2.decompress!()
|> Enum.into("")
# ...
# <page>
# <title>Wikipedia:Help</title>
# <ns>4</ns>
# <id>80</id>
# <redirect title="Main Page" />
# <revision>
# <id>438551</id>
# <parentid>438171</parentid>
# <timestamp>2020-07-01T07:11:36Z</timestamp>
# <contributor>
# <username>JohanahoJ</username>
# <id>37147</id>
# </contributor>
# <comment>Changed redirect target from [[Wikipedia:Main Page]] to [[Main Page]]</comment>
# <model>wikitext</model>
# <format>text/x-wiki</format>
# ...
I would have liked to give a fully streamed example, but :httpc requires more than a dab of glue to do so, and I can’t find any other currently working library to stream HTTP.
