Hello :slight_smile: We are trying to export a “huge” amount of data from a database using XML. Our XML exporter service is running on a cluster and it has a limited amount of memory (e.g. 4GiB of RAM) so we cannot keep in memory the whole XML document. Does anyone knows any good Elixir (or erlan…

Maybe this search will help Google Search This emerged GitHub - processone/fast_xml: Fast Expat based Erlang XML parsing library · GitHub

This does not help: [image] wolfiton: This emerged GitHub - processone/fast_xml: Fast Expat based Erlang XML parsing library · GitHub Fast Expat based Erlang XML parsing library. I’m looking for a writer, not a parser.

I hate to bring this suffering upon you but for excellent performance you should definitely just use Erlang’s :xmerl . It’s not exactly easy to work with, ask away if you get stuck. It does support streaming, both reading and writing, so RAM will not be a problem.

Thanks, I’ll look for it. IMHO there is a huge lack of documentation about how to use it for streaming purposes. Are you aware of any useful information?

I have a few scattered links that can help you get started. xmerl is not very intuitive for a newcomer so I see some cursing in your future. :003: Notes on xmerl and processing XML in Erlang . This one can help you understand how can you construct :xmerl XML nodes programmatically (as opposed to re…

Thanks for you reply :slight_smile: I’m still reading and trying to understand them. Anyway most of your links are not useful for us right now, since we need to write a XML file without keeping it in memory, while I think most of your links are good for stream parsing (which is the opposite).

You might consider keeping it simple and handle writing the opening/closing tags yourself. For example using :xmerl to format chunks of XML to binary or IO list form using Stream.map. Then write those to a custom file stream. For the File streams, it might just be easiest to write your own in one of…

The first link talks about constructing :xmerl structures to then serialise to files. As already mentioned, the rest of the links are for gaining understanding in the library.

[image] elcritch: Of course, even just writing a little library yourself with a reverse SAX style “write tag start” and “write tag end” wouldn’t take too much effort (I’d guess a few hundred lines to cover 99% of useful xml) using a Collectable stream based approach. BTW, :xmerl has that – fo…

We had a similar use case the difference being we need to generate xlsx files (which is just a zip file with multiple xml files). Apparently neither google nor hex.pm seems to show this library for xml stream https://github.com/activesphere/xml_stream

Elixir Programming Language Forum

Streaming XML to file

Questions & Help Questions

bettio March 5, 2022, 11:26pm 16

It works fairly well for us