Glazer - the Swiss Army knife of JSON / YAML / CSV parsing

News Announcing
, , , ,
1

glazer started as a blazing-fast NIF JSON codec (single-pass, straight to/from native terms - no AST). In 0.3.0, the same engine now parses and encodes YAML and CSV too.

The numbers:

  • JSON: faster encoding than everything I tried - jason, thoas, euneus, torque, jiffy, simdjsone, OTP json- and neck-and-neck with torque (Rust sonic-rs) on decoding

  • YAML: 10x faster than yaml_rustler/fast_yaml, up to 100x faster than yamerl/ymlr

  • CSV: 2-20x faster than nimble_csv, and csv/erl_csv don’t even finish on large files (timeout) while glazer is done

Plus: streaming decode for JSON/CSV, bignums support, configurable null, zero external C++ deps.

def deps do
  [
    {:glazer, "~> 0.3"}
  ]
end

Charts and full benchmarks: https://github.com/saleyn/glazer#performance
Hex: https://hex.pm/packages/glazer

Break it, measure it, tell me what’s slow.

Best,
Serge

16 Likes
2

Will be giving this a try. Thanks!

3

I made many improvements and added options in the last release. Boosted decoding performance by another 20%. Check it out.

2 Likes
4

You may want to benchmark your CSV parser against GitHub - jeffhuen/RustyCSV: High-performance CSV parsing for Elixir. Rust NIF with SIMD acceleration, parallel parsing, and bounded-memory streaming. Drop-in NimbleCSV replacement. · GitHub . It was pretty fast for me.

1 Like
5

I’ll include the stats, though at the first glance rustry_csv and yaml_rustler have incompatible rustler version constraints, which make it impossible to use both dependencies in one build. A rather significant maintenance challenge when you want to include multiple incompatible dependencies in your project.

6

Added the benchmark for rusty_csv. Generally speaking the decoding difference is not very significant and within the margin of error, and rusty_csv doesn’t support encoding. The advantage is that glazer doesn’t have dependencies, which has an edge in cases where you might have conflicting Rust versions, like the issue I described above.

7

The dependency concern is completely nullified if the maintainer wires up rustler_precompiled so the users of the library don’t have to have the Rust toolchain installed.

8

yaml_rustler pins rustler ~> 0.34.0 (and rustler_precompiled ~> 0.8.2), while rusty_csv pins rustler ~> 0.37.3 (and rustler_precompiled ~> 0.9). torque is flexible (rustler >= 0.0.0, rustler_precompiled ~> 0.8) so it can coexist with either. The actual incompatibility is between yaml_rustler and rusty_csv specifically - both pin a specific rustler minor version that’s mutually exclusive, since Hex resolves one global version of rustler per mix.lock.

After some trial and errors I got this deps config working without having having 3 rustler libraries fighting with each other:

      {:rustler,             "~> 0.37.3", only: :bench, override: true, runtime: false},
      {:rustler_precompiled, "~> 0.9",    only: :bench, override: true},
9

For those interested how Erlang/Elixir glazer library performance stacks up against leading Golang JSON parsers, I put together this benchmark: GitHub - saleyn/glazer: Fast JSON/YAML/CSV encoder/decoder for Erlang and Elixir · GitHub. Pretty neat!

1 Like
10

Announcing the first stable release 1.0.0 of glazer battle-tested in production under continuous 1M req/s rate for two weeks handling JSON 2k request payload without errors.

1 Like