Hello. I have benchmarked the JSON module ( JSON — Elixir v1.20.4 ) against a few libraries, including Erlang’s :jiffy. I’ve noticed a pattern that I find a bit hard to explain/understand and would like to hear your thoughts on it. Basically, JSON is ~2 times faster than :jiffy when encoding but :jiffy is ~2 times faster in decoding. This makes it quite hard to choose from one library to the other, given that one usually needs both encoding and decoding functionalities.
This is the benchmarking setup I had:
defmodule Benchmark.JsonLibrariesComparison do
def run(_) do
experiment()
end
def payload(size_kb \\ 1) do
:crypto.strong_rand_bytes(size_kb * 1024)
|> Base.encode64()
|> binary_part(0, size_kb * 1024)
end
def experiment() do
small_payload = payload(30)
medium_payload = payload(130)
large_payload = payload(230)
encode_inputs = %{
"small (30kb)" => small_payload,
"medium (130kb)" => medium_payload,
"large (230kb)" => large_payload
}
decode_inputs =
Map.new(encode_inputs, fn {label, term} ->
encoded = JSON.encode!(term)
{label, encoded}
end)
Benchee.run(
%{
"native encode" => fn binary -> JSON.encode!(binary) end,
"jiffy encode" => fn binary ->
IO.iodata_to_binary(:jiffy.encode(binary))
end
},
inputs: encode_inputs,
warmup: 3,
time: 10,
memory_time: 2,
formatters: [
{Benchee.Formatters.Console, extended_statistics: true}
]
)
Benchee.run(
%{
"native decode" => fn binary -> JSON.decode(binary) end,
"jiffy decode" => fn binary -> :jiffy.decode(binary, [:return_maps]) end
},
inputs: decode_inputs,
warmup: 3,
time: 10,
memory_time: 2,
formatters: [
{Benchee.Formatters.Console, extended_statistics: true}
]
)
end
end
And these are the benchmarking results:
Operating System: Linux
CPU Information: AMD Ryzen 5 3550H with Radeon Vega Mobile Gfx
Number of Available Cores: 8
Available memory: 13.59 GB
Elixir 1.19.5
Erlang 28.3.1
JIT enabled: true
Benchmark suite executing with the following configuration:
warmup: 3 s
time: 10 s
memory time: 2 s
reduction time: 0 ns
parallel: 1
inputs: large (230kb), medium (130kb), small (30kb)
Estimated total run time: 1 min 30 s
Excluding outliers: false
Benchmarking jiffy encode with input large (230kb) ...
Benchmarking jiffy encode with input medium (130kb) ...
Benchmarking jiffy encode with input small (30kb) ...
Benchmarking native encode with input large (230kb) ...
Benchmarking native encode with input medium (130kb) ...
Benchmarking native encode with input small (30kb) ...
Calculating statistics...
Formatting results...
With input large (230kb)
Name ips average deviation median 99th %
native encode 2.29 K 437.46 μs ±10.12% 429.31 μs 672.61 μs
jiffy encode 1.42 K 703.19 μs ±13.24% 676.83 μs 1129.40 μs
Comparison:
native encode 2.29 K
jiffy encode 1.42 K - 1.61x slower +265.73 μs
Extended statistics:
Name minimum maximum sample size mode
native encode 383.78 μs 1270.27 μs 22.66 K 419.61 μs
jiffy encode 640.02 μs 3358.87 μs 14.13 K 673.90 μs
Memory usage statistics:
Name Memory usage
native encode 0.133 KB
jiffy encode 3.40 KB - 25.59x memory usage +3.27 KB
All measurements for memory usage were the same
With input medium (130kb)
Name ips average deviation median 99th %
native encode 4.01 K 249.48 μs ±15.08% 240.11 μs 426.44 μs
jiffy encode 2.45 K 407.65 μs ±18.41% 389.29 μs 704.10 μs
Comparison:
native encode 4.01 K
jiffy encode 2.45 K - 1.63x slower +158.17 μs
Extended statistics:
Name minimum maximum sample size mode
native encode 215.32 μs 1215.31 μs 39.50 K 238.58 μs
jiffy encode 365.83 μs 3238.25 μs 24.28 K 375.40 μs
Memory usage statistics:
Name Memory usage
native encode 0.133 KB
jiffy encode 2.62 KB - 19.71x memory usage +2.48 KB
All measurements for memory usage were the same
With input small (30kb)
Name ips average deviation median 99th %
native encode 17.03 K 58.74 μs ±24.40% 55.66 μs 115.45 μs
jiffy encode 10.35 K 96.66 μs ±19.96% 90.58 μs 156.51 μs
Comparison:
native encode 17.03 K
jiffy encode 10.35 K - 1.65x slower +37.92 μs
Extended statistics:
Name minimum maximum sample size mode
native encode 46.16 μs 2409.24 μs 160.32 K 55.31 μs
jiffy encode 83.32 μs 2911.81 μs 99.80 K 90.09 μs
Memory usage statistics:
Name Memory usage
native encode 0.133 KB
jiffy encode 1.54 KB - 11.59x memory usage +1.41 KB
All measurements for memory usage were the same
Operating System: Linux
CPU Information: AMD Ryzen 5 3550H with Radeon Vega Mobile Gfx
Number of Available Cores: 8
Available memory: 13.59 GB
Elixir 1.19.5
Erlang 28.3.1
JIT enabled: true
Benchmark suite executing with the following configuration:
warmup: 3 s
time: 10 s
memory time: 2 s
reduction time: 0 ns
parallel: 1
inputs: large (230kb), medium (130kb), small (30kb)
Estimated total run time: 1 min 30 s
Excluding outliers: false
Benchmarking jiffy decode with input large (230kb) ...
Benchmarking jiffy decode with input medium (130kb) ...
Benchmarking jiffy decode with input small (30kb) ...
Benchmarking native decode with input large (230kb) ...
Benchmarking native decode with input medium (130kb) ...
Benchmarking native decode with input small (30kb) ...
Calculating statistics...
Formatting results...
With input large (230kb)
Name ips average deviation median 99th %
jiffy decode 4.38 K 228.56 μs ±14.40% 219.09 μs 385.41 μs
native decode 2.39 K 418.64 μs ±4.11% 416.74 μs 440.77 μs
Comparison:
jiffy decode 4.38 K
native decode 2.39 K - 1.83x slower +190.08 μs
Extended statistics:
Name minimum maximum sample size mode
jiffy decode 212.25 μs 1402.06 μs 42.96 K 218.95 μs
native decode 376.86 μs 2107.66 μs 23.70 K 413.60 μs
Memory usage statistics:
Name Memory usage
jiffy decode 64 B
native decode 328 B - 5.13x memory usage +264 B
All measurements for memory usage were the same
With input medium (130kb)
Name ips average deviation median 99th %
jiffy decode 7.80 K 128.29 μs ±5.77% 126.48 μs 144.29 μs
native decode 4.16 K 240.63 μs ±12.97% 234.46 μs 384.16 μs
Comparison:
jiffy decode 7.80 K
native decode 4.16 K - 1.88x slower +112.35 μs
Extended statistics:
Name minimum maximum sample size mode
jiffy decode 119.99 μs 1095.60 μs 75.82 K 126.41 μs
native decode 211.90 μs 943.48 μs 40.94 K 233.76 μs
Memory usage statistics:
Name Memory usage
jiffy decode 64 B
native decode 328 B - 5.13x memory usage +264 B
All measurements for memory usage were the same
With input small (30kb)
Name ips average deviation median 99th %
jiffy decode 27.23 K 36.72 μs ±42.34% 36.67 μs 47.14 μs
native decode 17.56 K 56.93 μs ±19.30% 56.01 μs 70.89 μs
Comparison:
jiffy decode 27.23 K
native decode 17.56 K - 1.55x slower +20.21 μs
Extended statistics:
Name minimum maximum sample size mode
jiffy decode 27.17 μs 3489.05 μs 249.05 K 36.67 μs
native decode 47.63 μs 2833.73 μs 165.92 K 55.80 μs
Memory usage statistics:
Name Memory usage
jiffy decode 64 B
native decode 328 B - 5.13x memory usage +264 B
All measurements for memory usage were the same
Any thoughts much appreciated!
