ExFuzz is a coverage-guided fuzzer for Elixir, and I just launched v0.1.0!
defmodule JasonTest do
use ExUnit.Case
use ExFuzz
@tag timeout: 100
fuzz "Jason never crashes", data <- binary(), seeds: ["{}", "[1]"] do
with {:ok, term} <- Jason.decode(data) do
assert Jason.decode!(Jason.encode!(term)) == term
end
end
end
ExFuzz is well-integrated with ExUnit, so your fuzz tests can run alongside your regular tests and use the seed corpus as an ever-growing regression suite. It is built on top of LibAFL as the underlying mutation engine.
Given the amount of critical software infrastructure built on Elixir, I really wanted to focus on an out-of-the-box Go-like developer experience to reduce adoption friction (would really appreciate feedback here).
I have many plans for v0.2.0:
Typed generators.
More Elixir-aware coverage analysis.
Concurrent and continuous fuzzing.
Improved performance (execs per second).
Looking for early feedback from the community. I am planning to take it for a test drive around the ecosystem and will let you all know how that goes!
I spent some time with it last night. I set up a small project with a few bugs planted on purpose and pointed mix fuzz at them. It found most of them in a few seconds: a byte-by-byte clause chain, a nesting-depth crash, a raise inside a GenServer.call, a bug in a protocol impl, and an infinite loop. I was getting around 200k execs/sec on the simple targets and ~87k on the Jason example, which is a lot faster than I expected. Running it in a separate node was a smart call too. I made it blow the atom table with String.to_atom and it still reported the crash cleanly instead of taking everything down.
A few things I ran into:
The one that hurt most is literals in binary patterns. Since coverage is per clause, the fuzzer gets no signal until the whole pattern matches. <<?B, ?X, _::binary>> took around 500k execs, and <<"MAGIC", _::binary>> never hit in 12M. Elixir parsers are full of this. You’re already walking the abstract code, so pulling the string/integer literals out of :bin_elements into LibAFL’s Tokens could help a lot (basically what AFL++ does with its autodictionary).
The corpus also grows really fast. After 60s on Jason the cache had 8,830 inputs and was still climbing. I think it’s because the raw counts go straight into MaxMapFeedback, so a probe hit 41 times instead of 40 counts as new. I replayed the corpus and simulated it: with AFL-style buckets (HitcountsMapObserver) you’d keep about 2,150 of them for the same coverage.
Small one: saved crash inputs aren’t shrunk. One I got was 19 bytes of junk when "FUZZ" was all it needed, so a quick minimize pass before saving would make the committed files a lot easier to read.
And an idea: since the atom-table crash left nothing to replay, checking atom count / ETS tables / leftover processes before and after each input might be a nice BEAM-only feature. Leaks like that are hard to catch any other way.
For typed generators, the Hypothesis approach could be worth a look, where generators read their random choices from the fuzzer’s byte buffer. That way the byte mutations and shrinking still work.
Happy to send a PR for the bucketing part if you want it.
Thank you so much for spending the time on this and for the excellent feedback. This is incredibly useful.
The one that hurt most is literals in binary patterns.
Agreed. It’s on the roadmap for 0.2.0 to add more probe types to cover many of Elixir’s inline semantics that simple clause coverage can’t give us.
The corpus also grows really fast.
Yeah​. On top of corpus size, this also creates a lot of noise for the engine, which can negatively affect fuzzing effectiveness given it’s now reacting to not-so-important things. LibAFL’s standard fuzzing configuration has bucketing implemented, so it already falls under the issue I linked above, but I would call bucketing a bug fix. Would appreciate the PR, and we can get it out under a patch release.
Small one: saved crash inputs aren’t shrunk.
This is a nice suggestion. Just checked, Go and LibAFL have some prior art we can use as reference around this. I have created an issue for this, targeting 0.2.0.
Leaks like that are hard to catch any other way.
Great idea, kinda like LeakSanitizer. Will need some experimentation around this, and I think it can be really useful. Created the RFC for it. (tbh I’m kinda stoked to explore this!)
For typed generators, the Hypothesis approach could be worth a look