The way you were using EEx indeed does not result in very good performance: It re-compiles the EEx template on every benchmarking iteration.
I’ve sent you a PR which contains an implementation using EEx’s function_from_string to compile the EEx template once and then use it during the benchmark run.
Running the benchmark on my computer shows the following results:
$ mix run benchmarks/string_templating.ex
Compiling 1 file (.ex)
"ALL THE FOLLOWING SHOULD BE TRUE"
true
true
true
true
Operating System: Linux
CPU Information: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
Number of Available Cores: 8
Available memory: 7.60 GB
Elixir 1.10.2
Erlang 22.3.4.1
Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 35 s
Benchmarking base...
Benchmarking eex...
Benchmarking eex_compiled...
Benchmarking eex_eval_string...
Benchmarking manual...
Name ips average deviation median 99th %
eex_compiled 758.61 1.32 ms ±20.23% 1.27 ms 2.41 ms
manual 125.65 7.96 ms ±14.40% 7.89 ms 11.80 ms
base 17.07 58.57 ms ±5.70% 58.51 ms 65.98 ms
eex 2.22 450.36 ms ±5.85% 442.40 ms 500.30 ms
eex_eval_string 1.42 704.79 ms ±9.19% 714.84 ms 787.16 ms
Comparison:
eex_compiled 758.61
manual 125.65 - 6.04x slower +6.64 ms
base 17.07 - 44.43x slower +57.25 ms
eex 2.22 - 341.65x slower +449.04 ms
eex_eval_string 1.42 - 534.66x slower +703.47 ms
warning: redefining module EexExampleModule (current version defined in memory)
lib/templating_benchmarks.ex:27
Operating System: Linux
CPU Information: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
Number of Available Cores: 8
Available memory: 7.60 GB
Elixir 1.10.2
Erlang 22.3.4.1
Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 35 s
Benchmarking base...
Benchmarking eex...
Benchmarking eex_compiled...
Benchmarking eex_eval_string...
Benchmarking manual...
Name ips average deviation median 99th %
eex_compiled 5.23 0.191 s ±41.72% 0.175 s 0.47 s
manual 2.43 0.41 s ±7.35% 0.41 s 0.47 s
base 0.199 5.02 s ±0.00% 5.02 s 5.02 s
eex 0.0213 47.00 s ±0.00% 47.00 s 47.00 s
eex_eval_string 0.0142 70.61 s ±0.00% 70.61 s 70.61 s
Comparison:
eex_compiled 5.23
manual 2.43 - 2.15x slower +0.22 s
base 0.199 - 26.24x slower +4.83 s
eex 0.0213 - 245.76x slower +46.81 s
eex_eval_string 0.0142 - 369.18x slower +70.42 s
