The issue has already been described accurately while I was working up an illustrative example, so this is probably unnecessary, but here’s a comparison that I think is more fair.
Name ips average deviation median 99th %
seq 16378.54 0.00006 s ±230.19% 0.00006 s 0.00009 s
par 70.85 0.0141 s ±11.70% 0.0138 s 0.0221 s
par_long 0.40 2.53 s ±0.00% 2.53 s 2.53 s
seq_long 0.0990 10.10 s ±0.00% 10.10 s 10.10 s
Comparison:
seq 16378.54
par 70.85 - 231.19x slower +0.0141 s
par_long 0.40 - 41355.92x slower +2.52 s
seq_long 0.0990 - 165422.20x slower +10.10 s
defmodule SeqVsPar do
def seq(n), do: 1..n |> Stream.map(fn i -> i + 1 end) |> Enum.to_list()
def par(n), do: 1..n |> Task.async_stream(fn i -> i + 1 end) |> Enum.to_list()
def seq_long(n),
do:
1..n
|> Stream.map(fn i ->
Process.sleep(100)
i + 1
end)
|> Enum.to_list()
def par_long(n),
do:
1..n
|> Task.async_stream(fn i ->
Process.sleep(100)
i + 1
end)
|> Enum.to_list()
end
Benchee.run(
%{
"seq" => fn -> SeqVsPar.seq(1_000) end,
"par" => fn -> SeqVsPar.par(1_000) end,
"seq_long" => fn -> SeqVsPar.seq_long(100) end,
"par_long" => fn -> SeqVsPar.par_long(100) end
},
parallel: 3
)
In this example seq uses Stream rather than Enum to make the comparison with par more accurately reflect the parallelization step rather than the sequential list creation step. You see that for extremely short calculations the parallelization does take longer because the calculation is shorter than the time of process generation and closure. When you make the calculation take longer by adding the sleep the cost of process generation and closure is now less than that of the calculation step.
Just for grins, below you can see the parallelization effect on my CPU as benchee runs the tasks with :parallel set to the number of available cores. On the left is running par, then par_long then the sequential variants tail after. This shows that the parallelization does actually occur.
