Why does this simple GenServer timeout?

The Genserver is a single threaded process. If you sleep in the handle call, the second message sent will not start processing until the first sleep is finished, so 3000 + 3000 > 5000.