How to use GenServer in leetcode?

This line in subsequent calls to init_/1 won’t do what you think it will. There might be only one instance with the name RLEIterator running, meaning the second and all other calls would return {:error, {:already_started, pid}} tuple and next/1 will send a message to the very first instance started. That is one of the reasons you are better to directly pattern match on the return value of GenServer.start_link/3.

{:ok, _pid} = GenServer.start_link(__MODULE__, []} # , name: __MODULE__)

You likely need an agent, keeping the map %{encoding ⇒ pid} and a bunch of anonymous instances of RLEIterators.