Phoenix.Pubsub subscriptions performance

[chrismccord]

@blquinn

Reading a bit of source code shows me that every time that you “subscribe” (add a proc to the process group) it does a global transaction around the name of the group. I believe that this has to contact all nodes and acquire some type of dist. lock. This seems like it would be tough to scale…

We don’t do this, which you are right would be a huge bottleneck

Each node only has 1…N PubSub shards, where N is no more than a handful, but we default to 1. Only these shards join the global pg2 group, and they exist to broker broadcasts across the cluster, and relay those messages to their local node subscribers. So the only pg2 action that happens is when these processes start up and join the group, or shutdown and leave, which is infrequent. The real hot-code paths happen at broadcast level, but this is all message sending. When local process A broadcasts a message on Node A, the following happens;

• the broadcast is sent via send to local-node subscribers on that topic, which is pull from ets
• a single message is sent to the pg2 group telling the remote nodes to replay this broadcast to local subscribers
• the message is picked up by the remote nodes and then send is called for every local subscriber, again pulled from ets

@blquinn

Have people load tested pubsub on subscription creations/deletions and proven it is production worthy, if so where can I see that test? Or, are there examples of metrics from a production system somewhere?

Yes, we did this with our 2M channel client load tests. The arrival rate was 10k connections/second, which means we achieved 10k subscriptions/second on the box, which included all the HTTP code in the mix, in addition to the pubsub code paths.

The Road to 2 Million Websocket Connections in Phoenix - Phoenix Blog

@blquinn

What happens generally to :pg2 and pubsub during a net split?

Messages broadcast during the split won’t be delivered to the remote nodes, but otherwise everything remains available. Phoenix.PubSub has no durable pubsub adapter today.

@blquinn

What is the main reasoning behind using :pg2 in the first place? (stdlib?)

You guessed it It’s built in, and critically, the small load we place on pg2 makes any high-load performance a non-issue. Again only a single process (or a few) join the pg2 group per node.

Hope that helps!