Presentation: Scaling Push Messaging for Millions of Devices @Netflix

Yes, Zuul is Netflix’ API gateway (all of the Netflix HTTP API traffic passed through Zuu). We took the Zuul code base and grafted Zuul Push on it, so it's a different offering. But they share 90% of the same code base.

With Zull Push, your client would connect directly to the Zull Push Cluster (which is in Amazon’s cloud) and use WebSockets to send things like movie recommendations to the client.

There are many areas where push notifications from Zull Push are handy. I will show an example of List of List of Movies (what we call LoLoMo). LoLoMo is a grid of shows and movies that Netflix shows you up when you log in.

The grids is actually a personalized to customer’s taste. It is not a static grade. For each member, we try to show movies and shows which are most likely to interest you. The typical experience on Netflix (at least for me) is once you log in, you keep on browsing at least for a few minutes (maybe for tens of minutes) to pick the best show to watch. During that time, there is a possibility that while you are browsing, the server would come up with better show recommendations based on personalized session data.

Now, once you know user’s test,  you get the best engagement if there are better recommendations generated in the cloud that you show them to the user. So if there is a better recommendation generated, you would want to let the client know sooner rather than later. That’s what push helps us with.

In our previous system, our application would poll for new recommendations.  Which worked, but it's wasteful and had some limitations such as freshness of the UI being limited to the polling intervals. With freshness, if intervals are too low, you are straining server’s capacity. If intervals are too high, you compromise UI freshness. So push is a really good way to address this. If new recommendations are available and the client is online, it just gets new recommendations via a push message without having to poll the server periodically.

SSE actually existed before WebSockets were widely supported. WebSockets are duplex. That means both the server and the client can send messages to each other. SSE is a half-duplex connection. Using SSE, only server can send messages to the client.  To initiate SSE, client sends a simple HTTP GET request indicating it can handle SSE event stream (Accept: text/event-stream). The web server then responds in a manner that keeps the connection open even after initial 200 OK response is returned. After this initial exchange, the server can send chunks of  JSON down the stream whenever needed in future.

Every time server sends a chunk down, an event fires on a browser saying you got a new chunk.  In our case, the half-duplex connection is fine because we don’t really have a use case where a client needs to send a message to the server.

SSE is not as widely supported in browsers as WebSockets. It is not supported by IE (Internet Explorer) which is the biggest gap, but Chrome and Safari do support SSE.

The biggest challenge was WebSockets are persistent and long-lived connections. That automatically makes your service more like a stateful service than a stateless REST service. Which means you can’t do quick deploys or a quick rollback. Really that was the biggest challenge to working with WebSockets.

Another issue we had to tackle was how to support large number of open connections efficiently on a single box. This is the classic C10K problem. while handling high volume RPS or requests/responses per second is a well-understood problem there is no continuou RPS with the push cluster. You make a connection and then wait for the server to send you something. Which means CPU and memory utilization is low but you are getting a really high number of mostly idle connections. So being able to support a very high number of open connections per box was a major problem. We’ll talk about this and some of our strategies for dealing with things like thundering herds in the talk.

These were the two major challenges.