WebSockets at Scale: What Breaks First and How to Fix It

A live chat can handle thousands of people quite happily, then struggle after a short server restart. The surprising part is what happens when everyone tries to reconnect. Thousands of devices suddenly ask to sign in again and rejoin their conversations. A system that had plenty of room a minute earlier can become overwhelmed by the effort of getting back to normal.

To understand why, it helps to know what keeps a chat or live game connected. Many real time features use WebSockets, which let a device and a server keep a conversation open. They can send updates to each other immediately, without starting a new request every time something changes.

That open connection is useful, but it changes what happens when traffic grows. Some devices receive updates much more slowly than others. One message might need to reach an enormous audience. And when connections break, the return journey can be harder than keeping them open in the first place.

Why a WebSocket Stays Open

Most familiar website requests are short. Your browser asks for a page, the server sends a response and that exchange finishes. A live chat needs something different. New messages should appear as soon as they’re sent, even when you haven’t clicked anything.

A WebSocket connection provides a two way channel between a device and a server. After the initial connection is established, either side can send a message over it. The connection can stay open for a long time, which is also why the server has to keep track of it.

The WebSocket specification describes small control messages used to check whether a connection is responding and to close it cleanly. Those signals help manage the channel, but they don’t save chat history or recover messages that arrived while you were disconnected. An application needs separate rules for that.

Now consider the scale. Ten thousand people reading quietly may require relatively little message processing. Ten thousand people receiving a constantly updating feed can create vastly more work even though the number of open connections is exactly the same.

Ten Thousand Connections Can Mean Very Different Things

When a service says it supports a million live connections, it’s worth asking what those connections were doing. Were the devices mostly idle, or were they receiving frequent updates? A server may be able to keep a great many quiet conversations open while struggling to deliver a busy feed to the same audience.

Each connection uses some computer memory and a limited operating system resource called a file descriptor. That is simply how the operating system keeps track of an open connection. The server may also hold a list of which conversations the device has joined. When messages start arriving, it has extra work to do.

Engineers therefore test more than the number of connected devices. They look at how much memory each connection uses, how many messages travel through it and whether some recipients fall behind. A slow receiver can turn a quiet connection into a growing pile of unsent data.

A useful performance test includes people sending messages and others receiving them on slower connections. Keeping a million idle sockets open proves far less than it sounds like. For the broader question of how shared services fail under load, see what breaks first when infrastructure scales.

The Hidden Timeout Between You and the Server

A live connection may travel through several computers before it reaches the application. Some intermediaries close connections after a long period of inactivity. The chat server could be working perfectly while an unnoticed timeout elsewhere cuts off the conversation.

The first step is a connection setup process called the handshake. WebSockets begin as a web request that asks to switch to an ongoing connection. NGINX, a common web server and proxy, documents the special Upgrade and Connection headers used for that switch. Its WebSocket proxy guide also documents a default timeout of 60 seconds when the server sends no data through the connection.

That creates a useful clue. If a connection repeatedly ends after roughly the same period of inactivity, look for a timeout along the route. Applications can send periodic keepalive signals or adjust appropriate settings, but the cause needs to be confirmed before changing everything.

To the person using the app, all these failures may look the same. The chat disconnects. Behind the scenes, a failed handshake, a time limit and an overloaded server require quite different repairs.

What the Disconnect Icon Doesn’t Tell You

Different failures can produce the same spinning reconnect icon. Looking at how the problem began often tells engineers more than staring at one server’s CPU graph.

• New people can’t join, but everyone already inside is fine. Something may be blocking the first connection or the server may have reached its connection limit.
• Messages arrive later and later. The application may be producing updates faster than it can send them out.
• One very busy chat room slows down. Every message may have to be copied to a huge number of people.
• The app struggles after a restart. A crowd of disconnected devices may all be trying to return at once.
• Some messages disappear or repeat. The application needs to examine what it stored and what happened during reconnection.

Those symptoms point to different problems. Buying a bigger server may help with one of them and do little for another. The unusual part of live systems is how much the shape of the traffic matters.

What Happens When Someone Receives Messages Too Slowly?

In a live feed, updates might arrive every second while one device can display only a few before the next batch appears. The server must decide what to do with the waiting updates. If it keeps everything in memory, that waiting pile can grow and grow. Now multiply the problem across thousands of devices.

Engineers call the ability to manage that mismatch backpressure. It means accounting for the difference between how fast information arrives and how fast the receiver can handle it. MDN notes that the browser’s standard WebSocket API doesn’t provide built in backpressure, so fast incoming messages can also overwhelm the app on a user’s device.

The answer depends on what the information means. If a live map sends someone’s current position repeatedly, an older position may no longer matter once the latest one arrives. A chat message someone sent deliberately is different. People generally expect it to remain available after they reconnect.

Servers can limit how much unsent data they hold. When a device falls too far behind, the application may send it the newest state or reconnect it and retrieve the important messages from storage. The essential decision is which updates may safely be skipped.

How One Message Can Turn Into Thousands

One person sends a message to a large chat room. From their side, it looks like a single action. Behind the scenes, the system may have to deliver a copy to thousands of connected devices. That’s a great deal more work than accepting one incoming message.

This multiplication is called fan out. It also happens with live sports scores and multiplayer games. A single change can become a flood of updates to everyone watching. If most conversations are small but one event draws a huge crowd, the busiest group can put more pressure on the system than all the quiet groups combined.

Some updates don’t need to go everywhere. An app might send changes only to people watching that particular event or combine several tiny updates into one. A server handling an enormous audience also needs enough network capacity to send all those copies.

So a statistic like “100 messages per second” can be misleading. The more revealing question is how many people each message has to reach.

Why Some Messages Can’t Be Recovered After a Disconnect

A notification sent while your phone is offline might be waiting when you return. Whether it appears depends on the system behind the app. That depends on whether the system kept a copy somewhere or simply passed it along to whoever was connected at the time.

Some messaging services are designed for immediate delivery. Redis Pub/Sub, for example, can distribute an update to listening servers, but it doesn’t keep missed broadcasts for later. The Redis documentation describes this as at most once delivery. A receiver that wasn’t listening when a message was sent can’t ask ordinary Pub/Sub to replay it.

That is fine for a typing indicator. It would be a poor guarantee for an important chat message. Applications that promise message history need to save the messages in a database or another durable system. On reconnect, they can fetch what the device missed.

WebSockets are useful for delivering a message while you’re connected. Keeping a dependable history is a separate responsibility.

Why You Might Return to a Different Server

A large service usually runs on many computers. When you first open a chat, one of those computers accepts your connection and keeps it open. After you disconnect, the next connection may go to a different machine. That machine might know nothing about the conversation you were in.

Software called a load balancer decides which available server receives a new connection. Some load balancers try to send the same person back to the same machine, a feature often called session affinity or stickiness. Kubernetes includes a ClientIP affinity option. But if the original server has restarted or disappeared, there’s nowhere to return to.

That’s why the useful information about your conversation must survive outside that individual connection. The new server needs a way to verify your identity and find your subscriptions, while important messages remain in shared storage.

Adding more servers gives a service more capacity, but it also means engineers have to make sure your experience doesn’t depend on remembering which machine you visited first.

The Second Traffic Surge Nobody Asked For

A restart can end thousands of live connections in a moment. Every disconnected phone or browser then tries to reconnect. They need to open a fresh connection, prove who they are and work out which conversations or feeds they were following. All that happens while the replacement servers may still be starting.

This rush is called a WebSocket reconnect storm. It can create more work than the steady activity the service handled before the restart. Some reconnection attempts fail under the pressure, and impatient devices try again, adding another wave.

This is a real concern even for large networks. Cloudflare’s WebSocket documentation notes that server restarts during network updates can terminate active WebSocket connections. The application therefore needs a way to recover from disconnections it didn’t initiate.

One solution is to spread reconnect attempts over time. Instead of all devices immediately returning at once, each waits for a slightly different interval. Engineers call the increasing waits backoff, often combined with randomness known as jitter. The idea is simple. Don’t make everyone knock on the server’s door at exactly the same second.

A good recovery also makes it clear when a device has caught up. Reopening the connection is only the beginning if the app still needs to restore the user’s subscriptions or fetch missed messages.

The Mystery of a Message That Appears Twice

A chat message reaches the server and gets saved. Just before the sender receives confirmation, the connection breaks. Their screen may show an error even though the message is already there. They tap Send again and the same message appears twice.

The opposite problem is possible too. An app may never store an important message. If the connection breaks before delivery, opening a new connection won’t bring it back from nowhere. The live channel alone doesn’t provide a permanent record.

Reliable systems therefore distinguish different kinds of updates. A live cursor position can be replaced by the newest position. A chat message generally deserves to be saved. An action with consequences, such as sending a payment instruction, needs protection against doing the same thing twice.

One useful technique gives a message a stable request ID. If the sender retries after losing confirmation, the server can recognize that the same action has already been accepted. Engineers call this deduplication or, when repeating an operation has the same effect, idempotency.

WebSockets deliver information over an open connection. Deciding what survives an interruption and what may safely be repeated belongs to the application.

What Must a Live Chat Remember?

Before choosing complicated software, decide what the app promises its users. Should every message remain available? Can a missed typing indicator disappear? What should someone see when they return after losing internet access?

Those answers determine the design. A service needs computers that hold the live connections. When people are connected to different computers, it also needs a way to share updates between them. Important history needs storage that continues to exist even after an individual computer restarts.

CMX’s architecture includes a distributed WebSocket layer alongside other services. Our article about CMX’s Kubernetes and Cilium infrastructure explores the network environment in greater detail. This piece focuses on what those open connections mean for people using a live app.

What Happens When Kubernetes Replaces a Live Server?

Large applications often run inside systems that can replace a server without taking the whole website offline. Kubernetes is one such system. It manages running applications and can gradually replace older versions with new ones during an update.

There’s a catch for WebSockets. The computer being replaced may still be talking to hundreds or thousands of connected devices. Kubernetes knows the computer should stop, but the application has to decide how to close those conversations.

The Kubernetes pod lifecycle documentation explains how applications get time to shut down through termination grace periods and signals. Engineers can use that time to stop admitting new connections and close existing ones in a controlled way. In Kubernetes, the replaceable running unit is called a pod.

Shutdown also needs coordination with the network component directing visitors to servers. A pod can appear ready to stop before every intermediary has stopped routing new connections toward it. The application should allow enough time for those changes to take effect and help disconnected clients reconnect gradually.

Keeping every conversation open forever would prevent deployment. Closing everything instantly risks creating the reconnect storm we just explored. The practical answer lies in a tested shutdown process.

Why Adding Servers Doesn’t Move the People Already Connected

When a website is struggling, adding another server sounds sensible. But people already connected by WebSockets don’t automatically move to the new machine. Their open conversations stay with the server that accepted them.

The new machine can help by taking fresh connections. To relieve an overloaded server that’s already busy, the system may need to close and redistribute some existing connections carefully. That causes reconnections, so the timing matters.

Kubernetes can automatically add application copies, called pods, when certain measurements show that more capacity is needed. Its autoscaling documentation describes how engineers can choose those signals. Measuring processor use alone may miss a server that’s short on memory or holding too many live connections.

There’s also the problem of shared services. If every reconnect must check the same overloaded database, adding more chat servers can merely send more requests to that database. More computers help only when they add capacity where the system needs it.

Sometimes Saying ‘Try Again Later’ Is the Safer Choice

A live service can reach a point where taking one more connection threatens everyone already using it. The server may have too little memory left or have reached its limit on open connections. Accepting more work can push it into a crash.

Deliberately refusing some new connections can protect the people already online. That sounds unfriendly, but a clear temporary failure may be preferable to disconnecting everybody. The rejected devices can try again after a delay, provided those retries don’t create another wave of traffic.

Systems can place these limits in the software between visitors and the application. Envoy, for example, documents an overload manager that can stop accepting connections under configured resource pressure.

There are two limits worth separating. How many connections can stay open? How many new connections can safely arrive at once? A service with thousands of long conversations needs to understand both.

How Engineers Tell What Went Wrong

A server can look healthy while its chat is effectively broken. It may have plenty of processor capacity but be unable to send messages quickly enough. Or people can connect successfully while their old conversations never finish loading.

Engineers monitor the stages of a live connection so they can see where the problem begins.

• Connecting. How many devices try to join, and how many succeed?
• Staying connected. How often do connections close, and are particular servers losing more than others?
• Receiving messages. Are updates waiting in memory because devices or networks can’t keep up?
• Returning after an interruption. How long does it take people to reconnect and receive missed important messages?

Those measurements can reveal a pattern that’s invisible in the total number of people online. During an incident, engineers can use detailed logs to investigate one connection without exposing private messages in public metrics.

The Most Impressive Connection Test May Miss the Worst Failure

A demonstration showing a million open connections can be a real achievement. But there’s a question it may never answer. What happens when the server holding some of those connections suddenly disappears?

A convincing performance test gives devices meaningful activity and deliberately interrupts part of the system in a controlled environment. Some devices should be receiving slowly enough to reveal growing queues. Others should be in a crowded room where one message must be delivered to many people.

Then the team restarts a server and watches the return. Do the clients reconnect gradually? Does the login system keep up? Do people recover important messages without duplicates? A short interruption that requires several minutes of recovery is part of the failure, even if every server eventually reports healthy.

Testing a failure is often more revealing than measuring how many connections the system can keep open when everything is perfect.

How to investigate a WebSocket outage

If you’re operating a live service, the details behind the symptoms matter. A practical investigation should find the point where a connection or message stops making progress, then confirm that a proposed fix relieves the cause.

1. Check whether new connections fail. Review handshake responses, proxy errors and accept or reject counts. Existing connections working normally while new ones fail can point to connection limits, authentication or the upgrade route.
2. Compare servers individually. Look at open connections, available memory, file descriptors and message throughput. One busy node can struggle while fleet averages remain comfortable.
3. Look for slow receivers and large broadcasts. Examine outbound queue size and the oldest waiting messages. Compare incoming events with the number of outgoing deliveries.
4. Investigate the return after disconnection. Measure reconnect attempts, how long authentication takes and whether subscription state is restored. A deployment or proxy change may explain a sudden surge.
5. Check messages between servers. Inspect the broker or routing layer and distinguish delayed live delivery from missing durable messages. Where events must be replayed, verify storage and the client’s recovery position.
6. Verify the fix through a client. Confirm connections stay open, queues fall and important messages return as expected. Record the trigger so a controlled test can reproduce the failure later.

Some fixes involve rejecting new connections temporarily. Others need a change to message distribution or to the service that verifies logins. The observed bottleneck should decide the response.

When Is a More Complicated System Worth Building?

It’s possible to build a working chat service on one machine. That’s a reasonable starting point when its limits and failure behavior are understood. Problems become more interesting when the number of users or the amount of traffic grows beyond what one machine can handle.

Adding servers creates more space for connections. It also means a message arriving on one server may need to reach people connected elsewhere. A system may use Redis, NATS, RabbitMQ or Kafka for different kinds of communication. These tools don’t all offer the same recovery guarantees, so the choice depends on what the app promises people.

Important history usually needs storage separate from the live broadcast path. Once that is reliable, larger systems can make further decisions about distributing enormous audiences across servers.

There is no benefit in making the architecture complicated before the problem demands it. The useful question is which limitation an additional component will solve and what new responsibility comes with running it.

The Real Test Comes When Everyone Returns

A live chat feels simple when messages appear immediately. The difficult part comes after an interruption, when thousands of people want to return at once. A dependable service helps them reconnect without another surge, restores the conversations they were having and remembers the messages that matter. That recovery is what people notice when something goes wrong.