WebSockets at Scale: What Breaks First and How to Fix It

WebSockets at scale become difficult when the number of open connections, message rate, fan out, reconnects, and failure recovery begin interacting. A server can show low CPU while connection buffers grow. A cluster can have spare memory while a proxy closes idle tunnels. A healthy node can still become the wrong place to send a message after a client reconnects somewhere else.

There is no universal failure order. The first limit depends on the workload, operating system, runtime, proxy path, message shape, client behavior, and how much state each connection carries. The useful engineering question is which resource or coordination mechanism is losing headroom first and what happens when it does.

This guide follows the full operating path, from protocol behavior and connection budgets through proxies, fan out, backpressure, routing, recovery, Kubernetes, autoscaling, observability, and load testing. Each layer gets its own failure model and measurements.

Why WebSockets Scale Differently From Request Traffic

A WebSocket begins with an HTTP upgrade and then becomes a persistent bidirectional connection. The connection can stay open for minutes or hours, which means the system carries state over time instead of handling one short request and releasing the resources immediately.

The WebSocket protocol specification defines control frames for close, ping, and pong, and allows message fragmentation. Ping frames can be used as keepalives or to check whether the remote endpoint still responds. Those protocol features become operational tools when thousands of connections need lifecycle management.

Long lived connections change the capacity model across the application, operating system, and proxy path. The main recurring costs include the following.

• each connection consumes a file descriptor or equivalent socket resource
• the runtime and application keep per connection state
• kernel, TLS, proxy, and application buffers consume memory
• load balancers and reverse proxies hold corresponding connections
• heartbeats, presence, and subscriptions create background traffic even when users are idle
• a deploy or node failure can cause many clients to reconnect at the same time

For a request service, requests per second may describe demand well. A WebSocket service needs at least two dimensions: concurrent connections and message work across those connections.

Build a Connection Budget Before You Pick a Scaling Strategy

Start with measurements from the actual server process. Open a known number of representative connections, let them reach steady state, then measure memory, file descriptors, CPU, network traffic, and any application state that grows with the connection count. Repeat with idle clients and with realistic message traffic.

• Connections: measure concurrent open sockets per process and per node. This defines the base concurrency the host and proxy path must carry.
• File descriptors: watch open descriptors alongside the operating system limit. A process can hit a descriptor ceiling while CPU and memory still look healthy.
• Memory: compare baseline process memory with the incremental memory added by connections and buffers. This exposes the cost of connection state and slow consumers.
• Inbound messages: measure messages and bytes per second by class so parsing, validation, authentication, and application work are visible.
• Outbound messages: track messages and bytes per second together with fan out. Broadcast patterns can dominate CPU and network cost.
• Send queues: monitor queued bytes or messages per connection. This shows clients receiving data more slowly than the server produces it.
• Reconnects: measure connection attempts and successful reconnects per second to understand churn and storm behavior during failures or deploys.

A rough memory estimate can start with baseline process memory plus measured incremental memory per connection, then add enough headroom for traffic bursts and the largest expected send queues. The measured value is more useful than a generic number from another stack because TLS, runtime objects, compression, subscriptions, and buffer policy can change the result substantially.

Connection limits also exist outside the application. Envoy’s current overload manager documentation exposes global and listener level connection limits and explicitly treats memory, CPU, and file descriptors as overload resources. The edge and proxy tier need their own headroom.

The HTTP Upgrade Path Can Fail Before the WebSocket Code Does

Every connection passes through an HTTP handshake before application messages start. Reverse proxies need to preserve the protocol upgrade correctly, and intermediate timeouts need to match the intended idle behavior of the application.

NGINX’s current WebSocket proxying documentation explains that the Upgrade and Connection headers require explicit proxy handling because they are hop by hop headers. It also notes that the proxied connection is closed by default when no data arrives for 60 seconds unless the read timeout is changed or the upstream periodically sends WebSocket ping frames.

That makes “random disconnects after about a minute” a proxy configuration problem in some environments, not an application scaling limit. Similar timeout and tunnel settings exist in other load balancers and proxies, so the complete connection path should be documented from client to edge to WebSocket server.

Document the connection path from client to server so timeouts and failures can be traced to a specific hop. Record the items below for each proxy or load balancer in that path.

• upgrade support and required headers
• idle and maximum connection timeouts
• TLS termination location
• connection and rate limits
• health check behavior
• whether a draining backend can receive new handshakes
• what logs identify handshake rejection or tunnel closure

What Usually Breaks First Depends on the Traffic Shape

Several constraints commonly surface early, but their order changes by workload. A chat application with thousands of mostly idle users has a different pressure profile from a live market feed broadcasting frequent updates to every subscriber.

• New connections fail while existing ones stay healthy: start with descriptor, listener, proxy, or admission limits. Check open descriptors, accepted connections, and rejected handshakes.
• Memory grows during message bursts: inspect outbound buffering and per connection queues. Queued bytes by connection and slow consumer counts are the first useful signals.
• CPU spikes during room broadcasts: investigate serialization, encryption, copying, and fan out work. Measure messages sent, recipients per event, and bytes copied.
• Latency rises only for some rooms or users: look for hot partitions, uneven connection placement, or a slow dependency. Compare load by node and fan out by topic.
• Deploys trigger a traffic spike: suspect a reconnect storm. Check connection attempts per second and handshake errors.
• Messages disappear after reconnect: inspect whether the system relies on best effort delivery with no resume path. Compare the disconnect timeline with sequence gaps and replay state.

The first fix should match the measured constraint. Adding nodes does little for an idle timeout. A larger Redis instance does little for unbounded per socket send queues. Sticky routing does little for messages that need durable replay after a disconnect.

Slow Consumers Turn Buffers Into a Memory Problem

A WebSocket server can produce data faster than one client can receive it. If the application keeps queueing outbound messages for that connection, the queue becomes hidden memory consumption. A few slow clients can be harmless; thousands can exhaust a process.

Every connection needs an explicit backpressure policy. Decide how much data can wait, which messages can be coalesced or dropped, and when the connection should be closed. The policy should reflect the product semantics instead of treating every event as equally valuable.

• Latest state: replace older queued state with the newest value. Cursor position, presence, and a price snapshot are common examples.
• Disposable signal: drop it under pressure. Typing indicators and transient animation events usually fit here.
• Ordered important event: bound the queue and use replay or resync after disconnect. Chat messages and account events often need this treatment.
• Critical transaction: persist it outside the socket path and acknowledge it separately. Payments and durable workflow actions belong here.

Queue depth should be observable in bytes as well as message count because one large payload can matter more than many small events. Track the largest queues, how long messages wait, and which clients repeatedly trigger pressure.

Fan Out Is a Multiplication Problem

An inbound event can become hundreds or thousands of outbound writes. The cost therefore depends on recipients per event as much as inbound message rate. Serialization, encryption, copying, broker traffic, and network egress all scale with that multiplier.

Presence is a common example. Broadcasting every join, leave, heartbeat, or status change to every connected user creates work that grows rapidly with audience size. Scope presence to the audience that needs it, batch updates where the product allows it, and send state snapshots instead of long histories when only the current state matters.

Measure the distribution of fan out across rooms and topics, not one global average. A system where most events reach two recipients and one room reaches 50,000 has a hot audience problem that the global average hides.

Multiple WebSocket Nodes Need a Routing Model

Once clients are spread across several servers, each node knows about the connections it currently owns. A message intended for a specific user or room may originate on another node, so the system needs a way to find or broadcast to the node that owns the connection.

Several routing patterns can work, and the right one depends on whether the system needs broadcast, targeted delivery, deterministic ownership, or durable recovery.

• publish room or topic events to every interested WebSocket node
• maintain a directory that maps users, sessions, rooms, or connection IDs to nodes
• partition topics or users so routing ownership is deterministic
• use a durable broker when delivery needs to survive subscriber or node downtime

Redis Pub/Sub is useful for best effort broadcast across active subscribers. Redis’s current Pub/Sub documentation states that delivery is at most once: a subscriber that is offline when a message is published misses that message. Durable state or replay therefore belongs in another system when missed events matter.

A broker should match the message semantics. A presence update may be safe to lose because the client can receive a fresh snapshot. A durable chat message or account event may need an authoritative store plus sequence or cursor state so reconnecting clients can recover what they missed.

Connection Affinity Is Different From Durable Session State

After the WebSocket handshake succeeds, that TCP connection stays attached to one backend for its lifetime. The more difficult moment is reconnect. A new handshake can reach a different node, especially after a deploy, failure, scaling event, or load balancer rebalance.

Kubernetes Services support optional ClientIP session affinity, and load balancers may offer other persistence mechanisms. Affinity can reduce movement in some designs, while important session state still needs to survive a different backend. Clients change networks, nodes disappear, and a reconnect may legitimately land elsewhere.

Keep state tied only to the live connection small. Put durable user, room, authorization, and replay state in systems that survive a node change, or make it cheap to reconstruct from an authoritative source.

Reconnect Storms Need Admission Control

A node restart can disconnect thousands of clients at once. If every client reconnects immediately, the recovery creates a second spike: TLS handshakes, authentication, subscription rebuilds, database reads, presence writes, and broker subscriptions all arrive together.

Clients should reconnect with exponential backoff and jitter. Servers and edge systems should also limit connection attempts so a failing dependency or abusive client is less able to turn reconnect traffic into sustained overload.

Design the client experience around temporary disconnection. Show connection state when useful, pause actions that require a live channel, and restore subscriptions or missed state after reconnect. A connection protocol that assumes uninterrupted transport will surface edge cases as duplicate actions, missing events, or stale state.

Delivery Guarantees Belong Above the WebSocket Transport

WebSocket runs over TCP and preserves the ordering of frames that belong to a message on the live connection. Product guarantees such as “the user receives every chat message exactly once” require additional application logic. Disconnects create uncertainty about what the client received or processed before the transport disappeared.

A robust protocol can assign event IDs or monotonically increasing sequence values, let the client acknowledge an appropriate cursor, and allow the server to replay from durable history or send a fresh state snapshot. The design depends on how costly duplicates and gaps are.

• Current state only: reconnect and fetch a fresh snapshot.
• Best effort live signal: send while connected and accept that some events may be missed.
• Ordered recoverable events: use sequence IDs plus replay from an authoritative log or store.
• Idempotent command: use a client generated request ID and server side deduplication.
• Critical side effect: persist and validate it through the application transaction path, then notify over the socket.

This separation keeps the socket layer honest. Presence, chat, notifications, collaborative state, and transactions can share one transport while using different durability rules.

A Scalable Architecture Can Stay Simple

A production WebSocket system needs clear boundaries more than a long list of technologies. One practical architecture has an edge or load balancer handling TLS and admission, stateless or lightly stateful WebSocket nodes holding live connections, an authoritative data store for durable state, and a routing or broker layer only where cross node delivery requires it.

CMX publicly describes a distributed WebSocket layer as part of its platform architecture. The implementation sits beside gRPC and REST service communication, with each protocol handling a different part of the platform. Our bare metal Kubernetes and Cilium article covers the cluster and network side, while this page focuses on the persistent connection layer.

A single node can still be the correct starting point. Add distribution when connection capacity, availability goals, deploy requirements, or team boundaries make it necessary. Introduce a broker when cross node messaging creates a real routing problem. Introduce durable replay when the product semantics require recovery.

Kubernetes Changes Deployment and Scaling Behavior

Kubernetes can run WebSocket workloads well, but persistent connections change what a healthy rollout looks like. A pod may stop receiving new Service traffic while hundreds or thousands of existing connections still terminate inside that process. The application needs a drain strategy that fits the connection lifetime.

Kubernetes provides a graceful pod termination flow. The current Pod lifecycle documentation describes the termination grace period, optional preStop hook, and termination signal sent before a process is forcefully stopped. That window gives the WebSocket server time to stop accepting new work and handle existing connections deliberately.

A practical drain sequence gives existing clients time to move without sending the entire connection population back through the handshake path at once.

1. mark the pod unavailable for new WebSocket handshakes
2. stop accepting new connections at the application or proxy layer
3. keep existing connections alive for a bounded drain period, or send a controlled close that tells clients to reconnect
4. allow clients to reconnect with backoff and restore subscriptions or missed state
5. terminate the process when the drain window ends

The right drain time depends on the product. Waiting for every connection to close naturally can make deployments take hours, while closing everything immediately can create a reconnect storm. Measure the reconnect load and choose a bounded policy.

Autoscale on the Pressure That Actually Limits the Socket Tier

CPU can be a weak scaling signal for a service carrying many mostly idle connections. A node may run out of connection headroom or memory while CPU remains modest. Another workload may be CPU bound because every connection receives frequent serialized updates.

Kubernetes Horizontal Pod Autoscaling supports CPU and memory metrics plus custom or external metrics through the appropriate metrics APIs. A WebSocket deployment can therefore scale on signals such as connection count, queue depth, or another workload metric when the metrics pipeline exposes them.

The scaling signal should correlate with a resource that new pods can relieve. Adding replicas helps when connection slots or per node message work are the constraint. It may have little effect when a broker, database, upstream API, or global fan out path is already saturated.

Autoscaling also needs connection redistribution logic. Existing WebSockets stay on their current servers until they disconnect, so adding pods primarily changes where new connections land. Scale out can increase future headroom quickly while leaving an overloaded node busy until its connections churn or are drained.

Protect Existing Connections During Overload

Once a node approaches a hard resource limit, accepting every new connection can turn local pressure into a full process failure. Admission control lets the system preserve useful work by rejecting or deferring new handshakes before buffers, file descriptors, or memory are exhausted.

Envoy’s overload manager includes actions that can stop or reject incoming connections and supports explicit downstream connection limits. The exact mechanism varies by proxy, but the architectural idea is portable: define a pressure threshold and a controlled behavior before the node reaches an uncontrolled limit.

Connection rate limits and total connection limits solve different problems. A total limit protects steady state capacity. A rate limit protects the handshake path during reconnect storms, bot traffic, or client bugs that repeatedly open new sockets.

Observe the Connection Lifecycle and the Server

Host CPU and memory are useful signals, while most WebSocket incidents also need context about individual connections. The service should expose enough telemetry to reconstruct what happened to connections and messages across time.

• Connection lifecycle: open connections, accepts, rejects, handshake errors, connection age, and disconnect reason.
• Client health: ping or heartbeat round trip, missed heartbeats, and reconnect rate.
• Message flow: inbound and outbound messages, bytes, recipients per event, and serialization time.
• Backpressure: send queue bytes, queue age, slow consumer count, and dropped or coalesced events.
• Distribution: connections and message work by node, room, topic, or partition.
• Routing layer: broker latency, publish failures, subscription count, and durable backlog where applicable.
• Recovery: reconnect success, replay gaps, resync frequency, and failed resume attempts.

Record close codes and application disconnect reasons where the client and privacy model allow it. A timeout, authentication rejection, deploy drain, explicit logout, network loss, and overload rejection are different events even if the product surfaces all of them as “disconnected.”

Logs should carry enough connection or session identity to join handshake, subscription, routing, and disconnect events without turning raw identifiers with high cardinality into every metric label. Traces can help when a WebSocket message triggers downstream RPC, database, or queue work.

Load Test Failure Modes Alongside the Happy Path

A useful WebSocket test holds realistic connections open while generating representative message traffic. A benchmark that connects thousands of idle clients and sends nothing measures only one part of capacity. A benchmark that sends messages without realistic connection duration misses churn and baseline state.

Build test scenarios around the system’s actual operating risks so the test exercises connection state, message work, and recovery together.

• gradual connection ramp to steady state
• normal message traffic at expected concurrency
• largest realistic room or topic fan out
• slow consumers with bounded outbound queues
• reconnect storm after a node or proxy interruption
• rolling deployment with connection draining
• loss of one WebSocket node
• broker or shared state latency
• authentication or subscription dependency slowdown
• traffic above the admission threshold to verify graceful rejection

Measure success at the client as well as the server. Track connection success, time to establish, end to end message latency, gaps or duplicates for recoverable streams, reconnect time, and whether the system returns to a stable state after the fault ends.

A WebSocket Incident Triage Runbook

When a realtime system starts dropping connections or delaying messages, diagnose the failure from the connection edge inward. The sequence below separates handshake problems, node pressure, routing failures, and downstream application problems before a mitigation changes the evidence.

1. Separate new connection failures from established connection failures. Check handshake status codes, proxy errors, and accept or reject counters. If existing sockets remain healthy while new ones fail, start with admission limits, descriptors, proxy capacity, authentication, or the upgrade path.
2. Check connection distribution by node. Compare open connections, memory, file descriptors, and message work across the fleet. One hot node can fail while cluster averages look normal.
3. Inspect outbound queues. Look for queued bytes, oldest queued message, and slow consumer counts. Growing queues point toward client backpressure or outbound bandwidth pressure.
4. Compare inbound events with outbound fan out. A modest inbound rate can produce extreme outbound work when one event reaches a large audience. Identify hot rooms, topics, or partitions.
5. Check reconnect rate. A sudden rise after a deploy, proxy change, network event, or dependency failure can create enough handshake and state rebuild work to prolong the incident.
6. Inspect the routing or broker layer. Check publish failures, latency, subscriber health, partition ownership, and durable backlog where the design uses one. Confirm whether the problem is transport delay or missing routing state.
7. Check dependencies used during connection and subscription setup. Authentication, databases, caches, and authorization services can make reconnects fail even when the WebSocket process itself has spare capacity.
8. Mitigate the measured pressure. Options include limiting new handshakes, shedding disposable events, reducing fan out, draining a hot node, increasing consumer capacity, or failing over a dependency. Choose the action that relieves the observed constraint.
9. Verify recovery from the client side. Confirm new connections succeed, message latency returns to baseline, queues drain, replay or resync completes, and reconnect rates settle.
10. Preserve the incident evidence. Record the limiting resource, trigger, mitigation, and missing telemetry so the next test can reproduce the failure before the next production event does.

The runbook also works during load testing. Stop at the first step where the system diverges from the expected behavior, then improve that boundary before increasing concurrency again.

A Practical Evolution Path

Architecture should move when the measured problem changes. The progression below describes capabilities to add, not mandatory technologies.

1. Single node: keep connection state in process, use a simple room registry, and broadcast directly. Add connection limits, queue limits, heartbeats, and lifecycle metrics before concurrency grows.
2. Several nodes: let each node own its live sockets. Add cross node routing or broadcast and shared authorization or state only where the design needs it.
3. Recoverable messaging: keep live delivery on WebSocket while adding sequence state, replay storage, or snapshot recovery for important events.
4. High fan out: keep the protocol and state model stable while adding partitioning, topic ownership, batching or coalescing, and broker capacity work.
5. Large operational footprint: preserve the same client contract while adding automated draining, custom autoscaling signals, overload controls, and regular failure exercises.

Redis, Kafka, NATS, RabbitMQ, or another messaging system may fit one of those stages, but the product semantics should choose the tool. A broker introduced before the routing or durability problem exists can create more failure modes than it removes.

WebSocket Production Checklist

• measure per node connection headroom under realistic traffic
• verify file descriptor and proxy connection limits
• document every idle timeout between client and server
• use ping, heartbeat, or application traffic intentionally instead of accidentally relying on idle tunnels
• bound per connection outbound queues
• define which message classes can drop, coalesce, replay, or require durable storage
• rate limit connection attempts and test reconnect storms
• make authorization and subscription state reconstructable after node changes
• measure fan out distribution and hot rooms or topics
• test rolling deployments with the real drain procedure
• scale on a metric new capacity can actually relieve
• log disconnect reasons and observe replay or resync failures
• test node, proxy, broker, and dependency failures before production needs the recovery path

Frequently Asked Questions

What usually breaks first when WebSockets scale?

The first constraint varies by workload. Common early limits include file descriptors, connection or proxy limits, memory used by connection state and buffers, slow consumer queues, fan out work, reconnect spikes, and cross node routing. Measure the failing resource before changing the architecture.

How many WebSocket connections can one server handle?

There is no useful universal number. The answer depends on runtime and kernel settings, TLS, memory per connection, message frequency, payload size, fan out, application state, proxy limits, and the latency target. Load test the exact stack with the traffic pattern the product expects.

Do WebSockets need sticky sessions?

An established WebSocket already stays on the backend that accepted it. Affinity can influence where a new handshake lands, but reconnect and node failure still require the application to reconstruct or recover important state. Durable session design matters more than assuming the client will always return to one node.

Is Redis Pub/Sub enough for WebSocket messaging?

It can work well for best effort fan out among active subscribers. Redis documents Pub/Sub as at most once delivery, so an offline subscriber misses published messages. Applications that require replay should keep authoritative or durable event state elsewhere.

How should Kubernetes autoscale WebSocket servers?

Use a metric that represents the pressure new replicas can relieve. CPU may work for servers with heavy message processing; connection count, memory, queue pressure, or another custom metric may be more useful for other workloads. Test how existing connections drain or rebalance because new pods primarily receive new handshakes.

How do you deploy WebSocket servers without disconnecting everyone at once?

Remove a terminating instance from new connection routing, give existing sockets a bounded drain period, and reconnect clients gradually with backoff and state recovery. Kubernetes provides a graceful termination window, while the application defines how persistent connections use it.

Should important actions be sent only over WebSocket?

Use WebSocket as a transport where it fits, but make critical state changes durable and idempotent in the application layer. The client can receive confirmation over the socket while the authoritative transaction lives in storage that survives disconnects and reconnects.

Scale the Connection System You Actually Have

WebSocket scaling gets easier when connection capacity, message work, routing, and durability are measured separately. A persistent connection is only one part of the system. The difficult failures appear where transport meets buffers, proxies, shared state, deployments, and recovery.

Start with connection and message budgets, bound the queues, document the proxy path, and decide what happens after a disconnect. Add cross node routing, durable replay, autoscaling, and overload controls when the measured workload requires them.

For the broader infrastructure layer around those decisions, see the CMX guide to what breaks first when infrastructure scales. Teams that need help designing or stabilizing realtime infrastructure can also start with our services.