What Breaks First When Infrastructure Scales

In October 2021, Roblox went offline for 73 hours. Its engineers tried replacing crucial servers with more powerful machines. It didn’t fix the problem. At one point, the platform was struggling even with no players sending traffic to it.

There’s something fascinating about that. You’d expect a busy website to recover when the crowds go away, and you’d expect faster computers to help. But large systems can create problems for themselves. The programs that keep everything connected may become overloaded just trying to communicate with one another.

Roblox’s experience offers a way into a bigger question. What breaks first when infrastructure scales, and why are the answers often so surprising? We’ll explore why adding servers sometimes makes things worse, how software accidentally starts a flood of its own requests, and what it takes to bring a huge platform back online. You don’t need to know how to run a server to follow the story.

The Roblox Outage That Bigger Servers Couldn’t Fix

The engineering account Roblox published after its 2021 outage is unusually detailed. The trouble centered on Consul, software that helped Roblox’s internal services find and communicate with one another. You can think of it as part of the platform’s address book. When that shared system slowed down, services had difficulty finding the systems they depended on.

Engineers initially suspected faulty hardware or too much work for the existing machines. They replaced servers with more powerful ones, but the system remained unhealthy. One attempt to restart the shared infrastructure looked promising until internal services resumed using it. The load returned, and performance deteriorated again. The players weren’t the source of that traffic. The platform’s own services were.

The eventual diagnosis involved two difficult performance problems inside Consul’s implementation, made worse by the way many Roblox services depended on one cluster. Roblox reported that higher-core-count hardware may even have aggravated one form of contention. Turning off a newer streaming feature helped restore performance. Engineers also had to deal with a separate underlying storage issue.

Here’s what makes the Roblox incident worth studying. A seemingly small shared component can determine whether an enormous application keeps working. Bigger computers aren’t a cure when the software is spending its time waiting on an internal bottleneck. A server can be powered on while the work it exists to perform has barely moved.

Why Growth Can Make Change Dangerous

More users often mean more code and more moving parts. A growing website may rely on separate systems for signing in, delivering messages or processing data. That specialization can help with scaling, but it also means a routine software update has more opportunities to interrupt something else.

Engineers sometimes spot this change before users do. Deployments take longer because nobody is sure what depends on the component being updated. A fix that works on one machine needs extra checks elsewhere. Rolling back becomes difficult if the latest version also changed shared data.

The DORA research on software delivery looks at how frequently changes are released and what happens when they fail. As a system grows, making deployments repeatable and reversible becomes part of keeping it reliable. A release process nobody trusts can slow improvement even when the existing application handles today’s traffic.

A strange sign of growth, then, is that software can continue to serve today’s traffic while becoming harder to improve. More capacity doesn’t solve a release process everyone is afraid to touch.

How a Website Can Be Up and Still Be Broken

A website’s health check may ask its server for a simple response. If the response arrives, a monitoring screen turns green. That doesn’t prove a person can sign in, send a message or finish a complicated request. Those actions might depend on another service that is slowing down.

Engineers can follow an individual request through the system using a trace. If most of the time is spent waiting on a database, the trace helps reveal that. You don’t need to know the monitoring software to appreciate the difference between “the server responded” and “the service worked.”

Roblox’s outage exposed another twist. The company reported that some monitoring tools depended on the same infrastructure that was failing. That made diagnosis harder just when the engineers needed reliable information. It’s a memorable reason to question any dashboard that insists everything is fine while the experience on the other side is clearly broken.

Why More Servers Can Make a Website Slower

Adding machines sounds like the obvious response to a busy website. Sometimes it is. If the web servers are all waiting for the same database, the situation changes. Giving the website twice as many servers can mean twice as many callers competing for a database that hasn’t changed.

Databases also have limits on simultaneous connections. Applications commonly use connection pools to reuse connections and control how much work reaches the database. If all connections are occupied, new requests wait. Raising the limit can move the queue into the database itself, where the extra competition may make everything slower.

A database has limited resources, including how many connections it can manage comfortably. Some delays also come from inefficient queries or several operations competing to change the same data. Adding more web servers won’t repair those problems.

The counterintuitive idea is simple. A website doesn’t have one capacity number. It has several components, each with its own limit, and increasing one component may push harder against another. The real bottleneck is the part that can’t keep up.

When a Website Starts Attacking Itself With Retries

Software sometimes repeats requests before the earlier attempts have finished. It happens when one service takes too long to reply and another assumes the request failed. When one service takes too long to respond, another may assume the request failed and try it again.

Retries are helpful when the problem is brief. During overload, they can have the opposite effect. The original requests are still occupying resources, so each new attempt adds work to a system that’s already falling behind. Several layers of software may retry the same operation, multiplying the number of requests without a single extra visitor.

Google’s engineering chapter on cascading failures explains how this kind of feedback can help spread a failure. Amazon’s Builders’ Library guide describes a useful defense called backoff. Instead of retrying immediately, a caller waits longer after repeated failures. Adding a little randomness to the delay, known as jitter, prevents thousands of callers from trying again at exactly the same moment.

Some work shouldn’t be retried blindly at all. A purchase might have completed even though the confirmation never arrived. Trying again could create a duplicate unless the application can recognize that it’s the same intended operation.

Persistent connections have their own version of this problem. After a server restart, thousands of devices may reconnect together. Our WebSockets at Scale article explores why the return of disconnected clients can be harder to manage than the traffic that came before the interruption.

The Small Component Everyone Depends On

In a large application, an apparently minor internal service can be more important than a powerful public-facing server. Think back to Roblox. Its service-discovery system helped programs find one another. When that shared part stopped working properly, healthy services struggled to communicate, and starting new ones became difficult.

This is the danger of a shared dependency. An application may need a database or another service to finish a request. If that dependency slows down, the caller spends time waiting and may run out of resources to serve anything else. A problem in one component spreads through the systems waiting on it.

Good designs don’t assume every supporting service will always be available. A video site may be able to show a page without personalized suggestions, while a payment system must be more careful about reporting a transaction as complete. The application needs to know which work can continue and which must wait for a reliable answer.

The bigger lesson is surprisingly easy to miss. A huge platform can depend on a small internal service everyone has stopped noticing. If that service fails, otherwise healthy parts of the application may be unable to finish their work.

Why Autoscaling Isn’t a Magic Button

Autoscaling means adding or removing computing capacity automatically as demand changes. It sounds ideal. A busy service gets more machines, then reduces them when things calm down.

But an autoscaler has to decide *when* to add capacity. Many systems use processor activity as a clue. That’s useful when processors are the constraint. It misses a service that’s spending its time waiting on a slow database or an external network connection.

Kubernetes can scale applications based on processor use or other configured measurements. Its Horizontal Pod Autoscaler documentation explains how those signals work. Yet creating an extra application instance doesn’t make an overloaded shared database faster, and it doesn’t move work away from a broken outside provider.

That’s what makes the Roblox story memorable. More powerful hardware didn’t solve the underlying contention problem. Capacity matters, but it must be added in the part of the system that is genuinely running short of it.

Getting Back Online Can Be Harder Than Staying Online

A recovering platform faces a strange problem. Its caches may be empty, its services may need to rediscover one another, and disconnected users may all try to return. The moment engineers announce that the outage is over can bring a surge that pushes the system into trouble again.

Roblox didn’t reopen the platform to everyone at once after restoring its core services. Its engineering report describes using DNS steering, a way of directing visitors to different destinations, to let groups of players return gradually. Engineers watched the system after each increase. It was a careful recovery rather than a dramatic switch being flipped.

That approach reveals something important about capacity. A platform needs to survive ordinary use, but it also needs a safe route back from failure. A backup file is valuable only if it can be restored and the recovered system can operate under real traffic.

For data that matters, teams set targets for how quickly the service must return and how much recent information could be lost. Those targets guide recovery design, but they’re meaningful only after a real test. A system that ran flawlessly for months may still have a recovery problem nobody discovered until it went down.

What Engineers Look For Before a Site Crashes

One lesson from large outages is that “up” and “down” are crude descriptions. The system may be serving most requests while a growing number take far longer than usual. Background work can pile up unseen. Engineers look for those changes before a complete failure occurs.

The most telling clues aren’t necessarily the ones with the biggest numbers.

• Slow requests. Does the slowest slice of requests keep getting slower even when the average looks fine?
• Growing delays. Are queued tasks waiting longer than they used to?
• Work that shouldn’t exist. Are retries or reconnect attempts growing faster than real demand?
• Shared bottlenecks. Are unrelated parts of the application all waiting on the same service?

The trick is connecting a warning to what people experience. A graph showing a busy processor may not explain a failed sign-in. A record of which service the sign-in request was waiting for is much more useful.

Can You Test a Failure Before It Happens?

There’s a limit to what a successful speed test tells you. A website that handles a large number of requests under ideal conditions may behave very differently when one supporting service becomes slow. The interesting tests deliberately disturb the system while it is busy.

Engineers can introduce a controlled delay to a nonproduction dependency or stop a test worker while requests are running. Then they watch what happens. Does the application slow down gracefully? Does it keep trying the same failed request? Can the remaining parts continue to do useful work?

These experiments are sometimes discussed under the name chaos engineering. The idea is much less dramatic than the name. You learn how software behaves when a piece of the environment stops cooperating, and you do it where the test is authorized and the damage can be contained.

A good result isn’t merely that the system survived. It’s that engineers understand which parts failed and how recovery worked. The next real incident becomes less mysterious because those behaviors have already been observed.

Why More Complicated Technology Isn’t Always Progress

Some of the worst performance problems aren’t solved by adding new infrastructure. A slow database query can remain slow after a move to a larger cluster. A confusing retry policy can still overwhelm an expensive new platform. Complexity has costs of its own, especially when it makes failures harder to trace.

Operating the platform is a separate challenge from writing the application. Our DevOps and infrastructure ownership guide discusses the practical work required once a team maintains more of its own systems.

Of course, some workloads benefit from unusual architecture. CMX uses bare metal Kubernetes with Cilium and BGP to operate its infrastructure with direct control over networking. Our article about that architecture goes into the engineering details. The important question is whether a design solves a real constraint well enough to justify everything else the team must now understand.

There’s no prize for the most complicated diagram. A design earns its place when it makes the software more dependable or enables something the simpler version couldn’t do.

One Expensive Query Can Slow a Whole Platform

A database can be doing useful work that happens to interfere with other useful work. A detailed report might require reading millions of records while a live application needs quick answers from the same database. If the operations compete for the same resources, the report can slow down the application even though nobody has asked the website to do anything unusual.

This kind of problem becomes visible only when different activities run together. A query can perform well in isolation but cause trouble during a busy period. Engineers investigate where the database spends time, whether requests are waiting for a free connection and whether a different query plan or index would reduce the work.

Sometimes reporting can run against a separate read replica, a copy of the database used for queries that can tolerate slightly older information. That helps in the right circumstances, though keeping replicas synchronized adds another layer to operate.

The fascinating part isn’t the database terminology. It’s that a website can become slow because two perfectly legitimate jobs are competing for the same resource. Finding the source of that competition often matters more than buying additional application servers.

The Strange Art of Letting Less Work In

There’s a surprisingly sensible response to some overloads. Stop trying to do everything.

Systems can deliberately turn away new requests when accepting them would make things worse for everyone already waiting. They can postpone work that isn’t urgent while preserving capacity for the essential parts of an application. During recovery, they may bring users back gradually instead of allowing every disconnected client to return at once.

Roblox’s 2021 outage showed why this matters on a huge scale. Restoring its underlying service was only part of the job. It still had to bring an entire platform back into use without recreating the pressure that had helped destabilize it.

The real mystery of scaling isn’t how many servers a website can acquire. It’s how the pieces behave when something stops keeping up, and whether the software can recover without making its own problem worse. That’s the engineering worth understanding, whether a site serves a few thousand visitors or millions.

CMX works on these questions in the context of applications and infrastructure. Our services page covers that work, while the linked technical articles explore particular systems in more depth.