How We Built CMX on Bare Metal Kubernetes with Cilium, eBPF, and BGP
CMX runs Kubernetes on physical servers we operate ourselves. Assigning an external address to a Service is only part of making it reachable. The routers outside the cluster also need a route to that address. We use Cilium to manage Service networking and BGP to advertise selected addresses upstream.
Our WebSocket services keep connections open while people use the platform. A routing change can disconnect those clients even when the applications are still running. We have to account for that when working on the network.
Why We Chose to Run Kubernetes Ourselves
Operating the cluster also makes CMX responsible for its control plane and the physical servers underneath it. We need spare capacity when a worker fails. Adding that capacity means provisioning another machine, which can take longer than requesting a cloud instance. Those delays affect how we plan for growth.
A company without experienced infrastructure staff would take on considerable extra work by running its own cluster. The Kubernetes production guidance explains the operational requirements. Managed Kubernetes can be a sensible starting point. Our DevOps staffing guide looks at the responsibility behind that choice.
If a deployment breaks connectivity, we need engineers who can distinguish a routing failure from an application fault. Operating the network includes that expertise.
What Runs Behind a CMX Connection
Chat traffic places different demands on the system from an ordinary page request. CMX uses a distributed WebSocket layer so clients can hold connections open while a conversation continues. Go runs the core services, while Node.js supports orchestration. Python handles AI and processing work. Some components use Rust where performance matters.
PostgreSQL stores the relational data we need to keep. Redis supports state used during live activity. Services communicate internally through gRPC and REST. When a WebSocket server disappears during a deployment, a client’s connection may drop even though its stored conversation remains intact. Reconnecting the client and preserving its history are separate pieces of recovery work.
The original CMX implementation separates control plane machines from workers and places workers across failure domains. Cilium handles cluster networking. Edge controls limit abusive traffic before it reaches the services.
Our WebSockets article covers reconnect storms and slow clients in more depth. The infrastructure scaling guide examines capacity and recovery at the wider platform level.
What Happens When a Physical Server Fails?
Kubernetes can reschedule a workload after a worker fails if the remaining nodes have enough capacity. A cluster that is already using all its memory may have nowhere to run the displaced workloads. We reserve room for failures when planning hardware capacity.
Worker loss and control plane failure need different recovery procedures. The first displaces workloads; the second can affect the cluster’s ability to manage them. We also own storage recovery. Those responsibilities stay with CMX even when Kubernetes can automatically restart a pod.
Recovery needs instructions somebody else can follow. The team should know where workloads move after a node failure and how to restore service if an important dependency goes down. We need to test those plans rather than rely on a diagram of redundant machines.
What Cilium Does When Traffic Reaches a Node
After traffic reaches a node, Cilium can choose a Service backend and enforce network policies before the packet reaches the application. It uses eBPF, which lets restricted programs run inside the Linux kernel. Cilium can also replace kube-proxy, the component that traditionally manages Kubernetes Service routing on nodes.
For troubleshooting, we need to know whether the cluster uses native routing or an overlay and whether kube-proxy replacement is enabled. Those choices affect where packets travel and which component to examine when a request disappears.
We run Cilium on bare metal as part of CMX’s Kubernetes networking. Its BGP integration helps advertise Service addresses to upstream routers. Before changing that configuration, we need to test the effect on Service traffic and persistent connections under realistic load.
An External IP Isn’t a Route
LoadBalancer is a Kubernetes Service type. It doesn’t create a physical load balancer by itself. A cloud integration normally provides the external network entry point. For our bare metal cluster, we supply the address assignment and routing ourselves.
Cilium’s LoadBalancer IP Address Management (LB IPAM) gives Services addresses from an administrator’s pool. You can see the address in Kubernetes, but an upstream router still needs to know how to reach it.
Cilium’s BGP Control Plane can advertise selected Service addresses to upstream routers. BGP carries information about which destinations a router can reach through its neighbors. A request can enter the cluster after the router learns the route, where Cilium still needs to deliver it to a healthy backend.
For a new Service, the sequence is worth checking directly:
- Confirm the Service received an address from the expected LB IPAM pool.
- Check that BGP policy selects that Service for advertisement.
- Verify that the upstream router learned a usable route to the address.
- Follow a request into the cluster and confirm it reaches a ready backend.
If a LoadBalancer Service still shows a pending external IP, check whether Cilium has an IP pool that matches it and read the Service’s status conditions. If it has an address but the router doesn’t know the route, inspect the BGP advertisement policy before debugging the application.
If you’ve used MetalLB, Cilium can cover similar jobs. LB IPAM can assign Service addresses while Cilium’s BGP Control Plane or its L2 announcements can expose them to the surrounding network. Check the upstream requirements before changing tools. Replacing a component doesn’t automatically preserve its routing behavior.
A Healthy Router Can Still Send Users to a Dead End
Requests can still fail with a healthy BGP session. A network policy might reject them at the node, or the Service may have no ready application endpoint. We check the routing information first, then follow the traffic into the cluster to find the failure.
Cilium’s BGP behavior also follows the Service traffic policy. With externalTrafficPolicy: Local, a node withdraws its advertisement when it loses its local endpoint for that Service. With externalTrafficPolicy: Cluster, a route can stay advertised even after every backend disappears. Cilium documents an enable-no-service-endpoints-routable setting that controls this behavior when no endpoints remain. A router may still see a valid route while the application can serve nobody. Cilium’s operations guide explains the failure cases.
For a failed request, we check the upstream route first. If the router points to a node, we inspect the Service’s endpoints and the policies Cilium applies to the traffic. This avoids spending time on application logs when the request never reached the application.
Test the kube-proxy Replacement Mode
Cilium can handle Kubernetes Service traffic through eBPF rather than relying on kube-proxy’s usual packet processing rules. The exact benefits depend on the workload and configuration. A fast path through the kernel still needs compatible versions and enough resources for the network state it maintains.
Changing how Service traffic is handled on an existing cluster can disrupt connections. Cilium’s documentation warns about this when switching the replacement mode. That’s particularly relevant to CMX’s WebSocket workload, where some clients may remain connected for a long time.
Before treating a networking upgrade as routine, find out whether a long running connection survives it. If it doesn’t, the reconnect behavior needs a test too.
Finding Where a Request Disappeared
Network debugging needs two views. We inspect the routes advertised to the physical network, then examine how Cilium handles traffic after it reaches a node. Application logs become useful once we know packets reached the service.
Hubble flow visibility helps us narrow down a failed request inside the cluster. When policy blocks traffic, we inspect the relevant Cilium rule. Traffic permitted toward a backend takes us to the receiving service next. If no flow appears, we revisit upstream routing while allowing for gaps in what Hubble recorded.
When long WebSocket sessions are affected, look at when connections were dropped and whether the clients tried to return together. A reconnect surge can overwhelm an otherwise healthy service after the original problem ends.
What Bare Metal Costs Beyond the Servers
It is easy to compare the rental cost of a server with the price of a cloud instance. That comparison leaves out a lot of the actual job. A server needs room to survive failures. Someone must replace failed hardware and keep its software current. For a production cluster, those responsibilities have a cost even when they don’t appear on a monthly invoice.
As of October 2026, Amazon EKS pricing lists a cluster fee of $0.10 per hour for versions in standard support and $0.60 per hour for extended support. The worker machines cost extra. AWS also bills for the storage and network services a deployment uses. That cluster fee alone tells you very little about the finished bill.
The bare metal calculation must include the network and backup systems we maintain. Engineering time counts too, particularly during upgrades and incidents. Those costs don’t appear as separate cloud service charges, but they still belong in the budget.
Our hardware can be economical when demand remains steady and the machines stay busy. A smaller team dealing with unpredictable traffic may find managed services easier to operate. The comparison needs to include spare capacity and the time engineers spend maintaining the platform.
An Upgrade Can Change the Network Under a Healthy App
An application can stay on the same version while a Kubernetes or Cilium upgrade changes how its traffic is handled. Linux kernel updates may affect the eBPF programs that Cilium relies on. That’s why the network needs a release process of its own.
Before a production change, test the intended versions against the actual cluster configuration. Check that traffic still reaches the right Services, and rehearse a way to reverse the change if it fails. For CMX, long lived connections are especially important during a rollout. Clients may need to reconnect even when the application code hasn’t changed.
After the rollout, check whether live traffic still reaches the application. The upgrade command may report success while existing connections are being dropped.
Questions to Answer Before Choosing Bare Metal
Before moving a production workload onto your own cluster, make sure someone can answer these questions with evidence from the actual setup.
- Recovery: can the team restore its data and recover when a control plane member fails?
- Capacity: can essential workloads continue if a worker goes offline?
- Routing: who manages BGP peers and IP address pools? What happens when a route is withdrawn?
- Access: can the team review who may change the cluster and how secrets are protected?
- Upgrades: where are changes tested before production, and how would the team reverse a bad rollout?
- People: who can respond when the engineer who designed the system isn’t available?
Give the recovery instructions to an engineer who didn’t build the cluster. If they can’t tell you how to respond to a withdrawn route or an unavailable node, the procedure needs work before more production traffic depends on it.
