How We Built CMX on Bare-Metal Kubernetes with Cilium, eBPF, and BGP

We run bare metal Kubernetes with Cilium in production. That means our cluster runs on physical servers, not AWS, not GCP, not some managed control plane. We control the networking, the routing, and how traffic moves through the system.

CMX isn’t a static site or a simple backend, but a real time platform where users stay connected, services communicate continuously, and state changes happen live. Because of that, infrastructure decisions directly impact performance and stability.

We chose this setup because of how CMX actually works.

CMX handles:

• Persistent WebSocket connections
• Real time coordination between services
• Continuous service to service communication
• High connection density
• Performance sensitive workloads

When your system behaves like that, networking is everything. You can’t just spin up a managed cluster and hope the defaults hold under pressure.

In this article, we’ll break down:

• Why we chose bare metal over AWS EKS
• How Cilium and eBPF power our networking
• How BGP replaces cloud load balancers
• What the cost comparison actually looks like
• What we learned running this in real production

Why We Didn’t Use AWS EKS

AWS EKS is solid. It handles control plane upgrades, gives you managed load balancers, and hides a lot of infrastructure complexity. For many teams, that’s exactly what they need.

You can see the pricing on the official AWS EKS pricing page, where they charge a control plane fee per cluster hour on top of compute and load balancers.

Cost was part of our evaluation, but it wasn’t the main driver.

CMX runs long lived WebSocket connections and real time services talking to each other constantly. When thousands of connections remain open and services communicate continuously, routing consistency and traffic visibility become critical. We didn’t want networking decisions hidden behind cloud abstractions where we couldn’t fully see or influence how traffic flows.

If your team doesn’t have strong DevOps experience, managed Kubernetes is usually the smarter move. We talk more about that in our breakdown of DevOps team capability and infrastructure ownership.

For us, bare metal just matched how the platform behaves.

What Running Bare-Metal Kubernetes Actually Means

Running Kubernetes on bare metal means we manage everything ourselves:

• The control plane
• Networking
• Route advertisement
• Storage replication
• Cluster upgrades
• Monitoring
• Failure handling

There is no provider automatically fixing routing or provisioning infrastructure behind the scenes. If traffic shifts, it’s because we configured it. If a node fails, routing adjusts because we designed it that way.

When something breaks, we fix it. When a node goes down, we already know how traffic should reroute. When upgrades happen, they’re planned and tested.

Production discipline matters here. Backups are tested. Alerts go somewhere useful. Node failure is expected, not surprising.

A cluster that works in perfect conditions doesn’t prove much. A cluster that keeps running through traffic spikes and hardware issues does.

CMX Architecture on Bare Metal

CMX runs a distributed microservices system built with:

• Golang for core services
• Node.js for orchestration
• Python for AI and processing
• Rust for performance heavy components
• PostgreSQL for core data
• Redis for real time state

Instead of one large application, CMX is made up of multiple services that communicate constantly. That architecture allows the platform to scale and evolve, but it also increases networking complexity.

Services communicate over gRPC and REST. A custom distributed WebSocket layer handles persistent client connections, meaning users stay connected instead of reconnecting for every request.

We previously broke down how systems start failing under connection pressure in our WebSockets at scale article, and that experience directly influenced our infrastructure decisions.

Long lived connections stress load balancing differently than short HTTP requests. When thousands of connections stay open, routing consistency matters a lot more.

Inside our cluster:

• Control plane nodes are isolated
• Worker nodes span failure domains
• Edge traffic goes through rate limiting and security controls
• Cilium handles networking through eBPF

We can see what’s happening at every stage, from packet routing to service communication.

Networking Without a Cloud Provider

In AWS, if you declare a LoadBalancer service, it provisions one for you automatically. On bare metal, nothing happens unless you build it.

That means we handle:

• IP advertisement
• Service exposure
• Failover behavior
• Route propagation

IP advertisement and route propagation are simply how the network learns where services live and how traffic should reach them. In cloud environments, that logic is abstracted away. On bare metal, you design it directly.

We’ve written about how networking is usually the first thing to break as systems scale in what breaks first under infrastructure load. Running bare metal makes you understand that layer deeply instead of ignoring it.

Why We Use Cilium

Cilium replaces traditional iptables based networking with eBPF running inside the Linux kernel.

In simple terms, that means traffic handling happens closer to the system core instead of passing through layers of rules.

What that gives us:

• Lower latency
• More stable load balancing
• Better visibility
• Cleaner policy enforcement

Because CMX relies heavily on persistent connections and predictable routing, reducing network overhead directly improves stability.

Some bare metal setups use MetalLB for load balancing. We chose Cilium because it combines networking, BGP, and policy control into one system.

What eBPF Actually Does

eBPF lets small verified programs run inside the Linux kernel.

In Kubernetes networking, that means:

• Faster packet processing
• Less iptables overhead
• Better traffic inspection
• More predictable routing

Instead of stacking complex iptables rules, packet handling becomes more efficient and easier to observe.

For CMX, that translated into better stability under high connection density.

How We Use BGP Instead of Cloud Load Balancers

Cilium includes a BGP control plane that lets services advertise IP routes directly to upstream routers. You can read how it works in the official Cilium BGP documentation.

Instead of provisioning a cloud load balancer:

• Service IPs are advertised through BGP
• Routes update automatically if nodes change
• Failover behavior is predictable
• Traffic paths stay visible

We prefer that transparency because it allows us to understand and control how traffic enters and exits the cluster.

Cost Comparison: Bare Metal vs AWS EKS

According to the AWS EKS pricing model, you pay for:

• Control plane hours
• Worker nodes
• Load balancers
• Storage

With bare metal, we pay for fixed infrastructure instead of per-hour cloud billing.

Savings usually show up when:

• Traffic is steady
• Infrastructure is fully utilized
• You already have internal expertise

If you need to hire senior engineers just to operate the cluster, the savings shrink fast.

Bare metal can be cheaper. It depends on your situation.

Tradeoffs of Self-Hosted Kubernetes

Running Kubernetes yourself means you own:

• Upgrades
• Incidents
• Routing failures
• Capacity planning
• Storage reliability

There’s no cloud provider automatically absorbing complexity. That requires operational maturity.

Infrastructure choices should match team capability.

Lessons We Learned

Operating bare metal Kubernetes with Cilium taught us:

• Networking needs early attention
• Observability must be built in from the start
• Storage planning matters more than most expect
• Upgrade processes should be rehearsed
• eBPF improves performance but requires understanding
• BGP gives you clarity during failover

Bare metal exposes weaknesses quickly. It also gives you full control.

Frequently Asked Questions About Bare-Metal Kubernetes

Is bare metal Kubernetes cheaper than AWS?

It can be, especially when workloads are steady and you already have DevOps expertise. Removing managed control plane and load balancer fees reduces recurring costs, but engineering time must be considered.

Can Cilium replace MetalLB?

Yes, in many setups. Cilium supports load balancer IP allocation and BGP route advertisement, so you don’t always need MetalLB.

What does eBPF do in Kubernetes networking?

eBPF handles packet processing and service routing inside the kernel, which improves performance and visibility.

Is self-hosted Kubernetes reliable?

It can be very reliable if designed correctly with redundancy, monitoring, and tested failover.

When should you avoid bare-metal Kubernetes?

Avoid it if your team lacks networking experience or if your traffic needs elastic cloud scaling.

Infrastructure Should Match How Your System Behaves

CMX runs on bare-metal Kubernetes with Cilium and eBPF because that architecture matches how our platform operates under real-time load.

That experience directly informs the infrastructure and DevOps work we provide through our DevOps and infrastructure services.

The best infrastructure is the one built for how your platform actually runs.