DevOps as a Service vs In House DevOps Teams: How SaaS Companies Should Decide
In January 2017, an engineer working to restore a database accidentally deleted data from GitLab.com’s main database server. The recovery team then discovered that the usual database backups were not available. GitLab later said nobody had been responsible for regularly testing that backup procedure. The incident cost the service many hours of downtime and roughly six hours of database changes that could not be recovered. Its published account explains what went wrong.
That’s the question I’d bring into any decision about DevOps as a service or an internal team. When production breaks, who can recover it? It doesn’t matter how reassuring the sales presentation is if nobody has tested the process.
Both an internal team and an outside provider can run reliable systems. Both can also miss important work when ownership isn’t clear. Judge the model by what people are responsible for and what they can prove they’ve tested.
Start With the Work Production Depends On
Someone must be responsible for getting changes into production safely. The team also needs to know whether services are working for customers and what to do if an update breaks them. Backups only become useful when somebody can restore the data.
Write down who is responsible for those jobs today. If the answer is simply the person who built the system, ask what happens when they’re away. A managed provider may cover some of that work, but its contract needs to say which parts and during which hours.
The hiring decision comes after you’ve identified what the product needs. The job shouldn’t be defined by a list of fashionable tools.
You Have More Choices Than Hiring or Outsourcing Everything
A short project with an experienced engineer can fix a specific deployment problem while the company’s own team continues running production. A managed service can take responsibility for ongoing work when the scope is clear. An internal platform team may make more sense when that work is constant and tightly tied to the product.
Many companies use a mix. A developer inside the business understands the application while an outside specialist helps with infrastructure changes. That can work, provided the two sides know who leads an incident and who is allowed to change production.
Never assume the other team is taking care of backups or overnight alerts. Make those responsibilities explicit before choosing the model.
One Excellent Engineer Still Needs Backup
A skilled infrastructure engineer can make a small company safer and faster. But one person cannot provide a healthy incident rotation every night of the year. They also need time away from alerts to fix the causes of recurring failures.
Google’s guide to being on call explains why sustainable coverage and clear incident procedures matter. A company building its own team needs a plan for those responsibilities, especially when its best engineer is away.
Internal ownership is valuable when daily platform decisions depend on deep knowledge of the product. It still needs a plan for recovering from incidents without relying on one person’s memory.
What Should Stay Inside the Company?
Some production decisions depend heavily on the product. An external engineer may understand the infrastructure but not the business effect of disabling a particular feature. Keep someone inside the company who can make those decisions, even when an outside team handles the technical repair.
A provider may be useful when urgent coverage is missing or the system needs specialist skills the company doesn’t use every day. An internal team becomes more attractive when infrastructure work is constant enough to justify hiring people who can share it.
For a hybrid model, write down who leads the incident. Don’t leave both sides assuming the other one has permission to act.
Use Operational Pressure as a Sourcing Signal
If every production release depends on one engineer being available, the company has a problem regardless of who employs that person. The same is true when nobody can explain how to restore a database. Look at the work production depends on before deciding whether an outside team or new employees should take responsibility.
External support can make sense when the gap is urgent and the company lacks enough internal specialists to cover it. Internal hiring can make sense when the work is constant, product context is unusually deep, and leadership is prepared to fund a real team instead of one isolated “DevOps person.”
The infrastructure scaling guide on what breaks first as systems grow provides a diagnostic for the technical pressure itself. This page focuses on who should own the response.
Compare the Real Cost of Keeping Production Running
A salary and a monthly retainer don’t buy the same thing. One employee may know the product extremely well but still need help during a holiday or a difficult overnight incident. A managed provider may offer broader coverage yet still need someone inside the business to approve risky changes.
Compare the cost of meeting the same responsibilities. Ask what happens outside ordinary hours and who pays for work that falls beyond the agreement. For an internal team, consider the time spent recovering from incidents instead of improving the platform. That’s work the business is already paying for, even when it isn’t labelled DevOps.
I’d be cautious of any quote that makes reliability look cheap by ignoring the person who has to wake up when something goes wrong.
Who Can Act During a Real Incident?
Paying for overnight monitoring isn’t the same as giving someone authority to restore service. The person responding needs to know what can be changed safely. They also need a way to reach the person who understands the application’s effect on customers.
In GitLab’s postmortem, regular restore testing hadn’t had a clear owner. The backups were supposed to protect the service, yet the recovery team couldn’t use them when it mattered. Ask a potential provider who performs those tests today and how you can see the results.
Ask for evidence of a recent recovery test. Then ask how the team would handle the next failure if the usual engineer were unavailable. The answer should describe a working process, not depend on somebody’s phone happening to be on.
Outside Engineers Need Access You Can Revoke
A contractor helping with a Kubernetes upgrade may need broad access while the change is in progress. That doesn’t mean the person should keep permanent administrator permissions after the work is finished. Give each engineer their own account and keep a record of what they changed.
NIST’s Zero Trust Architecture describes granting only the access needed for the task. The business should be able to revoke a provider’s access without losing its own infrastructure or the instructions needed to operate it.
Measure Whether Production Is Getting Better
A provider can generate a beautiful dashboard while the business still suffers the same outages. Ask whether releases are becoming safer and whether the team can recover more reliably after something fails.
The DORA delivery metrics look at how software changes move into production and how often they require recovery or extra work. They’re useful evidence when teams interpret them in context. For the service itself, measure something customers can feel. Does checkout work? Can the business restore data when needed?
Google’s error budget approach provides one way to agree on how much unreliability a product can tolerate before the team slows down releases to improve stability. A small business doesn’t need a complicated version. It needs targets people understand and can use to decide what work comes next.
What I’d Ask Before Hiring a DevOps Provider
Ask who answers when a serious alert goes off overnight. Then ask what they’re allowed to change without permission. A provider that monitors the system but can’t restore service has a very different job from a team that owns incident response.
Request a real example of the recovery process, with private details removed. How was the problem detected? Could the team restore data? What changed afterward? You should be able to inspect the process rather than simply hear that the provider follows best practices.
Finally, ask what happens if the relationship ends. The business should keep control of its infrastructure and enough documentation for another qualified team to operate it.
What the First Engagement Should Prove
Early work should reveal what the company is depending on and who has access to production. Fix gaps that could prevent the team from recovering. A provider should also show that it can release a change and roll it back safely when needed.
I’d want a real backup restore test before a long presentation about future automation. If the team can’t restore a system, you have learned something urgent about the operating model. A sensible plan follows those findings rather than installing new tools just to fill a schedule.
Make the Decision on Paper Before Making It in Production
For the most important system, write down who responds to an outage. Who can approve a rollback? When must someone from the business join the call? If no one knows the answers today, that gap should shape the hiring decision more than a comparison of monthly fees.
Use the same questions for an internal candidate and an outside provider. Agree when to review the choice after some real operating experience.
Could You Change Providers Without Losing the System?
A company should be able to end a support contract without losing access to production. Keep the main cloud accounts under the business’s control and make sure another qualified engineer can find the current recovery instructions.
When ownership changes, let the new team prove it can perform a recovery before removing the old team from the schedule. Then revoke permissions that are no longer needed. A good provider should make that handoff possible even if you never use it.
The Engineering Work Behind the Contract
Operating a platform involves decisions that can’t be settled by a sales brochure. CMX’s article on bare metal Kubernetes with Cilium explains a particular network and infrastructure design in depth. It gives technical readers something concrete to examine, although it isn’t a default setup we’d recommend for every company.
For help defining the work your platform needs, our services page is a starting point. The useful conversation begins with what production does today and what happens when it fails.
Ask Them to Restore Something
The GitLab outage is memorable because the usual fallback wasn’t available when the team needed it. That is the test I’d take into a DevOps hiring decision. Show me a recent recovery exercise. Explain what the team learned and what changed afterward.
Either staffing model can work. What matters is having somebody qualified to respond who has practiced the job before production breaks. A promise to be available is easier to make than proof that the system can recover.
