The Economics of DevOps

What comes to mind most often when we think about DevOps? Automated testing? CI/CD systems? Blue-green releases? What about economic concepts like pricing and incentives? Strangely, these are not often discussed, even though they’re at the heart of what DevOps is all about.

Although DevOps was born almost 15 years ago as an attempt to practice “Agile Infrastructure”¹, a seamless merging of development and operations activities, it too often gets reduced to tactical improvements such as fully automated builds, DevOps “teams” and so on (in my experience anyway)².

DevOps is about how people work together, not technology.

There’s nothing wrong, of course, with Docker or Jenkins, but these are not really “DevOps tools”. They’re simply automation tools that are more likely to be used by teams which have embraced DevOps. The real driver of a DevOps culture is not your build system, but how your organization allows people to work together to achieve the best results.

In theory, you can have DevOps with zero automation. Suppose a team is responsible for all the code of the application itself, plus testing it and deploying to production systems. Now let’s also say that, for whatever reason, they deploy it via shell scripts, executing CLI commands directly on servers or other scary ways to perform routine tasks. These are, obviously, bad ways to release code, but if it’s all the same team, it actually still fits within a DevOps paradigm. Because the team already works together, they will feel the incentive (urgently!) to improve their awful release processes, and will likely take time away from feature and bug work to do so. So while a CI/CD system might be the result of that work, it’s an output of their existing DevOps culture, not an input to it. The wrong (from a DevOps perspective) reaction in this case would have been to create a separate “Build Team” to take over the test and deployment stuff. Why?

The answer lies in the economic concept of market-clearing prices.

Consider:

• If a grocery store routinely charged $20/lb. for apples, we would expect to see lots of rotten apples in their dumpster. The price is too high.
• If Starbucks charged $1 for any drink, we would expect to see their stock price plummet. There would be no reason (other than preference) not to order the normally-expensive UltraMochaDoubleFrapSnickerBarMacchiato (extra whip!). With ingredients that cost more than the total sales price, they’d take a loss on every order. The price is too low.

In the case of apples, the actual price is normally around $2-4/lb., depending on the type of apple, and it’s no surprise that a black coffee is around $2, while the complicated dessert-like drinks can be well over $5.

In both cases, a functioning market exists because over time, buyers and sellers have figured out the value to each of them of apples and coffee, and market prices have been established (without involvement of a “National Coffee Board”).

This is the exact same motivation behind DevOps. If you can’t effectively release and operate your software, all the features in the world don’t really matter, and conversely, a gold-plated build and deploy system is worthless if your software is lacking key features, or is riddled with bugs. Both aspects are necessary, neither is sufficient on its own. A capable team can find the “market price” for all the necessary work if they are empowered to do so and held responsible. Incentives matter, and getting woken up at 3:30am provides motivation to increase testing and reduce risky deployments, which will both settle at sustainable levels in a true DevOps team.

All of this requires trust - if you do not trust your teams then you’ve either hired the wrong folks or are yourself mistrustful, cynical, or unwilling to delegate responsibility to an autonomous team. Any of those problems have to be fixed before adopting DevOps.

Is a car with two drivers better than one?

Another way to look at is, would you want to have one person steering the car while the other operates the pedals? Obviously not! The two-driver car would often come to rest well before a stop sign, over- or under-steer through turns, and smash into most things behind it when backing out of a parking space because it’s simply too complicated for two people to coordinate effectively enough to drive this way. You can do it better by yourself.

If there’s a bad driver in your family, maybe you let someone else take over, but having both of them drive is not an improvement.

In software, two drivers aren’t great either - you end up with the organizational equivalent of squabbling over how to merge into traffic (“steer… NOW!”). That means more layers of management, groups of “architects” and other such top-down measures, all trying to push and pull on the levers of the company to get the right result. It’s a lot harder to “decide” on how to build great software, than to just find out what the best way is through careful trial and error, and continuous feedback mechanisms.

Ultimately, DevOps is about putting the right people together to do a job from top to bottom, not about technology. This reduces friction between teams and requires fewer layers of management. It can also be more expensive, as you have to hire more folks with generalist skill sets, and there aren’t as many people who are as good at say, both Java and cloud infrastructure versus one of the two skills. In turn, this makes staffing-up such teams more difficult, as top-tier tech companies can pay the highest salaries for their scare talents. So yes, there are tradeoffs with DevOps, just as with anything else (also a fundamental lesson from economics³).

Footnote: What is a “DevOps Team”?

A DevOps team is not the people that work on build scripts, automated tests, or dashboards for management. That’s either a build automation team, QA, or something else entirely. An actual DevOps team is a group of folks that work on all aspects of “a thing” without handing it off to anyone else - there are no “Ops” people downstream of a DevOps team.

And, some further reading:

On market prices: “Supply” from the Concise Encyclopedia of Economics

A good synopsis of DevOps from AWS

Icons from Iconfinder, by KonKapp, Isaac Grant(CC-BY-3.0) and Wanida Sapsanthitikul.