Thank you for Subscribing to CIO Applications Weekly Brief
The First DevOps Hurdle
Jeffery Smith, Director, Production Operations, Centro
In the DevOps community, there is an acronym, CAMS. CAMS stands for Culture, Automation, Metrics and Sharing. These four concepts are the pillars on which all DevOps transformations are built. Many people focus on the tools and the automation, but the most important letter in this acronym is the first one, C. Without a change in culture, all of your attempts to move your organization forward will fail.
If you want the benefits of DevOps, holistic change is what you’re signing up for. Without it, you’ll gain all of the ritual with none of the substance. Many Agile implementations have suffered a similar fate. (You still release once a quarter, but now you have a daily standup) If you’re not committed to cultural change you’ll deliver DevOps in name only.
A definition of culture may be in order. Most companies have some sort of culture or values document that espouses their belief system. Documents and words do not make a culture, no matter how many times you repeat it. A company’s culture is formed by the values and or behaviors that it rewards and punishes. Employees will generally do more of what you reward and do less of what you punish. When companies focus too heavily on the punitive side, you get a fear-based culture.
Fear-based cultures are mired with notions of fault, blame and retributive justice. Failure is treated as the failing of individuals, never the failing of the complex system in which individuals operate. A poorly written requirements document is blamed on the project manager, but we never examine the larger system that let an unclear requirement document grow up to be a poorly launched product.
DevOps at its core is about empowering people to do their jobs safely, encouraging autonomy, and trusting your staff to make decisions
Those failures are not on individuals but on the entire system that failed to examine the warning signs. A fear-based culture never takes that deeper level look at their organization. The fault and its correction are with individuals and as a result, you’re left with an uninspired workforce too weary to try something extraordinary because the cost of failure is too high.
Every successful DevOps organization I have seen has a culture of embracing failure. Only through failure and learning from those failures, do we understand, improve and grow.
A DevOps transformation cannot succeed in a fear-based culture. DevOps at its core is about empowering people to do their jobs safely, encouraging autonomy, and trusting your staff to make decisions. With that trust, comes the risk of failure. But does your culture fear failure so much, that you avoid it at all costs? Create an environment where people can fail safely. Where experimentation is allowed and encouraged.
A real world example from my company—A team worked on a new feature for quite some time. On the day of the release, the feature ran into some major problems. We had prepared for failure, however, by wrapping the feature in a feature flag. Additionally, key metrics were identified for monitoring. When the metrics indicated failure, we turned the feature flag off and everything went back to normal. We celebrated this partial victory! We received valuable information that we couldn’t replicate in a test environment and we were able to do it with very little impact to the overall user community. We shared that experience with other teams so they could replicate the pattern as well. We fixed the production edge cases, and a week later, the second release was completed without a hitch.
When you have a fear-based culture, employees are always worried about retribution or punishment. This fear causes us to draw ourselves inward. It forces us to erect silos in order to ensure that our work has nice, clean, sharp edges. We want to make sure that my work and your work could never be confused as to avoid getting wrapped up in your failure. This fear creates a culture of disengagement amongst team members. People lose focus on the overall goal of a task and instead focus on delivering to the letter, the specific request that was asked of them. This absolves the team member of any responsibility, because they simply did as they were requested. This behavior robs the organization of the expertise an employee might have, because they don’t offer it.
There are great resources for helping to build a culture that can be free of fear. Sydney Dekker deals with the topic extensively in his book Just Culture: Restoring Trust and Accountability in your Organization. Blameless Postmortems are a tool used commonly in the DevOps community. David Zwieback tackles the subject in his book, The Human Side of Postmortems. These are both excellent starting points on how to begin engaging your teams in an open, honest and trust building manner.
Now, ask yourself honestly, do you have a culture that can embracefailure, learn from it and improve? Or are you a punitive culture that punishes mistakes and buries them instead. Are you losing the 98 successes just to avoid the two failures? If you are, you’ll never see that DevOps dream. You’ll just have new tools shackled by the same old fears.