We have published a set of resources for people who have attended one of the Experience DevOps workshops. Our new Resources page contains links to books, newsletters, articles, slides, along with photos and brief descriptions of the activities in the workshop.
First of all, let me apologise for the Upworthy-style headline, I just couldn’t resist! Of course, there are probably a lot more than 3 reasons why building a DevOps culture could fail, but I’m going to concentrate on a few that I’ve experienced.
Developing a DevOps culture can be tricky. It’s not a natural fit for some people, which is why there are divisions between dev and ops in the first place. But before we delve too deep into the DevOps problem, let’s first have a look at what an organizational culture actually is…
“Culture is how organisations ‘do things’.” – Robbie Katanga
“The sum of values and rituals which serve as glue to integrate the members of the organisation” – Richard Perrin
“Organizational culture is civilisation in the workplace” – Alan Adler
These are just a handful of definitions from a Harvard Business Review on why we should care about organizational culture (you can find it here). As you can see, there are many different definitions of an organisational culture, so we have to be quite careful when we use the term to describe DevOps.
The problem with culture is that it differs from one organisation to another, so there’s no single universal way to affect, change or develop a culture. What works in one organisation, might not work in another. This is why nobody has ever come up with the magic “DevOps culture” recipe, because it just doesn’t exist (or rather, an almost infinite number of them exist).
I like to think of an organisational culture as a system of “norms” – an approved way of doing things and equally a way of discouraging “bad” behaviour. Of course, it’s usually implicit, but is always there, bubbling just below the surface.
I like this definition because it also works well in the sense of a “DevOps Culture” – a system of standard behaviours that blur the boundaries between dev and ops by encouraging collaboration and goal sharing (these are the “approved ways”) and discourage silos and separation (the “bad behaviours”).
How Do You Develop A Culture?
Developing a culture is really easy. You just start off with yourself, decide which culture you want your organisation to have, and then only hire people who fit perfectly within the culture. Of course, you need to be a 1 person start-up, but that’s pretty much the only caveat!
In reality, we mostly work in existing organisations, complete with their existing personnel and their existing culture (and sub-cultures). Building a DevOps culture in this situation can be very challenging, and that brings me to my list of 3 reasons why your attempts to create a DevOps culture could fail…
Reason 1: You Have to Hire the Right People
At some point you’re going to hire new staff. These could be new devs, ops, testers, product owners, whatever. Anyone who works on the software delivery team can have an impact on your culture. When you hire these new people, it’s essential they fit in with your vision of a new DevOps culture, or your existing DevOps culture if you already have one. But that’s not as easy as it sounds. How do you truly get a measure of a person’s culture through a couple of interviews? Well, it’s pretty hard. That’s why companies who value their culture adopt a “slow to hire, quick to fire” mentality. In these organisations, the hiring process can often take several weeks or even months to complete, and involve all sorts of jumping through hoops, but the reward is worth it. Does your organisation have this sort of diligence and patience when it comes to hiring anyone on the delivery team? If you value your DevOps culture, you need to be obsessive about cultural fit when it comes to hiring. You need to make sure your new hires are going to work within the existing culture, won’t try to oppose it, and will value it exceedingly highly.
Reason 2: You Need to Fire the People Who Don’t Fit
Of course, the second part of “Slow to hire, quick to fire” is getting rid of people who don’t fit the culture. When was the last time someone in your organisation was “let go” because they didn’t fit the culture, no matter how good a programmer/engineer/tester they were? It’s very rare indeed.
Zappos are a good example of a company who value culture above all else during the hiring process. They actually offer new applicants a bunch of money (up to around £3,000) to actually leave. It’s their way of incentivising applicants who feel that they don’t fit the culture, to take the money and run before it’s too late! And it’s probably wise to take the money if you don’t fit the culture, because they won’t waste time when it comes to getting rid of anyone who isn’t the right fit.
“We feel it’s a bigger risk to keep a bad employee than it is to let them go” says Jamie Naughton of Zappos (read more of what she has to say on the matter here)
Unfortunately, I’ve met very few organisations that are this obsessive about culture when it comes to hiring and firing. In fact, I often get the impression that culture is almost entirely overlooked when hiring, especially when hiring engineers. For some reason it seems almost acceptable to focus entirely on technical skillsets when hiring these types of employees and that is a huge risk to a DevOps culture.
Reason 3: You’ll Need to Remove Any Existing Cultures That Work to Oppose a DevOps Culture
If you think creating a DevOps culture is hard, wait until you try to eliminate an existing culture! That’s the thing with cultures, they’re not tangible things, you can’t just catch them and throw them in the bin! Blame cultures are fairly common, and they’re highly divisive – it’s almost the opposite type of culture to a DevOps culture. Anyone who has worked in an environment where there’s a blame culture will know how it spreads, and can quickly become part of the fabric of an organisation. It can be highly toxic. Eliminating this sort of culture is exceptionally hard, as people might need to entirely change their behaviour – unfortunately you can’t just fire everyone and start again!
When developing a DevOps culture, we normally have to challenge and overcome a culture of “throwing it over the wall”. This is a culture where teams don’t collaborate and support one another, and virtual walls exist between teams and departments. We often describe this situation as working in “silos”. In this environment, Dev teams usually don’t work very closely with the Ops teams, their respective disciplines are treated very separately, and their responsibilities are very clearly different. This is precisely the culture we need to replace, but the danger is that teams and individuals will revert to bad habits unless we make a concerted effort to ensure the new DevOps culture takes root, after all, as the saying goes, “old habits die hard”.
To learn more about the challenges of building a DevOps culture, book a place on an Experience Devops Workshop!
One of the exercises we do during the Experience DevOps workshop is a ball flow scenario inspired by the Ball Point game. We set up different combinations and topologies of Dev teams and Ops teams, and then see how many balls the teams can pass in (say) 60 seconds.
When we ran the workshop recently in Bangalore, we had a large number of participants, which enabled some interesting experimentation with the topology of the teams. In the exercise, the ‘Dev’ team takes balls from the ‘backlog’, and eventually passes the balls to the ‘Ops’ team, who must ‘make the features live’ under some constraints designed to simulate real-world physical constraints.
With the large group of participants in Bangalore, we experimented with multiple value streams (or products). After ‘warm-up’ runs using a single ‘product’ (value stream) and therefore a single Dev team and a single Ops team, we split people into two separate ‘Dev’ teams (one team for each product, and a backlog each) but only a single Ops team servicing both Dev teams:
In this topology, the teams were able to roughly match the throughput and error rate from before the Dev people were split into two teams (around 16 balls per minute, with around 4 defects). Then we removed the single ‘Ops’ team, and instead aligned half of the ‘Ops’ people with one product (value stream) and half with the other, creating ‘product teams’ (or ‘service’ teams):
The results were very striking: across all teams the overall throughput more than doubled with end-to-end service teams compared to the situation with a shared Ops team. One service team managed 16 balls with no defects, and the second team managed 20 balls and one defect, for a total of 36 balls and a single defect. Compare this to the 16 balls and 4 defects that were managed by the 2x Dev + 1x Ops team:
It was clear that – in this scenario – the single Ops team acted as a bottleneck. Part of the reason was due to (simulated) shared infrastructure. When we split people into service teams, we also split the infrastructure too, so that each service team deployed to its own set of ‘servers’.
The simplicity of the exercise, and the speed with which different topologies and constraints can be tried out, makes this ball flow game very useful for exploring different team topologies in a DevOps context.
John Willis (@botchagalupe), one of the most respected and knowledgeable people in the DevOps movement, recently gave a talk at the Silicon Valley DevOps meetup group on DevOps Culture. John has been active in DevOps since the beginning (somewhere around 2008, and probably before) and has spoken to thousands of people about DevOps since then. Here’s what he had to say about culture in relation to DevOps:
If you don’t get the C, don’t bother with the A, the M, and the S.
This is a reference to Culture, Automation, Measurement, Sharing (CAMS) which has become the de facto shorthand moniker for DevOps. John is saying that Culture is the foundation of DevOps, and that Automation, Measurement, and Sharing are simply not worth doing if you do not understand that good Culture is essential for DevOps. He continues:
For three or fours years I have been going around saying “Culture, culture, it’s got to be culture”, and people look at you with stare-y eyes going “Shut the hell up” or “Yeah, we get it John. Our culture, absolutely, yeah. NOW CAN WE GET THAT CHEF RECIPE RUN?”. People think [culture] is a ‘kittens and puppy dogs’ thing which no-one really wants to deal with because it’s HARD.
He then goes on to characterise several different approaches to DevOps culture: organisations which try to IMPOSE culture, command-and-control style (doomed to failure); organisations which believe that culture is IMPOSSIBLE to change (also a failure); and organisations (the successful ones) which try to WORK WITH the existing culture and shape it gradually.
Experience DevOps is a workshop which helps you to take that third approach: working with and shaping your existing organisational culture to encourage DevOps to thrive.
(Start from 09’00” in to hear the quotations above)
In a recent talk at PuppetConf 2013, co-author of Continuous Delivery and DevOps champion Jez Humble advocated passionately for not hiring in DevOps ‘experts’ but for cultivating a culture of collaboration and learning (slides 5-7)
This approach of nurturing a DevOps culture within your own organisation is what you will take from attending one of the Experience DevOps workshops; see http://experiencedevops.org/ for forthcoming workshop dates and locations.