About the Guest

Dave Mangot is the author of Mastering DevOps from Packt Publishing. He’s formerly the head of Site Reliability Engineering (SRE) for the SolarWinds Cloud companies and an accomplished systems engineer with over 20 years' experience. He has held positions in various organizations, from small startups to multinational corporations such as Cable & Wireless and Salesforce, from systems administrator to architect. He has led transformations at multiple companies in operational maturity and in a deeper adherence to DevOps thinking. He enjoys time spent as a mentor, speaker, and student to so many talented members of the community.

Guest Links

• Twitter: @davemangot
• Mastering DevOps course by Dave Mangot

Links Referenced: 

• Familiar Smells I’ve Detected in Your Ops Organization and How to Fix Them by Dave Mangot (LISA 2018)
• Knight Capital story
• Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation book by Jez Humble and David Farley
• Book recommendation: How to Measure Anything by Douglas W. Hubbard
• Calvin and Hobbes comic on Bridge Capacity
• Dan McKinley - Choose Boring Technology
• Nicole Forsgren’s DevOps research
• Book recommendation: The Lean Startup by Eric Ries

Transcript

Mike Julian: Running infrastructure at scale is hard, it's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly and author of O’Reilly's Practical Monitoring.

Mike Julian: This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database, InfluxDB, but you may not be as familiar with their other tools. Telegraph for metrics collection from systems, coronagraph for visualization and capacitor for real-time streaming. All of this is available as open source, and they also have a hosted commercial version too. You can check all of this out at influxdata.com.

Mike Julian: Hey folks, this is Mike Julian, I'm here with Dave Mangot, former head of SRE for SolarWinds Cloud. Welcome to the show, Dave.

Dave Mangot: Thanks Mike. It's great to be here.

Mike Julian: So why don't you tell us a bit about yourself and what you've been up to lately.

Dave Mangot: Sure. Um, I've recently, like you said, left SolarWinds Cloud. There I was running the global SRE organization. We started that with about two people at Librato and wind up growing that into multiple teams in multiple locations with certainly lots of products you've heard of. Before that, I was an architect and technical operations for Salesforce, working on their internal infrastructure, monitoring, automation, configuration management, all kinds of fun stuff like that.

Dave Mangot: I've been able to take a bunch of the experience of those two companies plus lots of years of experience doing stuff before that into a pretty fun conference talk that I gave a few weeks ago in Nashville at the USENIX LISA ‘18 Conference. And I was talking about familiar smells I've detected in your systems engineering organization and how to fix them. And so it was really fun to be able to use a lot of the things that I've seen in my career and all the things I've seen as we try to mature engineering organizations, both operationally and all kinds of other fun DevOps ways, and be able to give a talk about that. And pretty, I guess you could say, controversial at times, but well accepted at other times sort of venue.

Mike Julian: It's always interesting when you start pointing out the smells and people's infrastructure, and their reactions are kind of all over the place. Like some people like, “Yes, you're absolutely right.” And other times it's like, “No, no, please don't say that. Like, I'm not cool with you pointing out my flaws.”

Dave Mangot: Yeah, I think the important thing is, you know, what I was trying to get across in the talk as a message is like nobody's perfect, we all have to get better. If I've seen those things in other organizations, there's a chance that I was involved in those things. It's not like I just kind of wandered in from Mars and I knew everything and everything was all good, like I've been growing throughout my career, and I try to continue to keep growing throughout my career. And if people feel a little bit uncomfortable, I think that's good, right? That's a sign of growth, if we're feeling uncomfortable. It's just, what do we do with that? If we get defensive, then we could double down on something that maybe we know in deep inside isn't really working for us as opposed to maybe we go home and sleep on it and wake up the next day and say like, "Well maybe there's a couple of things here that we could actually get some incremental improvement on." And that's ultimately how we make all this successful changes to our infrastructure. 

Mike Julian: I've had people come find me where they've taken my place in their previous job, and they'll be like, "Why did you do this really dumb thing?" And my answer is, "Yeah, it was really dumb. I'm sorry about that." Like that that was a technical smell that I created, and I'm really sorry.

Dave Mangot: Yeah.

Mike Julian: These are things that we did, like they're not necessarily ... we didn't do them because they're good ideas, we did them for all sorts of reasons.

Dave Mangot: Yeah. And also I think that's important with like John [Allspaw 00:04:53] things that it's easy to come back later and say, why did you do that? That was a bad idea. But we make the best decisions that we can with the information that we have at the time.

Mike Julian: Absolutely.

Dave Mangot: It turns out that was a bad idea.

Mike Julian: Yup. So this talk that you gave at LISA, sadly I couldn't make it to LISA to see it live, but you and I have been talking...