About the Guest

Thai helps teams build more resilient systems and improve their ability to effectively respond to incidents. A former EMT, he applies his experience managing emergency situations to the software industry. He writes about resilience engineering each week at ResilienceRoundup.com

Links Referenced: 

• Strongbad’s The System is Down 
• https://resilienceroundup.com/
• https://re-deploy.io/

Transcript

Mike Julian: This is the Real World DevOps podcast, and I'm your host, Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools to the organizers of amazing conferences, from the authors of great books to fantastic public speakers. I want to introduce you to the most interesting people I can find.

Ah, crash reporting, the oft forgotten about piece of solid monitoring strategy. You struggle to replicate bugs or elusive performance issues you're hearing about from your users, you should check out Raygun. Whether you're responsible for web or mobile applications, Raygun makes it pretty easy to find and diagnose problems in minutes instead of what you usually do, which, if you're anything like me, is ask the nearest person, “Hey, is the app slow for you?” and getting a blank stare back because hey, this is Starbucks, and who's the weird guy asking questions about mobile app performance? Anyways, Raygun, my personal thanks to them for helping to make this podcast possible. You can check out their free trial today by going to Raygun.com.

Mike Julian: Hi folks, I'm Mike Julian, your host for the Real World DevOps podcast. My guest this week is Thai Wood. He's an internal tools specialist at Fastly, the editor of Resilience Roundup, a fantastic newsletter that I'm a huge fan of. And perhaps really interesting is he's a former EMS professional. For those that don't know, EMS is what most of the Ops world keeps looking to figure out how can we improve our own on-call and incident management procedures. So, welcome to show Thai.

Thai Wood: Hey Mike, thanks for having me.

Mike Julian: So for those that really have no idea or frame of reference for what EMS is, what is it?

Thai Wood: EMS is sort of the broad umbrella of what happens anytime someone has a medical emergency, usually starts with someone calling 911. Someone gets sick or injured, they call 911 and the people who show up in whatever capacity are EMS workers and EMS professionals.

Mike Julian: So, would this also be like firefighters, police, doctors, nurses? Everyone's included in that or?

Thai Wood: Yeah, typically it's a big group of people just because who shows up actually, in the US strangely depends on where you live, what state you're in, what your county rules are, and a lot of things like that. So, here in Las Vegas, for example, we actually have three different fire departments, some of which run their own ambulance services, some of them do not. So it can also depend on what area of town you're in.

Mike Julian: Yes. San Francisco is the same way. The San Francisco Fire Department actually, I guess consumed the San Francisco Emergency Management into themselves so there's no such thing in San Francisco County is separate EMS. So that's why you hear so many fire trucks everywhere, it's because most of the incidents are not for fires, the city is not always burning down, it's just someone's always hurt. So they send a fire truck and police and more fire trucks. Where I'm from in Knoxville, Tennessee we actually had separate EMS, which was like they had the same vehicles as the ambulances except they were green instead of red, which was kind of cool. So yeah, that's fun.

So one of the things I really find interesting about EMS is when you look at people responding to incidents, and I'm using incident in a very broad, vague way here, someone gets hurt. And the people responding have just like the most cool, collected nature I've ever seen. And it's always like two people show up and everyone knows exactly what's going on. Meanwhile, the people who have called are freaking out, but the EMS personnel are calm and collected. How do you even get that way when someone's bleeding on the sidewalk, but EMS is perfectly cool about it.

Thai Wood: I think that's a really good question. It's actually something that I feel is missing in a lot of software, which is just a lot of it is experience. The hundredth time that you've been on a similar call, the thousandth time, of course like anything we habituate to it, and it gets easier. A lot of places with EMS, you have an opportunity to practice these things, and to get better at them. Oftentimes, that might be that you do ride-alongs even before you are certified. So, you're already becoming immersed in this world. You have an opportunity to do some clinical hours at hospitals, for example, sometimes, and you get to just be in these different situations. And of course, the first, maybe the fifth, maybe even the 10th, you feel the same way. There's also of course a culture where a lot of times what you're seeing is not in fact the truth.

Mike Julian: Okay. Tell me more about that.

Thai Wood: Well, depending on what it is that you're walking into, I or others might have an internal dialogue that say, “Oh, no, I have not seen this before.” But, we're not doing either of us a service by letting that dictate our outward response. If I'm letting that change my behavior or letting it allow me to be distracted or unfocused, I'm not helping you nor am I helping myself be more effective or be more effective in helping you.

Mike Julian: Yeah, that makes sense. I mean, I've definitely had that situation when I've been on call. You walk in has something like, well, everything's exploded, I have absolutely no idea what's going on. But as the senior engineer, everyone's looking to you like, expecting, you've seen this before. So you just kind of put on that veneer of, “Nope, I've got this,” while screaming internally.

Thai Wood: Absolutely. And I think that, depending on why it's done, I think it's actually a good thing. In software, I do tend to question a little bit, I think it should be okay to say, “I don't know, this thing. I haven't seen it before, but I'm going to figure it out,” whereas with … Because you're probably with personnel that you know at least somewhat if you're on call, right? Network operations, staff, people you've at least talked to before, which is not necessarily the case with emergency management, in the physical realm. These are strangers and you don't know how they're going to react. So your best service to them is to just keep that cool.

Mike Julian: Right. Yeah. It's just really hard to do sometimes or, most the time for me. Man, I do not miss being on call at all. I never liked the idea of having to present that coolness that everyone's looking for.

Thai Wood: I definitely understand that. It's a very visceral, physical experience. And I know a lot of us can habituate good and bad to some of this stuff. We know that noise, whatever our pagerduty alert noise is, we know that noise, right? Or even if you put it on vibrate, the sound of that phone vibrating against your nightstand, you know the ...