Gunnar: When I started out, I perhaps didn’t expect to find that many unexpected things that actually showed more resilience or more reliability than we actually thought.

Jason: Welcome to the Break Things on Purpose podcast, a show about Chaos Engineering and building more reliable systems. In this episode, we chat with Gunnar Grosch, a Senior Developer Advocate at AWS about Chaos Engineering with serverless, and the new reliability-related projects at AWS that he’s most excited about.

Jason: Gunnar, why don’t you say hello and introduce yourself.

Gunnar: Hi, everyone. Thanks, Jason, for having me. As you mentioned that I’m Gunnar Grosch. I am a Developer Advocate at AWS, and I’m based in Sweden, in the Nordics. And I’m what’s called a Regional Developer Advocate, which means that I mainly cover the Nordics and try to engage with the developer community there to, I guess, inspire them on how to build with cloud and with AWS in different ways. And well, as you know, and some of the viewers might know, I’ve been involved in the Chaos Engineering and resilience community for quite some years as well. So, topics of real interest to me.

Jason: Yeah, I think that’s where we actually met was around Chaos Engineering, but at the time, I think I knew you as just an AWS Serverless Hero, that’s something that you’d gotten into. I’m curious if you could tell us more about that. How did you begin that journey?

Gunnar: Well, I guess I started out as an AWS user, built things on AWS. As a builder, developer, I’ve been through a bunch of different roles throughout my 20-plus something year career by now. But started out as an AWS user. I worked for a company, we were a consulting firm helping others build on AWS, and other platforms as well. And I started getting involved in the AWS community in different ways, by arranging and speaking at different meetups across the Nordics and Europe, also speaking at different conferences, and so on.

And through that, I was able to combine that with my interest for resiliency or reliability, as someone who’s built systems for myself and for our customers. That has always been a big interest for me. Serverless, it came as I think a part of that because I saw the benefits of using serverless to perhaps remove that undifferentiated heavy lifting that we often talk about with running your own servers, with operating things in your own data centers, and so on. Serverless is really the opposite to that. But then I wanted to combine it with resilience engineering and Chaos Engineering, especially.

So, started working with techniques, how to use Chaos Engineering with serverless. That gained some traction, it wasn’t a very common topic to talk about back then. Adrian Hornsby, as some people might know, also from AWS, he was previously a Developer Advocate at AWS, now in a different role within the organization. He also talked a bit about Chaos Engineering for serverless. So, teamed up a bit with him, and continue those techniques, started creating different tools and some open-source libraries for how to actually do that. And I guess that’s how, maybe, the AWS serverless team got their eyes opened for me as well. So somehow, I managed to become what’s known as an AWS Hero in the serverless space.

Jason: I’m interested in that experience of thinking about serverless and reliability. I feel like when serverless was first announced, it was that idea of you’re not running any infrastructure, you’re just deploying code, and that code gets called, and it gets run. Talk to me about how does that change the perception or the approach to reliability within that, right? Because I think a lot of us when we first heard of serverless it’s like, “Great, there’s Nothing. So theoretically, if all you’re doing is calling my code and my code runs, as long as I’m being reliable on my end and, you know, doing testing on my code, then it should be fine, right?” But I think there’s some other bits in there or some other angles to reliability that you might want to tune us into.

Gunnar: Yeah, for sure. And AWS Lambda really started it all as the compute service for serverless. And, as you said, it’s about having your piece of code running that on-demand; you don’t have to worry about any underlying infrastructure, it scales as you need it, and so on; the value proposition of serverless, truly. The serverless landscape has really evolved since then. So, now there is a bunch of different services in basically all different categories that are serverless.

So, the thing that I started doing was to think about how—I wasn’t that concerned about not having my Lambda functions running; they did their job constantly. But then when you start building a system, it becomes a lot more complex. You need to have many different parts. And we know that the distributed systems we build today, they are very complex because they contain so many different moving parts. And that’s still the case for serverless.

So, even though you perhaps don’t have to think about the underlying infrastructure, what servers you’re using, how that’s running, you still have all of these moving pieces that you’ve interconnected in different ways. So, that’s where the use case for Chaos Engineering came into play, even for serverless. So, testing how these different parts work together to then make sure that it actually works as you intended to. So, it’s a bit harder to create those experiments since you don’t have control of that underlying infrastructure. So instead, you have to do it in a few different ways, since you can’t install any agents to run on the platform, for instance, you can’t control the servers—shut down servers, the perhaps most basic of Chaos Engineering experiment.

So instead, we’re doing it using different libraries, we’re doing it by changing configuration of services, and so on. So, it’s still apply the same principles, the principles of Chaos Engineering, we just have to be—well, we have to think about it in different way in how we actually create those experiments. So, for me, it’s a lot about testing how the different services work together. Since the serverless architectures that you build, they usually contain a bunch of different services that you stitch together to actually create the output that you’re looking for.

Jason: Yeah. So, I’m curious, what does that actually look like then in testing, how these are stitched together, as you say? Because I know with traditional Chaos Engineering, you would run a blackhole attack or some sort of network attack to disrupt that connectivity between services. Obviously, with Lambdas, they work a little bit differently in the way that they’re called and they’re more event-driven. So, what does that look like to test the connectivity in serverless?

Gunnar: So, what we started out with, both me...

Rate