About the Guest

Steve Mushero is CEO of Ops Platform provider Siglos.io, and CEO of ChinaNetCloud, China's first Internet Managed Service Provider, AWS Partner, and manager of hundreds of large-scale (up to hundreds of millions of users each) systems. He's previously been CTO in a variety of organizations in Silicon Valley, Seattle, New York, and around the world. You can follow along with his work and insights on LinkedIn, Medium and Twitter.

Links Referenced

• ChinaNetCloud
• Siglos
• Taking Over & Managing Large Messy Systems (LISA 2018)
• Incidents as we Imagine Them Versus How They Actually Are by John Allspaw (PagerDuty Summit 2018)
• Brandon Gregg’s USE Model
• Steve’s tool, runqstat
• Observability vs. Monitoring, is it about Active vs. Passive or Dev vs. Ops? By Steve Mushero
• Dao of Troubleshooting
• How to Monitor the SRE Golden Signals

Transcript

Mike: Running infrastructure at scale is hard. It's messy. It's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps Podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly, and author of O'Reilly's Practical Monitoring.

Mike: This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools, and that's where Influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database, InfluxDB, but you might not be as familiar with their other tools, Telegraf for metrics collection from systems, Chronograf for visualization, and Kapacitor for real-time streaming. All of this is available as open source, and they also have a hosted commercial version too. You can check all of this out at influxdata.com.

Mike: Hi, folks. My name is Mike Julian. I'm here with Steve Mushero, CEO of ChinaNetCloud, and his new startup, Siglos. Welcome to the show, Steve.

Steve: Thank you. Glad to be here.

Mike: You and I met through Monitoring Weekly some time ago, and we've just been chatting since then, and learning all about, or me learning, primarily, about the wild world of systems in China. I believe the company you run, ChinaNetCloud is one of the largest, if not the largest Chinese AWS partner?

Steve: Probably. We're the first one. We were their first sort of operations partner here in China, and also their first MSP, or Managed Service Provider. We've been AWS here and around the world almost 10 years. Yeah, it's been an interesting time.

Mike: Yeah, I'm sure. Why don't you tell us a bit more about what your company does? What are you doing in the Chinese market?

Steve: Here in China, we're basically a managed service provider, so we started 10 years ago now, really managing systems. I was CTO for Tudou, similar to YouTube in the US, so online video and so on. Actually, older and bigger than YouTube. We saw how that market really needed operations help so we really launched this to start helping people design, build, operate, manage, monitor, all that sort of stuff, internet systems. This was before the clouds and all that. Then, we were the first cloud company in China actually doing online VMs. You could buy actual virtual servers, now, of course you have AWS, and Aliyun, and others doing that. Today, our job is really just design, build, manage, monitor, troubleshoot, and do 7-by-24 support for large-scale internet systems, so we're way in the backend, you might say, but really sort of ditch-digging operations guys and girls, really dealing with the just 24-hour systems in clouds, and hybrids, and all the interesting problems you can have. Especially with Chinese scale that we do. That's what we do.

Mike: I think it's absolutely wild that there is ... To think about the size of YouTube, like, pretty much everyone everyone in the western world thinks about YouTube, it's like, oh, wow, that's absolutely huge. To think that there's a competitor in China that's bigger than that, the scale is staggering.

Steve: Yeah. I think now YouTube is larger, but back, this is 10 years ago, now-

Mike: Oh, okay.

Steve: ... It was different, because YouTube, even now in the US, think of it more like Netflix and YouTube combined, because YouTube historically has been short video clips, right? 10 years ago, it was, hey, my cat is cute, stuff like this, you know, 90-seconds, three-minutes, five-minutes. We were running full length movies and TV, and all that, not entirely legally, you might say. It was more like Netflix.

Steve: We had a 100 million viewers a day doing sort of full length stuff, so YouTube's average thing was whatever, you know, a couple minutes, ours was an hour. Completely different infrastructure needs. This is before, especially China had good infrastructure, and BGP routing, and all kinds of stuff, so you had sort of massive CDM problems, and lots of other stuff. I think we were the world's biggest bandwidth buyer for a while.

Mike: Oh, wow.

Steve: I know. Now, everybody else is bigger, but this is, again, a long time ago.

Mike: Oh, sure.

Steve: Because, DSL had just hit China, and everybody wanted to watch Western movies and TV, and a lot of people a lot of times had nothing else to do during the day, and so they're watching this everywhere. Before smartphones too. Yes, it was quite the interesting- Actually, that company's older than YouTube, so even though a lot of things come into China are sort of copies of Western things, those companies actually had much more features, and much richer than YouTube and even Netflix today ... 10 years ago. More advanced, innovative, actually, than the Americans at this point.

Mike: Right.

Steve: It was pretty cool. I learned a lot about infrastructure here, and the craziness of data centers, and large scale systems, you know, thousands of servers, and all kinds of locations, all this kind of stuff. It's all physical servers, no virtualization, no nothing, right? This is started in 2005, and I was here in 2007. It's hundreds of racks of equipment, and all that kind of stuff.

Mike: What's it like today? Thinking back to what I was doing for systems in 2007 ....