Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017

Channel:GOTO Conferences · Length:52:34 · Published:2017-05-10 · Views:101171

Watch on YouTube

Summary

Bryan Cantrill's GOTO Chicago 2017 talk on debugging production systems under pressure, centered on the incident where an operator's mistyped command rebooted an entire Joyent datacenter. It covers human factors, automation and architecture.

Why HN recommends it

A perennial HN recommendation for outage and postmortem threads, praised as funny, educational and one of the best postmortems anyone has seen.

The best postmortem I've seen was actually a conference talk — HN, 2023

Testing, debugging & developer tools · Cloud, DevOps & reliability

Subtitles (excerpt)

[Music] uh good morning um I am um I'm Brian Cantrell um I uh apparently am still being haunted by uset comments I made um over 20 years ago so everything lasts forever keep that in mind um so I I'm going to be um presenting this morning on a subject um near and dear to my heart um even though it is uh it often induces Cardiac Arrest or feels that way um and that is uh debugging under the most stressful possible conditions um debugging Under Fire debugging when systems have failed debugging during an outage um and uh the Genesis of this is a very very very bad day that we had several years ago so um this is an IRC log I I've redacted it slightly and i' I've obviously changed the names but this is an internal IRC log um that shows uh this is right before disaster strikes here um when one of the operators says I'm going to reboot um a bunch of empty RBS RBS is a Richmond B that's a type of compute node we have at joyant um I'm just getting the list right now now I should tell you now um that joyant is a public Cloud company um so yes we compete with AWS and and Google and Azure and so on so we have our own public Cloud um and this is an operator of that public Cloud about to reboot some of these empty nodes

On Hacker News

The best postmortem I've seen was actually a conference talk: "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" by Bryan Cantrill (in the GOTO conference, 2017) Here's a YouTube video of it: [VIDEO] Here's the slides: [LINK] It goes into detail about a pretty bad outage (when an entire data center was brought down), the human aspects, automation, how they handled it, the va… — KronisLV, 2023-03
Time to share one of my favorite talks (and speakers) ever - "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" (Bryan Cantrill, GOTO 2017) [VIDEO] — int0x2e, 2023-01
Perhaps not famous, but Bryan Cantrill, who gives my favorite talks, has an interesting and funny talk on one of the Joyent outages: [VIDEO] — OmarAssadi, 2022-01
Bryan Cantrill draws some excellent parallels between this incident and software systems in his "Debugging Under Fire" talk (beware of semi-automated systems): [VIDEO] — cls59, 2023-07
I’m surprised I haven’t seen this linked yet, but Bryan Cantrill of dtrace, Sun, lawnmower, Joyent, etc, fame gave an amazing talk for Monktoberfest 2016, titled “Oral Tradition in Software Engineering”, which features The Story of Mel [1]. Highly recommend checking it out — there are loads of little gems and stories like this throughout. All of his other presentations are great too and definitel… — OmarAssadi, 2022-08

Related videos