Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017
Channel:GOTO Conferences · Length:52:34 · Published:2017-05-10 · Views:101171
Summary
Bryan Cantrill's GOTO Chicago 2017 talk on debugging production systems under pressure, centered on the incident where an operator's mistyped command rebooted an entire Joyent datacenter. It covers human factors, automation and architecture.
Why HN recommends it
A perennial HN recommendation for outage and postmortem threads, praised as funny, educational and one of the best postmortems anyone has seen.
The best postmortem I've seen was actually a conference talk — HN, 2023
Testing, debugging & developer tools · Cloud, DevOps & reliability
Subtitles (excerpt)
[Music] uh good morning um I am um I'm Brian Cantrell um I uh apparently am still being haunted by uset comments I made um over 20 years ago so everything lasts forever keep that in mind um so I I'm going to be um presenting this morning on a subject um near and dear to my heart um even though it is uh it often induces Cardiac Arrest or feels that way um and that is uh debugging under the most stressful possible conditions um debugging Under Fire debugging when systems have failed debugging during an outage um and uh the Genesis of this is a very very very bad day that we had several years ago so um this is an IRC log I I've redacted it slightly and i' I've obviously changed the names but this is an internal IRC log um that shows uh this is right before disaster strikes here um when one of the operators says I'm going to reboot um a bunch of empty RBS RBS is a Richmond B that's a type of compute node we have at joyant um I'm just getting the list right now now I should tell you now um that joyant is a public Cloud company um so yes we compete with AWS and and Google and Azure and so on so we have our own public Cloud um and this is an operator of that public Cloud about to reboot some of these empty nodes
On Hacker News
The best postmortem I've seen was actually a conference talk: "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" by Bryan Cantrill (in the GOTO conference, 2017) Here's a YouTube video of it: [VIDEO] Here's the slides: [LINK] It goes into detail about a pretty bad outage (when an entire data center was brought down), the human aspects, automation, how they handled it, the va… — KronisLV, 2023-03
Time to share one of my favorite talks (and speakers) ever - "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" (Bryan Cantrill, GOTO 2017) [VIDEO] — int0x2e, 2023-01
Perhaps not famous, but Bryan Cantrill, who gives my favorite talks, has an interesting and funny talk on one of the Joyent outages: [VIDEO] — OmarAssadi, 2022-01
Bryan Cantrill draws some excellent parallels between this incident and software systems in his "Debugging Under Fire" talk (beware of semi-automated systems): [VIDEO] — cls59, 2023-07
I’m surprised I haven’t seen this linked yet, but Bryan Cantrill of dtrace, Sun, lawnmower, Joyent, etc, fame gave an amazing talk for Monktoberfest 2016, titled “Oral Tradition in Software Engineering”, which features The Story of Mel [1]. Highly recommend checking it out — there are loads of little gems and stories like this throughout. All of his other presentations are great too and definitel… — OmarAssadi, 2022-08
Related videos
- Gender Neutral Pronouns: They're Here, Get Used To Them · Tom Scott · 3:48
- BlueHat IL 2019 - Andrew "bunnie" Huang - Supply Chain Security: "If I were a Nation State...” · Microsoft Israel R&D Center · 45:37
- But what is the Fourier Transform? A visual introduction. · 3Blue1Brown · 19:41
- Cybersecurity as Realpolitik by Dan Geer presented at Black Hat USA 2014 · Black Hat · 54:35
- Mark Blyth ─ Global Trumpism · Watson School of International and Public Affairs · 1:26:03
- Chernobyl 3828 · TeleconStudio · 26:57
- Flat Earth: A Measured Response · hbomberguy · 45:17
- TasBlock at AGDQ 2017: TASBot · TASVideosChannel · 58:44
- George Lakoff: Moral Politics · University of California Television (UCTV) · 58:57
- Tom Lehrer - Wernher von Braun · The Tom Lehrer Wisdom Channel · 1:46
- Congressional Fundraising: Last Week Tonight with John Oliver (HBO) · LastWeekTonight · 21:25
- DCEmu Reviews - 27C3 - Chaos Communication Congress 2010 - fail0verflow - FULL VIDEO · 1KIND Reviews · 44:39
- They're Made Out of Meat · niccollsdp · 6:16
- Hovering a Helicopter is Hilariously Hard - Smarter Every Day 145 · SmarterEveryDay · 9:45
- Data | Philosophy Tube · Philosophy Tube · 26:14
- The Alt-Right Playbook: Control the Conversation · Innuendo Studios · 10:47
- Super Mario World -- Credits Warp in 5:59.6 (First Time Ever on Console) · SethBling · 9:00
- Deadly Truth of General AI? - Computerphile · Computerphile · 8:30
- Velocity NY 2013: Richard Cook, "Resilience In Complex Adaptive Systems" · O'Reilly · 19:09
- The more general uncertainty principle, regarding Fourier transforms · 3Blue1Brown · 18:06