Zebras All the Way Down - Bryan Cantrill, Uptime 2017
Channel:Code & Supply · Length:46:36 · Published:2017-10-23 · Views:49474
Summary
Bryan Cantrill's Uptime 2017 talk on the engineering challenges of the stateful data path, including storage, firmware and hardware bugs. It uses the 'zebras vs. horses' diagnostic heuristic to explain why rare failures keep appearing.
Why HN recommends it
A perennial HN favorite whenever firmware, disks, ECC or opaque hardware come up, valued as both entertaining and sobering.
As with all his talks, you can expect it to be quite entertaining as well as informative — HN, 2020
Testing, debugging & developer tools · Databases & storage · Cloud, DevOps & reliability
Subtitles (excerpt)
[Applause] all right good afternoon it's great to be here I love a conference named uptime I guess you could argue that a conference named downtime would be even more appropriate for me but I'll take up time and it would be interesting to kind of watch all of these of these talks today and in particular you know I really enjoyed if the clicker is gonna work if not it's probably not because it's not plugged in so we'll just do it old-school um I like I thought a crakes talk was really interesting and Craig talked about all these great attributes on that we have with kind of kubernetes orchestrated things and they're beautiful I mean you you have all these things like you know immutability and we can recoverability and so on but the underlying them is a as a foundational assumption that these services are essentially stateless and we love statelessness I love statelessness don't catch me being pejorative about statelessness because I love it it's like the ability to model program as our problem as a finite state machine if you have the ability to do that you should seize it because you can make code that's totally correct and it's such a relief to be able to do that I love statelessness so stateless stateless yay
On Hacker News
For an entertaining/terrifying perspective on firmware, obligatory Bryan Cantrill talk "Zebras All the Way Down" [VIDEO] — thomasjudge, 2025-04
>Why does anyone really care about non-free firmware blobs? Aside from the philosophical considerations, stuff breaking in a black-box firmware that you're lead into believing "just works" is particularly aggravating. I'll link Bryan Cantrill's talk on the subject, he's more entertaining than I am. [1] [1] [VIDEO] — Bayart, 2021-01
In my time writing databases, I found that not only the mount-flags could result in data-loss despite fsync, given you use fsync as it is supposed to be used [0], but even enterprise-grade hardware would at times (at that scale [1]) drop and/or corrupt a few bits here and there. That reminds me of the case where AWS famously revealed that a single NIC in their ginaromous S3 fleet flipped a single… — ignoramous, 2020-02
There's a good Bryan Cantrill talk about that.[1] The gist is that eventually, when you throw enough resources at a problem, all that's left are the really uncommon problem and bugs, and this is specifically what you get in the data path (including drive firmware) where things get harder and harder to figure out as the code gets more hidden and obscure. As with all his talks, you can expect it to… — kbenson, 2020-01
On this topic (and related in many ways) is Bryan Cantrill's talk Zebras All the way Down.[1] The gist is that in medicine there's the concept of diseases that are zebras and horses, rare and common, and the saying goes "when you hear hoof beats think horses not Zebras". Unlike medicine, in software most the easy problems have already been abstracted away or made trivial through our software stac… — kbenson, 2019-07
Related videos
- AWS re:Invent 2016: Tuesday Night Live with James Hamilton · Amazon Web Services · 1:33:27
- Tracy Young Speaks at Female Founders Conference 2015 · Y Combinator · 26:51
- AWS re:Invent 2018: How AWS Minimizes the Blast Radius of Failures (ARC338) · Amazon Web Services · 55:43
- Jeff Bezos 1997 Interview · Chuck Severance · 5:39
- Simulating an epidemic · 3Blue1Brown · 23:12
- Paxos lecture (Raft user study) · Diego Ongaro · 1:06:33
- Bret Victor - Inventing on Principle · Rui Oliveira · 54:20
- Advanced Topics in Programming Languages: A Lock-Free Hash Table · GoogleTalksArchive · 58:35
- DNA animation (2002-2014) by Drew Berry and Etsuko Uno wehi.tv #ScienceArt · WEHImovies · 7:19
- Hey Underscore, You're Doing It Wrong! · InfoQ · 36:30
- Keynote - JSON Graph: Reactive REST at Netflix · Association for Computing Machinery (ACM) · 54:57
- AWS re:Invent 2015 | (NET403) Another Day, Another Billion Packets · Amazon Web Services · 42:25
- A Philosophy of Software Design | John Ousterhout | Talks at Google · Talks at Google · 1:01:40
- Replace Your Exploit-Ridden Firmware with Linux - Ronald Minnich, Google · The Linux Foundation · 38:02
- CppCon 2015: Chandler Carruth "Tuning C++: Benchmarks, and CPUs, and Compilers! Oh My!" · CppCon · 1:29:53
- Run containers on bare metal already! · Bryan Cantrill · 41:12
- Principles of Technology Leadership | Bryan Cantrill | Monktoberfest 2017 · RedMonk Tech Events · 51:22
- Bryan Cantrill from Joyent on Manta: internet-facing object storage facility that features compute · AI Council · 1:31:05
- "Voice Driven Development: Who needs a keyboard anyway?" by Emily Shea · Strange Loop Conference · 41:17
- Maciej Ceglowski - Barely succeed! It's easier! · webdirections · 39:23