Zebras All the Way Down - Bryan Cantrill, Uptime 2017

Channel:Code & Supply · Length:46:36 · Published:2017-10-23 · Views:49474

Watch on YouTube

Summary

Bryan Cantrill's Uptime 2017 talk on the engineering challenges of the stateful data path, including storage, firmware and hardware bugs. It uses the 'zebras vs. horses' diagnostic heuristic to explain why rare failures keep appearing.

Why HN recommends it

A perennial HN favorite whenever firmware, disks, ECC or opaque hardware come up, valued as both entertaining and sobering.

As with all his talks, you can expect it to be quite entertaining as well as informative — HN, 2020

Testing, debugging & developer tools · Databases & storage · Cloud, DevOps & reliability

Subtitles (excerpt)

[Applause] all right good afternoon it's great to be here I love a conference named uptime I guess you could argue that a conference named downtime would be even more appropriate for me but I'll take up time and it would be interesting to kind of watch all of these of these talks today and in particular you know I really enjoyed if the clicker is gonna work if not it's probably not because it's not plugged in so we'll just do it old-school um I like I thought a crakes talk was really interesting and Craig talked about all these great attributes on that we have with kind of kubernetes orchestrated things and they're beautiful I mean you you have all these things like you know immutability and we can recoverability and so on but the underlying them is a as a foundational assumption that these services are essentially stateless and we love statelessness I love statelessness don't catch me being pejorative about statelessness because I love it it's like the ability to model program as our problem as a finite state machine if you have the ability to do that you should seize it because you can make code that's totally correct and it's such a relief to be able to do that I love statelessness so stateless stateless yay

On Hacker News

For an entertaining/terrifying perspective on firmware, obligatory Bryan Cantrill talk "Zebras All the Way Down" [VIDEO] — thomasjudge, 2025-04
>Why does anyone really care about non-free firmware blobs? Aside from the philosophical considerations, stuff breaking in a black-box firmware that you're lead into believing "just works" is particularly aggravating. I'll link Bryan Cantrill's talk on the subject, he's more entertaining than I am. [1] [1] [VIDEO] — Bayart, 2021-01
In my time writing databases, I found that not only the mount-flags could result in data-loss despite fsync, given you use fsync as it is supposed to be used [0], but even enterprise-grade hardware would at times (at that scale [1]) drop and/or corrupt a few bits here and there. That reminds me of the case where AWS famously revealed that a single NIC in their ginaromous S3 fleet flipped a single… — ignoramous, 2020-02
There's a good Bryan Cantrill talk about that.[1] The gist is that eventually, when you throw enough resources at a problem, all that's left are the really uncommon problem and bugs, and this is specifically what you get in the data path (including drive firmware) where things get harder and harder to figure out as the code gets more hidden and obscure. As with all his talks, you can expect it to… — kbenson, 2020-01
On this topic (and related in many ways) is Bryan Cantrill's talk Zebras All the way Down.[1] The gist is that in medicine there's the concept of diseases that are zebras and horses, rare and common, and the saying goes "when you hear hoof beats think horses not Zebras". Unlike medicine, in software most the easy problems have already been abstracted away or made trivial through our software stac… — kbenson, 2019-07

Related videos