"Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson
Channel:Strange Loop Conference · Length:40:20 · Published:2014-09-20 · Views:55933
Summary
Will Wilson of FoundationDB explains how the team tested their distributed database using deterministic simulation. By running the whole system in a simulator with injected faults and reproducible seeds, they could find rare concurrency and network bugs.
Why HN recommends it
HN treats it as the canonical talk on simulation testing and a mind-blowing example of how far rigorous testing can go.
this is one of the most amazing talks on distributed systems — HN, 2018
Testing, debugging & developer tools · Databases & storage · Distributed systems
Subtitles (excerpt)
all right good morning everybody uh so my name is Will I work at Foundation DB if you haven't heard of us we make a scalable and fault tolerant database with acid transactions I'm not here to talk about that today though uh instead I'm going to talk about simulation testing and why it might make your life easier so I want to start with an observation which I hope everybody here agrees with which is that debugging distributed systems is preferable to sticking a fork in your eye uh but not that preferable it's actually like it's actually surprisingly close um and people will tell you a lot of reasons why this is true uh one of the most common ones I hear is it's because they're complicated and I'll I'll grant that it's true they are complicated and debugging complicated stuff is harder than debugging simple stuff but I want to propose that this is not the real reason why they're really hard to debug um so to to look at the real reason let's look at a super simple example so we've got a server on the left here and a server on the right here and the server on the right is going to send a packet to the server on the left and this is a non item potent packet that means if it gets received twice it has a different effect than if
On Hacker News
"FDB can tolerate f failures with only f+1 replicas." Wait a minute, I know that formula... Viewstamped Replication?? I need to read the foundation DB paper. (I mainly read CRDT stuff so hopefully it's understandable). --- In general I'm really impressed foundation DB folks. The talk "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson blew my mind. TL;DR they spent the majori… — LAC-Tech, 2023-09
Here's another excellent talk at Strangeloop on FoundationDB's simulation testing by Will Wilson in 2014: [VIDEO] — sgk284, 2021-06
Soooo stoked. This should make futzing with it to get started a lot easier. I've been thinking about going to the FoundationDB summit to hopefully learn more... Does anyone have any good resources for learning about it? I've tried a few times but find the documentation a bit hard to grok. Maybe I'm just slow. Also if you haven't seen it, this is one of the most amazing talks on distributed system… — rubyn00bie, 2018-11
This talk was about Foundation DB was brought up recently, and it's pretty amazing. I recommend watching the whole thing, but to be brief they are taming the "infinite interleavings" problem through determinism. "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson [VIDEO] They wrote an interesting Actor DSL that compiles to C++ and is completely deterministic, and they torture… — chubot, 2018-05
One of their engineers gave a great talk at Strangeloop ( [VIDEO] ) that shows some of the amazing simulations they ran FoundationDB through. It's well worth a watch if you want to understand the punishment that they put this database through to make it stand out. — teeray, 2015-03
Related videos
- Principles for Dealing with the Changing World Order by Ray Dalio · Principles by Ray Dalio · 43:42
- John Carmack Tech Talk with UMKC-SCE · UMKC School of Science and Engineering · 1:20:28
- "Data Driven UIs, Incrementally" by Yaron Minsky · Strange Loop Conference · 36:18
- Maybe Not - Rich Hickey · ClojureTV · 1:03:28
- The Clean Architecture in Python · Next Day Video · 49:53
- GopherCon 2016: The Design of the Go Assembler - Rob Pike · Gopher Academy · 23:56
- RustConf 2017 - Closing Keynote: Safe Systems Software and the Future of Computing by Joe Duffy · Confreaks · 42:45
- Dtrace Review · GoogleTalksArchive · 1:18:02
- 2017 ACM PPoPP Keynote: It's Time for a New Old Language · Guy Steele · 1:12:12
- Travis Oliphant: NumPy, SciPy, Anaconda, Python & Scientific Programming | Lex Fridman Podcast #224 · Lex Fridman · 3:05:24
- Armin Ronacher - A Python for Future Generations · EuroPython Conference · 42:43
- The Secret History of Silicon Valley · Google TechTalks · 56:32
- A Narrative History of BSD, Dr. Kirk McKusick · bsdconferences · 53:54
- "Production Prolog" by Michael Hendricks · Strange Loop Conference · 39:57
- TypeScript types can run DOOM · Michigan TypeScript · 6:59
- C++Now 2018: Matt Godbolt “What Else Has My Compiler Done For Me Lately?” · CppNow · 1:30:03
- CppCon 2017: Matt Godbolt “What Has My Compiler Done for Me Lately? Unbolting the Compiler's Lid” · CppCon · 1:15:45
- The economics of open source by C J Silverio | JSConf EU 2019 · JSConf · 37:43
- What is in a Lucene index? Adrien Grand, Software Engineer, Elasticsearch · LuceneSolrRevolution · 1:04:30
- Autonomous Testing and the Future of Software Development - Will Wilson · The Linux Foundation · 32:01