Dev Deletes Entire Production Database, Chaos Ensues
Channel:Kevin Fang · Length:10:20 · Published:2023-04-26 · Views:3646384
Summary
Kevin Fang comically retells GitLab's 2017 incident in which an engineer deleted the production database, based on the company's postmortems. It walks through the confusion and failed backups.
Why HN recommends it
Shared as a funny, accurate retelling of a classic outage, especially on a bad ops day.
The reenactment by Kevin Fang is always a good source of comedy on a bad day. — HN, 2024
Databases & storage · Cloud, DevOps & reliability · Satire & parody
Subtitles (excerpt)
At 6 PM UTC, a beautiful day in Silicon Valley, work was going on as usual at GitLab HQ. Just kidding, GitLab is fully remote. GitLab is a code sharing platform just like GitHub except open source. The main character of this story, team-member-1 was about to sign off for the day, unaware of the disaster that was ahead. He was working in UTC time, so one of these countries. Unfortunately his evening was interrupted at 6 PM when he was paged in for high database load. After scrolling through the dashboards, he saw that the total number of snippets was rising dramatically, and assumed the root cause was spam. This was a fair assumption as GitLab had been experiencing similar spam issues in the past week, just less severe. Over the hours the team attempted to block certain spam IPs as well as delete spam users. By 9 pm, they were alerted to the database having an elevated number of locks. Whenever write transactions are made to a record, it will enforce a lock, forcing further writes to the record to wait for the first one to finish. This ensures that the writes do not interfere with each other. More spam, more writes, more locks, more latency. The engineers continue to search for other sources
On Hacker News
Every time GL is mentioned, I am reminded of the 2017 incident where a person 'rm -rf' the primary pg data logs. The reenactment by Kevin Fang is always a good source of comedy on a bad day. [VIDEO] It’s a shame they are selling out, but I suppose it was bound to happen sooner or later. GH is vacuuming up all of the market share. I kind of liked the ability to self host my own GL. But for small/p… — xyst, 2024-08
"When a program has nothing surprising to say, it should say nothing." Which sounds good in theory, until you run into issues like this: [VIDEO] TLDW: Most of the Gitlabs outage where they deleted the primary prod and backup prod database could have been solved if they just waited for the command line op to complete - but because of a lack of feedback they thought it had frozen, leading to severa… — laserDinosaur, 2023-09
Here is a retelling of the time when gitlab went down because someone deleted a production database: "Dev Deletes Entire Production Database, Chaos Ensues" [VIDEO] — sovietmudkipz, 2023-07
If the primary fails, will the replica fail over? How do you know? Are the backups working? Who's checking? Can anyone delete a directory when tired at 6PM on a Friday? [VIDEO] — renegade-otter, 2024-04
Related videos
- Pig Butchering Scams: Last Week Tonight with John Oliver (HBO) · LastWeekTonight · 24:27
- "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson · Strange Loop Conference · 40:20
- The Value of Values with Rich Hickey · InfoQ · 31:43
- Velocity NY 2013: Richard Cook, "Resilience In Complex Adaptive Systems" · O'Reilly · 19:09
- Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017 · GOTO Conferences · 52:34
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- AWS re:Invent 2018: Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB (DAT401) · Amazon Web Services · 59:56
- The Loving Trap · Ben Woodhams · 2:58
- "Turning the database inside out with Apache Samza" by Martin Kleppmann · Strange Loop Conference · 47:42
- Literate Devops with Emacs · Howard Abrams · 28:36
- Conflict Resolution for Eventual Consistency • Martin Kleppmann • GOTO 2016 · GOTO Conferences · 51:35
- NaN Gates and Flip FLOPS · suckerpinch · 19:07
- 01 - Course Introduction & Relational Model (CMU Databases Systems / Fall 2019) · CMU Database Group · 1:06:44
- Erlang The Movie II: The Sequel · gar1t · 9:55
- On The Turing Completeness of PowerPoint (SIGBOVIK) · Tom Wildenhain · 5:33
- Chicken chicken chicken · Yoram Bauman · 4:06