NYC Tech Talk Series: How Google Backs Up the Internet
Channel:Google TechTalks · Length:1:14:18 · Published:2014-01-31 · Views:40807
Summary
A Google Tech Talk by Raymond Blum on how Google backs up massive amounts of data such as Gmail and Picasa. It covers scaling challenges, failure types, and approaches that did not work, including tape.
Why HN recommends it
Cited in backup and data-loss threads as a rare inside look at Google's backup practices, including its use of tape.
Backups are a tax you pay for the luxury of restore — HN, 2019
Databases & storage · Distributed systems · Cloud, DevOps & reliability
Subtitles (excerpt)
RAYMOND BLUM: Hi, everybody. So I'm Raymond. I'm not television's Raymond Blum, that you may remember from "The Facts of Life." I'm a different Raymond Blum. Private joke. So I work in site reliability at Google in technical infrastructure storage. And we're basically here to make sure that things, well, we can't say that things don't break. That we can recover from them. Because, of course, we all know things break. Specifically, we are in charge of making sure that when you hit Send, it stays in your sent mail. When you get an email, it doesn't go away until you say it should go away. When you save something in Drive, it's really there, as we hope and expect forever. I'm going to talk about some of the things that we do to make that happen. Because the universe and Murphy and entropy all tell us that that's impossible. So it's a constant, never-ending battle to make sure things actually stick around. I'll let you read the bio. I'm not going to talk about that much. Common backup strategies that work maybe outside of Google really don't work here, because they typically scale effort with capacity and demand. So if you want twice as much data backed up, you need twice as much stuff to do it.
On Hacker News
FWIW... "Backups are a tax you pay for the luxury of restore" "How Google Backs Up the Internet" [VIDEO] Detailed notes: [LINK] — carapace, 2019-09
The "petabyte for a century" problem statement he mentions near the top is fun: can you preserve 1 PB with better-than-even odds of all the bits coming back right in a century? He wrote something about how he saw the durability situation in 2010: [LINK] . He seems to define the problem to allow for maintenance, e.g. check-and-repair operations are discussed. Broadening a little, I read it as "if … — twotwotwo, 2021-06
I also remember that talk. I haven't verified by watching again, but if anyone's interested I suspect this is the one: [VIDEO] — stegro32, 2016-11
This should serve as warning to anyone relying blindly on cheap cloud to keep themselves up and running. This also remind me of importance of offline tape backup. Google still uses tapes[1], yet some of the companies have started thinking they can eliminate tapes and move their backups to cloud. [1] [VIDEO] — drieddust, 2016-10
There may occasional bugs, but I don't know of any cases when Gmail actually lost emails. According to this talk [0], the bug referred to by that article caused some users' emails to be temporarily inaccessible, but all the emails were eventually recovered. [0]: [VIDEO] — timothya, 2014-10
Related videos
- Václav Smil at Driva Climate Investment Meeting 2019 · Nysnø Klimainvesteringer · 52:10
- Treating the Farm as an Ecosystem with Gabe Brown Part 1, The 5 Tenets of Soil Health · Living Web Farms · 2:31:00
- Should Google Go Nuclear? Clean, cheap, nuclear power (no, really) · GoogleTalksArchive · 1:32:38
- Exponential growth and epidemics · 3Blue1Brown · 8:56
- The Future of Mathematics? · Microsoft Research · 1:14:48
- That scene from War Games · Greg Granito · 4:05
- [4k, 60 fps] San Francisco, a Trip down Market Street, April 14, 1906 · Denis Shiryaev · 14:22
- Gwynne Dyer -- Geopolitics in a Hotter World · The University of British Columbia · 1:31:52
- The Real Reason Jaywalking Is A Crime (Adam Ruins Everything) · Dropout · 3:24
- Stanford Seminar - Concatenative Programming: From Ivory to Metal · Stanford Online · 1:04:44
- William Byrd on "The Most Beautiful Program Ever Written" [PWL NYC] · PapersWeLove · 1:31:05
- Michael Levin: Anatomical decision-making by cellular collectives · Thomas Ouldridge · 1:18:06
- India's Water Revolution #1: Solving the Crisis in 45 days with the Paani Foundation · Andrew Millison · 9:02
- Domain Modeling Made Functional - Scott Wlaschin · NDC Conferences · 51:39
- The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · GOTO Conferences · 42:02
- "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson · Strange Loop Conference · 40:20