NYC Tech Talk Series: How Google Backs Up the Internet

Channel:Google TechTalks · Length:1:14:18 · Published:2014-01-31 · Views:40807

Watch on YouTube

Summary

A Google Tech Talk by Raymond Blum on how Google backs up massive amounts of data such as Gmail and Picasa. It covers scaling challenges, failure types, and approaches that did not work, including tape.

Why HN recommends it

Cited in backup and data-loss threads as a rare inside look at Google's backup practices, including its use of tape.

Backups are a tax you pay for the luxury of restore — HN, 2019

Databases & storage · Distributed systems · Cloud, DevOps & reliability

Subtitles (excerpt)

RAYMOND BLUM: Hi, everybody. So I'm Raymond. I'm not television's Raymond Blum, that you may remember from "The Facts of Life." I'm a different Raymond Blum. Private joke. So I work in site reliability at Google in technical infrastructure storage. And we're basically here to make sure that things, well, we can't say that things don't break. That we can recover from them. Because, of course, we all know things break. Specifically, we are in charge of making sure that when you hit Send, it stays in your sent mail. When you get an email, it doesn't go away until you say it should go away. When you save something in Drive, it's really there, as we hope and expect forever. I'm going to talk about some of the things that we do to make that happen. Because the universe and Murphy and entropy all tell us that that's impossible. So it's a constant, never-ending battle to make sure things actually stick around. I'll let you read the bio. I'm not going to talk about that much. Common backup strategies that work maybe outside of Google really don't work here, because they typically scale effort with capacity and demand. So if you want twice as much data backed up, you need twice as much stuff to do it.

On Hacker News

FWIW... "Backups are a tax you pay for the luxury of restore" "How Google Backs Up the Internet" [VIDEO] Detailed notes: [LINK] — carapace, 2019-09
The "petabyte for a century" problem statement he mentions near the top is fun: can you preserve 1 PB with better-than-even odds of all the bits coming back right in a century? He wrote something about how he saw the durability situation in 2010: [LINK] . He seems to define the problem to allow for maintenance, e.g. check-and-repair operations are discussed. Broadening a little, I read it as "if … — twotwotwo, 2021-06
I also remember that talk. I haven't verified by watching again, but if anyone's interested I suspect this is the one: [VIDEO] — stegro32, 2016-11
This should serve as warning to anyone relying blindly on cheap cloud to keep themselves up and running. This also remind me of importance of offline tape backup. Google still uses tapes[1], yet some of the companies have started thinking they can eliminate tapes and move their backups to cloud. [1] [VIDEO] — drieddust, 2016-10
There may occasional bugs, but I don't know of any cases when Gmail actually lost emails. According to this talk [0], the bug referred to by that article caused some users' emails to be temporarily inaccessible, but all the emails were eventually recovered. [0]: [VIDEO] — timothya, 2014-10

Related videos