AWS re:Invent 2018: How AWS Minimizes the Blast Radius of Failures (ARC338)

Channel:Amazon Web Services · Length:55:43 · Published:2018-11-27 · Views:36689

Watch on YouTube

Summary

An AWS re:Invent 2018 talk on how AWS limits the blast radius of failures. It covers design techniques such as cell-based architecture and shuffle sharding.

Why HN recommends it

It is the go-to reference whenever HN discusses cell-based architecture and outage isolation, as an explanation straight from AWS engineers.

This YouTube video[1] of a re:invent presentation does a great job of explaining it. — HN, 2020

Distributed systems · Cloud, DevOps & reliability

Subtitles (excerpt)

good afternoon thanks for coming to this session who's here at reinvent for the first time awesome welcome should be an exciting week lots of sessions lots of parties lots of fun so I am Peter Voss all I am a distinguished engineer at AWS and today I'm going to talk about how AWS minimizes the blast radius of failures so I spent my career focused on building highly available large-scale distributed systems and one of the things I've learned and something you learn quickly in this field is that there are some fundamental properties of the universe that you basically have to contend with the first is the speed of light so 186 miles per millisecond is pretty darn fast but as the folks at JPL this morning trying to land their their Mars rover learned it really gets in the way when you travel any long distance communications by the way they did land the rover successfully so congratulations to our our friends at JPL a happy Amazon customer a tip us customer the next thing is sort of this pesky property universe known as the cap theorem so this is a an observation you can't Simas simultaneously have consistency and availability in a distributed system that is built to be partitioned tolerant so this was first postulated by Eric Brewer in

On Hacker News

Multi-AZ doesn't protect against a software/OS issue like this, Multi-AZ would be relevant if it was an infrastructure failure (e.g. underlying EC2 instances or networking). The relevant resiliency pattern in this case would be what they refer to as cell-based architecture, where within an AZ services are broken down into smaller independent cells to minimize the blast radius. They specifically m… — talawahtech, 2020-11
This is a great outline. Martin Kleppmann's Designing Data Intensive Applications [0] is very much in the same domain as what's in these notes. Speaking about scale and distributed systems: AWS specific resources I find interesting from their EdgeEngineering/NetEng teams (I haven't seen many other service teams in AWS openly share as much about design as them): - [LINK] series of articles on Rout… — ignoramous, 2019-03
From what you describe the 'AWS cells' are very different than Borg's. My view is that at a high level aws cells are isolated, fuzzy-partitioned, migratable, heterogeneous instances of a complete service stack that are managed such that they always fail independently of each other; contact points among these cell instances, if required, are established via highly available intermediate services (… — ignoramous, 2019-03
ex-AWS here May be marketing but it is an architecture born out of Amazon's (and AWS's) use of AWS: - Reliable scalability: How Amazon.com scales in the cloud , [LINK] (2022) - How AWS minimizes the blast radius of failures , [VIDEO] (2018) For massive enterprise products like Slack that need close to 100% uptime across all their services, cells make sense. — ignoramous, 2023-08
AWS invested a ton into limiting the blast radius of failures by isolating AZs, regions and using a cellular (service level isolated shards) architecture. I am surprised these ideas have not propagated to newer companies trying to build clouds: [VIDEO] AWS isn’t perfect but these lessons were learned by fire because these sorts of global outages can seriously harm reputations. — faizshah, 2023-03

Related videos