AWS re:Invent 2014 | (SPOT302) Under the Covers of AWS: Its Core Distributed Systems
Channel:Amazon Web Services · Length:46:14 · Published:2014-11-18 · Views:10337
Summary
An AWS re:Invent 2014 talk on the core distributed systems primitives behind AWS and Amazon.com, such as consensus, key-value stores and transactional logging. It describes how AWS builds large services from a few common building blocks.
Why HN recommends it
HN points to it as insight into how AWS thinks about distributed systems and the difficulty of consensus.
That presentation is full of great insight into how AWS thinks about distributed systems. — HN, 2022
Distributed systems · Cloud, DevOps & reliability
On Hacker News
> “There are only three modes for a distributed system - it implements paxos/raft itself, it relies on a data store that implements paxos/raft or it’s wrong.” That sounds like something Al Vermeulen would say. Though my recollection of the quote is more along the lines of: "There are three types of consistent distributed systems: paxos, broken protocols, and single points of failures." I'm guessi… — rubiquity, 2022-05
> But I did not expect something as core as the service managing the leases on the physical EC2 nodes to not have recovery procedure. I guess they don't have a recovery procedure for the "congestive collapse" edge case. I have seen something similar, so I wouldn't be frowning at this. A couple of red flags though: 1. Apparent lack of load-shedding support by this DWFM, such that a server reboot h… — throwdbaaway, 2025-10
I just watched a talk on this the other day (assuming it's same person since he's saying the same thing) the other day: [VIDEO] — dtjohnnymonkey, 2022-08
> Basic Paxos, is well, too basic and people mostly run modifications of this to get higher throughput and better latencies. After those modifications, it does not look very different from Raft. Alan Vermeulen, one of the founding AWS engineers, calls inventing newer solutions to distributed consensus an exercise in re-discovering Paxos. [VIDEO] — ignoramous, 2021-07
> A few things around S3 were unique. We launched with a set of ten distributed systems tenets in the press release. The ten tenets were separate from S3, stating that this is how you would want to build distributed systems at scale. We just demonstrated that S3 is a really good example of applying those skills. In my own naive way I believe that AWS, today, probably embodies James Hamilton's vis… — ignoramous, 2021-03
Related videos
- The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · GOTO Conferences · 42:02
- "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson · Strange Loop Conference · 40:20
- Velocity NY 2013: Richard Cook, "Resilience In Complex Adaptive Systems" · O'Reilly · 19:09
- Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017 · GOTO Conferences · 52:34
- AWS re:Invent 2018: Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB (DAT401) · Amazon Web Services · 59:56
- Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp · Erlang Solutions · 49:15
- "Turning the database inside out with Apache Samza" by Martin Kleppmann · Strange Loop Conference · 47:42
- The Language of the System - Rich Hickey · ClojureTV · 1:02:49
- Google I/O 2012 - Go Concurrency Patterns · Google for Developers · 51:27
- Literate Devops with Emacs · Howard Abrams · 28:36
- Conflict Resolution for Eventual Consistency • Martin Kleppmann • GOTO 2016 · GOTO Conferences · 51:35
- "Unison: a new distributed programming language" by Paul Chiusano · Strange Loop Conference · 41:05
- Paxos lecture (Raft user study) · Diego Ongaro · 1:06:33
- Hewitt, Meijer and Szyperski: The Actor Model (everything you wanted to know...) · jasonofthel33t · 42:34
- Containers From Scratch • Liz Rice • GOTO 2018 · GOTO Conferences · 42:53
- Over 200,000 Servers in One Place! Visiting Hetzner in Falkenstein (Germany) · der8auer EN · 20:29