AWS re:Invent 2014 | (SPOT302) Under the Covers of AWS: Its Core Distributed Systems

Channel:Amazon Web Services · Length:46:14 · Published:2014-11-18 · Views:10337

Watch on YouTube

Summary

An AWS re:Invent 2014 talk on the core distributed systems primitives behind AWS and Amazon.com, such as consensus, key-value stores and transactional logging. It describes how AWS builds large services from a few common building blocks.

Why HN recommends it

HN points to it as insight into how AWS thinks about distributed systems and the difficulty of consensus.

That presentation is full of great insight into how AWS thinks about distributed systems. — HN, 2022

Distributed systems · Cloud, DevOps & reliability

On Hacker News

> “There are only three modes for a distributed system - it implements paxos/raft itself, it relies on a data store that implements paxos/raft or it’s wrong.” That sounds like something Al Vermeulen would say. Though my recollection of the quote is more along the lines of: "There are three types of consistent distributed systems: paxos, broken protocols, and single points of failures." I'm guessi… — rubiquity, 2022-05
> But I did not expect something as core as the service managing the leases on the physical EC2 nodes to not have recovery procedure. I guess they don't have a recovery procedure for the "congestive collapse" edge case. I have seen something similar, so I wouldn't be frowning at this. A couple of red flags though: 1. Apparent lack of load-shedding support by this DWFM, such that a server reboot h… — throwdbaaway, 2025-10
I just watched a talk on this the other day (assuming it's same person since he's saying the same thing) the other day: [VIDEO] — dtjohnnymonkey, 2022-08
> Basic Paxos, is well, too basic and people mostly run modifications of this to get higher throughput and better latencies. After those modifications, it does not look very different from Raft. Alan Vermeulen, one of the founding AWS engineers, calls inventing newer solutions to distributed consensus an exercise in re-discovering Paxos. [VIDEO] — ignoramous, 2021-07
> A few things around S3 were unique. We launched with a set of ten distributed systems tenets in the press release. The ten tenets were separate from S3, stating that this is how you would want to build distributed systems at scale. We just demonstrated that S3 is a really good example of applying those skills. In my own naive way I believe that AWS, today, probably embodies James Hamilton's vis… — ignoramous, 2021-03

Related videos