AWS re:Invent 2018: Amazon DynamoDB Under the Hood: How We Built a Hyper-Scale Database (DAT321)
Channel:Amazon Web Services · Length:53:42 · Published:2018-11-27 · Views:112232
Summary
An AWS re:Invent 2018 session on how DynamoDB was built as a hyper-scale NoSQL database. It discloses underpinnings of the service and how it is run for over 100,000 customers.
Why HN recommends it
HN points to it as an authoritative look at DynamoDB internals and as evidence it is a complex distributed system.
here's a talk you might find interesting from one of the folks who built and ran DDB for a long time — HN, 2022
Databases & storage · Distributed systems · Cloud, DevOps & reliability
Subtitles (excerpt)
good afternoon or good evening I'm not sure I think the Sun is set right I don't know this is one of those transition periods my name is josue Sorensen technically that's not my name my name is James Christopher Sorensen the third gesso is my Amazon given name so if you want to email me email gesso at amazon.com I'm a senior principal engineer at Amazon I've been with AWS for about 14 years as one of the original developers on s3 way back when I launched our IOT product and for the last couple of years I've been working on DynamoDB and today what I wanted to do was give a talk about under the hood kind of how dynamo works and the way I plan on approaching this is how we onboard engineers to the Dynamo team so the first thing that an engineer when they joined dynamo has to do is interact with the API they kind of take the role that you guys as customers have you know they probably haven't used dynamo you know especially that guys we hire out of college we have them do things like make a table create indexes set up streams a lot of the you know the public facing API of dynamodb and once they get done with that we move on to what this talk is going to be or the onboarding so what are the goals
On Hacker News
We don't have a paper on DynamoDB's internals (yet?), but here's a talk you might find interesting from one of the folks who built and ran DDB for a long time: [VIDEO] And Doug Terry talking through the details of how DynamoDB's transaction protocol works: [LINK] If we did publish more about the internals of DDB, what would you be looking to learn? Architecture? Operational experience? Developer … — mjb, 2022-01
Here are a couple videos from reInvent 2018: Jaso talking about DynamoDB internals [VIDEO] Marc talking about Lambda internals [LINK] — appwiz, 2019-06
Saying DynamoDB is built on top of InnoDB is a pretty big oversimplification of a much more complex distributed system[1] and for all we know they could have switched out the low level the storage engine on the backend to something like RocksDB or WiredTiger. The Aurora storage subsystem is much more limited in terms of horizontal scalability and performance, they probably chose it because it was… — talawahtech, 2019-01
DynamoDB is different from their published paper, which is mostly about designing a highly available key/value system that can be run on top of any other datastore. The paper does mention MySQL as a storage option. The recent MySQL info comes from this 2016 thread on DynamoDB storing empty strings: [LINK] Confirmed by this comment specifically: [LINK] The latest 2018 ReInvent deep dive on DynamoD… — manigandham, 2018-12
Related videos
- AWS re:Invent 2018: [REPEAT 1] A Serverless Journey: AWS Lambda Under the Hood (SRV409-R1) · Amazon Web Services · 59:12
- The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · GOTO Conferences · 42:02
- "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson · Strange Loop Conference · 40:20
- The Value of Values with Rich Hickey · InfoQ · 31:43
- Velocity NY 2013: Richard Cook, "Resilience In Complex Adaptive Systems" · O'Reilly · 19:09
- Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017 · GOTO Conferences · 52:34
- AWS re:Invent 2018: Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB (DAT401) · Amazon Web Services · 59:56
- Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp · Erlang Solutions · 49:15
- "Turning the database inside out with Apache Samza" by Martin Kleppmann · Strange Loop Conference · 47:42
- The Language of the System - Rich Hickey · ClojureTV · 1:02:49
- Google I/O 2012 - Go Concurrency Patterns · Google for Developers · 51:27
- Literate Devops with Emacs · Howard Abrams · 28:36
- Conflict Resolution for Eventual Consistency • Martin Kleppmann • GOTO 2016 · GOTO Conferences · 51:35
- "Unison: a new distributed programming language" by Paul Chiusano · Strange Loop Conference · 41:05
- 01 - Course Introduction & Relational Model (CMU Databases Systems / Fall 2019) · CMU Database Group · 1:06:44
- Paxos lecture (Raft user study) · Diego Ongaro · 1:06:33