AWS re:Invent 2018: Amazon DynamoDB Under the Hood: How We Built a Hyper-Scale Database (DAT321)

Channel:Amazon Web Services · Length:53:42 · Published:2018-11-27 · Views:112232

Watch on YouTube

Summary

An AWS re:Invent 2018 session on how DynamoDB was built as a hyper-scale NoSQL database. It discloses underpinnings of the service and how it is run for over 100,000 customers.

Why HN recommends it

HN points to it as an authoritative look at DynamoDB internals and as evidence it is a complex distributed system.

here's a talk you might find interesting from one of the folks who built and ran DDB for a long time — HN, 2022

Databases & storage · Distributed systems · Cloud, DevOps & reliability

Subtitles (excerpt)

good afternoon or good evening I'm not sure I think the Sun is set right I don't know this is one of those transition periods my name is josue Sorensen technically that's not my name my name is James Christopher Sorensen the third gesso is my Amazon given name so if you want to email me email gesso at amazon.com I'm a senior principal engineer at Amazon I've been with AWS for about 14 years as one of the original developers on s3 way back when I launched our IOT product and for the last couple of years I've been working on DynamoDB and today what I wanted to do was give a talk about under the hood kind of how dynamo works and the way I plan on approaching this is how we onboard engineers to the Dynamo team so the first thing that an engineer when they joined dynamo has to do is interact with the API they kind of take the role that you guys as customers have you know they probably haven't used dynamo you know especially that guys we hire out of college we have them do things like make a table create indexes set up streams a lot of the you know the public facing API of dynamodb and once they get done with that we move on to what this talk is going to be or the onboarding so what are the goals

On Hacker News

We don't have a paper on DynamoDB's internals (yet?), but here's a talk you might find interesting from one of the folks who built and ran DDB for a long time: [VIDEO] And Doug Terry talking through the details of how DynamoDB's transaction protocol works: [LINK] If we did publish more about the internals of DDB, what would you be looking to learn? Architecture? Operational experience? Developer … — mjb, 2022-01
Here are a couple videos from reInvent 2018: Jaso talking about DynamoDB internals [VIDEO] Marc talking about Lambda internals [LINK] — appwiz, 2019-06
Saying DynamoDB is built on top of InnoDB is a pretty big oversimplification of a much more complex distributed system[1] and for all we know they could have switched out the low level the storage engine on the backend to something like RocksDB or WiredTiger. The Aurora storage subsystem is much more limited in terms of horizontal scalability and performance, they probably chose it because it was… — talawahtech, 2019-01
DynamoDB is different from their published paper, which is mostly about designing a highly available key/value system that can be run on top of any other datastore. The paper does mention MySQL as a storage option. The recent MySQL info comes from this 2016 thread on DynamoDB storing empty strings: [LINK] Confirmed by this comment specifically: [LINK] The latest 2018 ReInvent deep dive on DynamoD… — manigandham, 2018-12

Related videos