DuckDB – The SQLite for Analytics (Mark Raasveldt, CWI)
Channel:CMU Database Group · Length:1:08:00 · Published:2020-04-27 · Views:28571
Summary
Mark Raasveldt of CWI presents DuckDB, an embeddable analytical database described as the SQLite for analytics, in a CMU Database Group talk. He explains why it was built, what it is for, and how it works.
Why HN recommends it
HN recommends it as the definitive talk from the implementer on DuckDB's motivation and design.
one of the implementers giving a talk to CMU students about what DuckDB is, why they made it, and how it works — HN, 2020
Databases & storage · Data science & statistics
Subtitles (excerpt)
So, uh we are super excited to have uh Mark Resball from CWI come give a give a talk for us today. Uh Mark did his PhD at CWI and now is halfway in between being a postdoc and halfway in between being a PhD student cuz you you haven't defended yet. Yeah. But you are officially a postdoc. Are they paying you like a postdoc? Sorry? Are they paying you They're paying me almost like a postdoc. It's uh So you are in between. I'm in I'm in between. I'm a predoc. Got it. Nice. No, no, you're you're You're pre-postdoc. A pre-postdoc. A pre-postponed doc. Postponed doc. Okay. So, uh DuckDB is probably, in my opinion, the most exciting academic database system coming out in the last 2 years. And as the title says, it's it's SQL-like for analytics. So, I don't think there's anything I need to say. I think Mark's awesome, DuckDB's awesome, so go for it. Well, thank you very much. And actually, the way we'll do this is uh people can just unmute themselves and ask Mark as as we as we go along with questions. Like there's Yeah, no problem. Yeah. So, at at any point just uh don't literally shout, but uh it is a bit late here, so I guess my neighbors won't like that, but just ask. That's fine. All right, so I guess I don't need to introduce myself.
On Hacker News
If you are a data scientist or do anything with data... duckdb is like a swiss army knife. So many great ways it can help your workflow. The original video from CMU in 2020 [1] is a classic. Minutes 3-8 present a good argument for adding duckdb to your data cleaning/processing workflow. And if you want to add a semantic layer on top of data, Malloy [2] is my favorite so far (it has duckdb built i… — mrtimo, 2026-04
Enjoyed his roundup in the "Shoving Ducks into Everything" section. DuckDB is a great tool. In April 2020, the creator of DuckDB gave a talk at CMU. In the beginning he makes a convincing argument (in 5 minutes) why data scientists don't use RDBMS and how this was the genesis of DuckDB. Here is a video that starts 3 minutes into the talk (where is argument starts): [VIDEO] — mrtimo, 2025-01
I was wondering the same--I guess because it's fairly new (2018) and it came out of a database research group at a European university, rather than a SV tech firm. Therefore, limited marketing budget. By the way, here's a YT video of a talk given by one of the DuckDB implementers about why they made it, what it's for, and how it works: [VIDEO] — kyllo, 2020-05
Here's a YT video of one of the implementers giving a talk to CMU students about what DuckDB is, why they made it, and how it works: [VIDEO] In a nutshell, it's a file-per-database embedded DB like SQLite, and it's installable as a Python or R package from PyPI or CRAN, but unlike SQLite, it's a column store with vectorized operations so it's more performant for analytical queries. — kyllo, 2020-05
I agree with this 100%. The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research, in the first 5 minutes of his talk here [1]. I've been using Malloy [2], which compiles to SQL (like Typescript compiles to Javascript), so instead of editing a 1000 line SQL script, it's only 18 lines of Malloy. I'd love to see a blog post comparing a pa… — mrtimo, 2025-09
Related videos
- A Better Default Colormap for Matplotlib | SciPy 2015 | Nathaniel Smith and Stéfan van der Walt · Enthought · 19:08
- "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson · Strange Loop Conference · 40:20
- The Value of Values with Rich Hickey · InfoQ · 31:43
- AWS re:Invent 2018: Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB (DAT401) · Amazon Web Services · 59:56
- "Turning the database inside out with Apache Samza" by Martin Kleppmann · Strange Loop Conference · 47:42
- Is Most Published Research Wrong? · Veritasium · 12:22
- Conflict Resolution for Eventual Consistency • Martin Kleppmann • GOTO 2016 · GOTO Conferences · 51:35
- 01 - Course Introduction & Relational Model (CMU Databases Systems / Fall 2019) · CMU Database Group · 1:06:44
- How not to be ignorant about the world | Hans and Ola Rosling · TED · 19:09
- I don't like notebooks.- Joel Grus (Allen Institute for Artificial Intelligence) · O'Reilly · 56:12
- VisiData Lightning Demo at PyCascades 2018 · Saul Pwanson · 4:50
- Introduction to NoSQL • Martin Fowler • GOTO 2012 · GOTO Conferences · 54:51
- Emacs + org-mode + python in reproducible research; SciPy 2013 Presentation · Enthought · 21:16
- I Like Notebooks · Jeremy Howard · 52:09
- Rich Hickey rants about HttpServletRequest in 1080p: Death By Specificity from Clojure Made Simple · Fred Overflow · 4:56
- TigerStyle! (Or How To Design Safer Systems in Less Time) by Joran Dirk Greef · TigerBeetle · 1:01:59