Breakout Session Presented by Dan Robinson of Heap at PGConf Silicon Valley 2015
Channel:Citus Data · Length:54:53 · Published:2015-11-24 · Views:14307
Summary
Dan Robinson, then CTO of Heap, explains how Heap runs its product analytics backend on PostgreSQL and CitusDB. It covers conversion funnels, retention and cohort queries across tens of billions of events and 50 TB of data, plus Postgres and Kafka together and the indexing strategy.
Why HN recommends it
HN commenters point to it as a first-hand overview of Heap's Postgres/Citus architecture and its distinctive indexing approach.
Dan from Heap Analytics has a great talk about how they use Postgres and Citus. — HN, 2016
Databases & storage · Distributed systems · Data science & statistics
Subtitles (excerpt)
um so I'm Dan I'm here to talk about Heap and how it works under the hood and why it presents a particularly nasty data problem and how we use postgress to power all the analytical features in our product so a little bit about me I joined Heap in July of 2013 uh previous uh before that I was at paler doing various backend analysis things uh before that I was at Stanford where I studied math and computer science and did a bunch of research on machine learning which has nothing to do with this talk at all um so briefly uh we'll cover what Heap is and why the particular way we've defined uh our web analytics product makes this a particularly difficult one to build and uh we'll also talk about how the how the queries and the read path is distributed so that we can power with more than a single node uh then we'll talk about how we make those individual subqueries how we make the local queries fast using various postest tricks and then uh we'll discuss how the all the distributed Systems Operations uh work to keep all the data in sync so I'll start with a brief overview of what Heap is and how the product works in a basic way so Heap is a web and iOS Analytics tool uh the goal of this is to help you understand
On Hacker News
I was just watching this[0] excellent talk from one of the Heap Analytics engineers on their use of CitusDB + Postgres to do event processing. What use-case would TrailDB be the obvious, hands-down way to go (vs maybe Postgres + Citus)? [0]: [VIDEO] — Judson, 2016-05
You need a queue in front of your database irregardless of write latency. Otherwise you tie your availability to database availability as downtime (even for upgrades) is often unavoidable and network problems are common. Dan gave a pretty good talk about the high-level details of how/why postgres a couple of years back: [VIDEO] . One reason of why pg is that SQL is really powerful for building co… — macobo, 2017-06
Regarding a), a sharded postgresql (i.e. with citus data) can easily accomodate that workflow with just a tiny bit of extra overhead. Re c) this strongly depends on the usecase, I've seen companies use a) to avoid split-brain problems and having to manage two data pipelines to great success at similar scales. You might find [VIDEO] interesting. — macobo, 2017-01
Keep in mind that Heap collects a huge amount of data and has a huge dataset. I like this[1] talk, it gives you an overview of their architecture. Great post though, thank you for sharing. [1] [VIDEO] — simiano, 2016-09
If you haven't already seen it, Dan from Heap Analytics has a great talk about how they use Postgres and Citus. [VIDEO] — devbug, 2016-08
Related videos
- The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · GOTO Conferences · 42:02
- A Better Default Colormap for Matplotlib | SciPy 2015 | Nathaniel Smith and Stéfan van der Walt · Enthought · 19:08
- "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson · Strange Loop Conference · 40:20
- The Value of Values with Rich Hickey · InfoQ · 31:43
- AWS re:Invent 2018: Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB (DAT401) · Amazon Web Services · 59:56
- Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp · Erlang Solutions · 49:15
- "Turning the database inside out with Apache Samza" by Martin Kleppmann · Strange Loop Conference · 47:42
- Is Most Published Research Wrong? · Veritasium · 12:22
- The Language of the System - Rich Hickey · ClojureTV · 1:02:49
- Google I/O 2012 - Go Concurrency Patterns · Google for Developers · 51:27
- Conflict Resolution for Eventual Consistency • Martin Kleppmann • GOTO 2016 · GOTO Conferences · 51:35
- "Unison: a new distributed programming language" by Paul Chiusano · Strange Loop Conference · 41:05
- 01 - Course Introduction & Relational Model (CMU Databases Systems / Fall 2019) · CMU Database Group · 1:06:44
- How not to be ignorant about the world | Hans and Ola Rosling · TED · 19:09
- I don't like notebooks.- Joel Grus (Allen Institute for Artificial Intelligence) · O'Reilly · 56:12
- Paxos lecture (Raft user study) · Diego Ongaro · 1:06:33