Breakout Session Presented by Dan Robinson of Heap at PGConf Silicon Valley 2015

Channel:Citus Data · Length:54:53 · Published:2015-11-24 · Views:14307

Watch on YouTube

Summary

Dan Robinson, then CTO of Heap, explains how Heap runs its product analytics backend on PostgreSQL and CitusDB. It covers conversion funnels, retention and cohort queries across tens of billions of events and 50 TB of data, plus Postgres and Kafka together and the indexing strategy.

Why HN recommends it

HN commenters point to it as a first-hand overview of Heap's Postgres/Citus architecture and its distinctive indexing approach.

Dan from Heap Analytics has a great talk about how they use Postgres and Citus. — HN, 2016

Databases & storage · Distributed systems · Data science & statistics

Subtitles (excerpt)

um so I'm Dan I'm here to talk about Heap and how it works under the hood and why it presents a particularly nasty data problem and how we use postgress to power all the analytical features in our product so a little bit about me I joined Heap in July of 2013 uh previous uh before that I was at paler doing various backend analysis things uh before that I was at Stanford where I studied math and computer science and did a bunch of research on machine learning which has nothing to do with this talk at all um so briefly uh we'll cover what Heap is and why the particular way we've defined uh our web analytics product makes this a particularly difficult one to build and uh we'll also talk about how the how the queries and the read path is distributed so that we can power with more than a single node uh then we'll talk about how we make those individual subqueries how we make the local queries fast using various postest tricks and then uh we'll discuss how the all the distributed Systems Operations uh work to keep all the data in sync so I'll start with a brief overview of what Heap is and how the product works in a basic way so Heap is a web and iOS Analytics tool uh the goal of this is to help you understand

On Hacker News

I was just watching this[0] excellent talk from one of the Heap Analytics engineers on their use of CitusDB + Postgres to do event processing. What use-case would TrailDB be the obvious, hands-down way to go (vs maybe Postgres + Citus)? [0]: [VIDEO] — Judson, 2016-05
You need a queue in front of your database irregardless of write latency. Otherwise you tie your availability to database availability as downtime (even for upgrades) is often unavoidable and network problems are common. Dan gave a pretty good talk about the high-level details of how/why postgres a couple of years back: [VIDEO] . One reason of why pg is that SQL is really powerful for building co… — macobo, 2017-06
Regarding a), a sharded postgresql (i.e. with citus data) can easily accomodate that workflow with just a tiny bit of extra overhead. Re c) this strongly depends on the usecase, I've seen companies use a) to avoid split-brain problems and having to manage two data pipelines to great success at similar scales. You might find [VIDEO] interesting. — macobo, 2017-01
Keep in mind that Heap collects a huge amount of data and has a huge dataset. I like this[1] talk, it gives you an overview of their architecture. Great post though, thank you for sharing. [1] [VIDEO] — simiano, 2016-09
If you haven't already seen it, Dan from Heap Analytics has a great talk about how they use Postgres and Citus. [VIDEO] — devbug, 2016-08

Related videos