dotScale 2015 - John Graham-Cumming - I got 10 trillion problems, but logging ain't one
Channel:dotconferences · Length:25:17 · Published:2015-06-25 · Views:6791
Summary
John Graham-Cumming of Cloudflare talks at dotScale 2015 about handling roughly 10 trillion log lines a month. The talk touches on Cloudflare's hardware, its logging stack and its use of Lua in Nginx.
Why HN recommends it
Mostly cited as a source for details of Cloudflare's architecture, such as LuaJIT-based request inspection and request volumes.
I discovered a moderately old video from CloudFlare hiding in there, and had a rather big TIL moment. — HN, 2016
Distributed systems · Networking & internet infrastructure · Cloud, DevOps & reliability
Subtitles (excerpt)
[Applause] [Music] [Applause] [Music] [Applause] so um I work at a company called Cloud flare which is a startup or it's getting to the point now where it's so large we shouldn't be calling a startup um and we have a logging well we did have a logging problem and now we don't have one but we do have 10 trillion other problems which are our log lines and to sort of give you an idea of what cloud flare does we have three offices one in London one in in San Francisco we have San Jose on here because we have servers there and one in Singapore but we operate thousands of physical machines um despite being a cloud service we actually don't use the cloud we use Hardware uh all over the world so we have data centers all over um the United States North America uh Europe Far East and then uh obviously in the southern hemisphere so South America in Africa and in australasia and those data centers are where you go if you visit quite a lot of popular websites so for example I imagine some of you have been to Reddit uh when you actually go to Reddit with your browser you hit one of those data centers and cloudflare looks at the data to see if you're an attacker and it accelerates it so it'll choose an appropriate protocol depending on your device it cach data um so
On Hacker News
I discovered a moderately old video from CloudFlare hiding in there, and had a rather big TIL moment. It's one thing to read "LuaJIT is one of the fastest dynamic languages" etc blah blah on [LINK] ... ...it's quite another to learn that EVERY SINGLE CLOUDFLARE REQUEST - all 4 million/sec of them - goes through a LuaJIT-based attack detection system. Oh. Video: [VIDEO] (from [LINK] ) Found in the… — i336_, 2016-01
> Even putting aside that the computer tech part of the business isn't their core competency, custom hardware is hard. Hmmm. Good point. (Maybe I scared OP away, heh.) > Is there an example of any large company, other than Google, who has mastered this? Well, I certainly don't know. I've heard Cloudflare gets their systems made, for some definition of "from scratch", by Quanta. Presumably there's… — exikyut, 2018-06
For reference, two years ago Cloudflare was serving more than three million requests per second[0]. I can only imagine this figure has gone up a lot. 0: [VIDEO] — Artemis2, 2017-03
You make incredibly fair points and I don't have the data to answer what you've raised. This could be lack of research or the fact that CF hasn't released it. I expect a 20/80 non-researched/non-available mix. My understanding is that the WAF detects trivial stuff like SQL injection and XSS, and the video also mentions it detected ShellShock and so forth (yeah, this was a while ago). "Thousands o… — i336_, 2016-04
Rewriting the hodge podge of C, Perl and other custom bits of Nginx almost entirely in Lua and hiring its maintainer Yichun in the process where important steps for scaling up the CloudFlare's CDN and make sure each node is running as hot as possible. More on this from dotScale 2015: [VIDEO] — mtourne, 2015-07
Related videos
- 8088 MPH by Hornet + CRTC + DESiRE (final version) · andrewmjenner · 8:33
- Ivan Sutherland Sketchpad Demo 1963 · Interface Studies · 19:19
- The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · GOTO Conferences · 42:02
- "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson · Strange Loop Conference · 40:20
- Velocity NY 2013: Richard Cook, "Resilience In Complex Adaptive Systems" · O'Reilly · 19:09
- Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017 · GOTO Conferences · 52:34
- AWS re:Invent 2018: Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB (DAT401) · Amazon Web Services · 59:56
- Harder Drive: Hard drives we didn't want or need · suckerpinch · 36:46
- Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp · Erlang Solutions · 49:15
- "Turning the database inside out with Apache Samza" by Martin Kleppmann · Strange Loop Conference · 47:42
- The Language of the System - Rich Hickey · ClojureTV · 1:02:49
- This Video Is Sponsored By ███ VPN · Tom Scott · 7:25
- Google I/O 2012 - Go Concurrency Patterns · Google for Developers · 51:27
- Literate Devops with Emacs · Howard Abrams · 28:36
- Conflict Resolution for Eventual Consistency • Martin Kleppmann • GOTO 2016 · GOTO Conferences · 51:35
- "Unison: a new distributed programming language" by Paul Chiusano · Strange Loop Conference · 41:05