dotScale 2015 - John Graham-Cumming - I got 10 trillion problems, but logging ain't one

Channel:dotconferences · Length:25:17 · Published:2015-06-25 · Views:6791

Watch on YouTube

Summary

John Graham-Cumming of Cloudflare talks at dotScale 2015 about handling roughly 10 trillion log lines a month. The talk touches on Cloudflare's hardware, its logging stack and its use of Lua in Nginx.

Why HN recommends it

Mostly cited as a source for details of Cloudflare's architecture, such as LuaJIT-based request inspection and request volumes.

I discovered a moderately old video from CloudFlare hiding in there, and had a rather big TIL moment. — HN, 2016

Distributed systems · Networking & internet infrastructure · Cloud, DevOps & reliability

Subtitles (excerpt)

[Applause] [Music] [Applause] [Music] [Applause] so um I work at a company called Cloud flare which is a startup or it's getting to the point now where it's so large we shouldn't be calling a startup um and we have a logging well we did have a logging problem and now we don't have one but we do have 10 trillion other problems which are our log lines and to sort of give you an idea of what cloud flare does we have three offices one in London one in in San Francisco we have San Jose on here because we have servers there and one in Singapore but we operate thousands of physical machines um despite being a cloud service we actually don't use the cloud we use Hardware uh all over the world so we have data centers all over um the United States North America uh Europe Far East and then uh obviously in the southern hemisphere so South America in Africa and in australasia and those data centers are where you go if you visit quite a lot of popular websites so for example I imagine some of you have been to Reddit uh when you actually go to Reddit with your browser you hit one of those data centers and cloudflare looks at the data to see if you're an attacker and it accelerates it so it'll choose an appropriate protocol depending on your device it cach data um so

On Hacker News

I discovered a moderately old video from CloudFlare hiding in there, and had a rather big TIL moment. It's one thing to read "LuaJIT is one of the fastest dynamic languages" etc blah blah on [LINK] ... ...it's quite another to learn that EVERY SINGLE CLOUDFLARE REQUEST - all 4 million/sec of them - goes through a LuaJIT-based attack detection system. Oh. Video: [VIDEO] (from [LINK] ) Found in the… — i336_, 2016-01
> Even putting aside that the computer tech part of the business isn't their core competency, custom hardware is hard. Hmmm. Good point. (Maybe I scared OP away, heh.) > Is there an example of any large company, other than Google, who has mastered this? Well, I certainly don't know. I've heard Cloudflare gets their systems made, for some definition of "from scratch", by Quanta. Presumably there's… — exikyut, 2018-06
For reference, two years ago Cloudflare was serving more than three million requests per second[0]. I can only imagine this figure has gone up a lot. 0: [VIDEO] — Artemis2, 2017-03
You make incredibly fair points and I don't have the data to answer what you've raised. This could be lack of research or the fact that CF hasn't released it. I expect a 20/80 non-researched/non-available mix. My understanding is that the WAF detects trivial stuff like SQL injection and XSS, and the video also mentions it detected ShellShock and so forth (yeah, this was a while ago). "Thousands o… — i336_, 2016-04
Rewriting the hodge podge of C, Perl and other custom bits of Nginx almost entirely in Lua and hiring its maintainer Yichun in the process where important steps for scaling up the CloudFlare's CDN and make sure each node is running as hot as possible. More on this from dotScale 2015: [VIDEO] — mtourne, 2015-07

Related videos