Zach Tellman - Everything Will Flow

Channel:ClojureTV · Length:40:12 · Published:2015-04-22 · Views:26585

Watch on YouTube

Summary

Zach Tellman's Clojure West talk covers the basics of queueing theory and its practical use in systems, including backpressure, how queues break down under load, and pitfalls of naive core.async use.

Why HN recommends it

HN recommends it as a fascinating explanation of queues, flow control and overload that applies well beyond Clojure.

He talks about queues and how they break down under load. — HN, 2017

Functional programming · Software performance & optimization · Distributed systems

Subtitles (excerpt)

so we're here to talk about cues uh and you know some of you may have wondered uh why uh cues really are pretty simple right you have a producer that Ines messages you have a consumer that dqes messages typically but not always in the order that they were added and that's about it uh but something that is maybe non obvious is that in addition to the active ining being a side effect right we get back to just confirmation that was added because what we're getting back is not a function of the data uh the data itself can only describe a side effect right Q's are a way of dealing with actions and for this reason I think in the closure Community it's always been a bit of a sideline right you know Java provides us with cues we use them occasionally we abstract over them where possible but we don't really uh deal with them headon or at least not until we had core async which brought it sort of more into the central conversation which I think is a good thing because we of course are in the business of making computers do things right we don't just raise the ambient temperature of the room and so looking under the covers uh a coring channel has not one but three cues it has a buffer that holds the messages but it

On Hacker News

Yeah this is true, typically you want to keep blocking functions outside this. By default, core.async allocates as many threads as cores in your CPU. In addition to this, I personally find core.async to deal poorly with flow control (e.g. slow subscribers slowing down the entire flow for every subscriber), and seems to not have a lot of telemetry / it’s difficult to find out what’s going on in th… — stingraycharles, 2019-02
No. Check out this video by Zach Tellman. He talks about queues and how they break down under load. One of the least intuitive things he points out is that when you have more processors, the breakdown tends to be more of a step function: everything is running smoothly till the moment that it isn't. [VIDEO] The point he makes arises from basic queue theory and is applicable to all kinds of systems… — lkrubner, 2017-08
On dealing with backpressure, I would recommend what Zach Tellman writes about in his video "Everything will flow": [VIDEO] Tellman also tells his own adventure about a system something like this: "a system that would latch up into an overload state because in overload it would log errors at a severity high enough to force the log file to be flushed on every line" He talks about some problems he … — susan_hall, 2016-08
Anything by Zach Tellman: [LINK] [LINK] In terms of his ideas, watch "Always be composing": [LINK] And he offers some great thoughts about queues and backpressure in "Everything will flow": [VIDEO] — lkrubner, 2016-03
See also a very nice video from ClojureWest about queues in system architectures: [VIDEO] The queue are everywhere - your messaging queue, the threadpool, the hardware threads, and other layers of the stack and APIs you use. The video adds the interesting detail that as you add more tellers (workers) you learn of impending disaster only in the outlier p99 (or higher) latencies; by the time your p… — cabacon, 2015-06

Related videos