Zach Tellman - Everything Will Flow
Channel:ClojureTV · Length:40:12 · Published:2015-04-22 · Views:26585
Summary
Zach Tellman's Clojure West talk covers the basics of queueing theory and its practical use in systems, including backpressure, how queues break down under load, and pitfalls of naive core.async use.
Why HN recommends it
HN recommends it as a fascinating explanation of queues, flow control and overload that applies well beyond Clojure.
He talks about queues and how they break down under load. — HN, 2017
Functional programming · Software performance & optimization · Distributed systems
Subtitles (excerpt)
so we're here to talk about cues uh and you know some of you may have wondered uh why uh cues really are pretty simple right you have a producer that Ines messages you have a consumer that dqes messages typically but not always in the order that they were added and that's about it uh but something that is maybe non obvious is that in addition to the active ining being a side effect right we get back to just confirmation that was added because what we're getting back is not a function of the data uh the data itself can only describe a side effect right Q's are a way of dealing with actions and for this reason I think in the closure Community it's always been a bit of a sideline right you know Java provides us with cues we use them occasionally we abstract over them where possible but we don't really uh deal with them headon or at least not until we had core async which brought it sort of more into the central conversation which I think is a good thing because we of course are in the business of making computers do things right we don't just raise the ambient temperature of the room and so looking under the covers uh a coring channel has not one but three cues it has a buffer that holds the messages but it
On Hacker News
Yeah this is true, typically you want to keep blocking functions outside this. By default, core.async allocates as many threads as cores in your CPU. In addition to this, I personally find core.async to deal poorly with flow control (e.g. slow subscribers slowing down the entire flow for every subscriber), and seems to not have a lot of telemetry / it’s difficult to find out what’s going on in th… — stingraycharles, 2019-02
No. Check out this video by Zach Tellman. He talks about queues and how they break down under load. One of the least intuitive things he points out is that when you have more processors, the breakdown tends to be more of a step function: everything is running smoothly till the moment that it isn't. [VIDEO] The point he makes arises from basic queue theory and is applicable to all kinds of systems… — lkrubner, 2017-08
On dealing with backpressure, I would recommend what Zach Tellman writes about in his video "Everything will flow": [VIDEO] Tellman also tells his own adventure about a system something like this: "a system that would latch up into an overload state because in overload it would log errors at a severity high enough to force the log file to be flushed on every line" He talks about some problems he … — susan_hall, 2016-08
Anything by Zach Tellman: [LINK] [LINK] In terms of his ideas, watch "Always be composing": [LINK] And he offers some great thoughts about queues and backpressure in "Everything will flow": [VIDEO] — lkrubner, 2016-03
See also a very nice video from ClojureWest about queues in system architectures: [VIDEO] The queue are everywhere - your messaging queue, the threadpool, the hardware threads, and other layers of the stack and APIs you use. The video adds the interesting detail that as you add more tellers (workers) you learn of impending disaster only in the outlier p99 (or higher) latencies; by the time your p… — cabacon, 2015-06
Related videos
- The Value of Values with Rich Hickey · InfoQ · 31:43
- "Agility & Robustness: Clojure spec" by Stuart Halloway · Strange Loop Conference · 39:59
- Ruby Conf 12 - Y Not- Adventures in Functional Programming by Jim Weirich · Confreaks · 53:21
- How Python was Shaped by leaky Internals, Armin Ronacher, Flask framework · Видео с мероприятий {speach! · 49:55
- The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · GOTO Conferences · 42:02
- "Simple Made Easy" - Rich Hickey (2011) · Strange Loop Conference · 1:01:38
- "Making Impossible States Impossible" by Richard Feldman · elm-conf · 25:06
- CppCon 2015: Andrei Alexandrescu “std::allocator...” · CppCon · 1:12:27
- William Byrd on "The Most Beautiful Program Ever Written" [PWL NYC] · PapersWeLove · 1:31:05
- "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson · Strange Loop Conference · 40:20
- Why Isn't Functional Programming the Norm? – Richard Feldman · Metosin · 46:09
- Maybe Not - Rich Hickey · ClojureTV · 1:03:28
- Solving Problems the Clojure Way - Rafal Dittwald · Clojure/north · 1:02:26
- Lecture 1A: Overview and Introduction to Lisp · MIT OpenCourseWare · 1:12:55
- Preventing the Collapse of Civilization / Jonathan Blow (Thekla, Inc) · DevGAMM · 1:02:41
- Conway's Game Of Life in APL · Dyalog · 7:47