GradIEEEnt half decent: The hidden power of imprecise lines
Channel:suckerpinch · Length:55:02 · Published:2023-05-01 · Views:392064
Summary
Tom 7 (suckerpinch) delivers a long, comedic digression on gradient descent and floating-point imprecision. He shows how rounding behavior can supply the nonlinearity needed to build a neural network.
Why HN recommends it
Cited as a fun, unconventional example of neural networks built from floating point error.
A fun example of that (making a neural network using floating point error as a source of non-linearity) — HN, 2024
Software performance & optimization · Machine learning & deep learning · Stand-up & sketch comedy
Subtitles (excerpt)
Imagine you're my professor. Maybe you actually were my professor, in which case you may already be sweating before I say anymore. The subject matter is neural networks. You draw an illustration on the board with the nodes, inputs, and its outputs via a transfer function. You inform us of this mathematical fact that the transfer function cannot be linear, or the whole model would reduce to a linear function. I immediately raise my hand. The speed with which I raise it and the not very subtle forward pose suggest that I want to pluck an abstract idea from the whiteboard and pervert it. You know this look, and you are reluctant to call on me, but no other students are raising their hands. You have no choice. Tom. It's more like a statement than a question. It includes the tone of spoken punctuation that, if it could, ends the entire conversation before it begins. I go on and on about some technicality that due to approximate math on the computer, this mathematical fact won't be true. You say, okay, technically that's right, but for all practical purposes it doesn't matter. And I say, well, what about impractical purposes? And you in a moment of weakness vigorously strangle me. And that's how I died. Murdered in cold blood.
On Hacker News
A fun example of that (making a neural network using floating point error as a source of non-linearity): [VIDEO] — jcparkyn, 2024-05
> squash an output into a range This isn't the primary purpose of the activation function, and in fact it's not even necessary. For example see ReLU (probably the most common activation function), leaky ReLU, or for a sillier example: [VIDEO] — jcparkyn, 2024-10
I don’t get what people are trying to say when they say these kinds of things about AI. That human-level writing is as simple as a linear regression? That we could’ve had computer programs capable of human-level writing decades ago? Have they not used these AIs enough to see how powerful they are? Are they seeing the bad outits and thinking that AIs are always doing that poorly? Like seriously, i… — Uehreka, 2023-10
Cool stuff. I looked at [LINK] and I noticed that that the fp8 support is only for the tensor cores and not the CUDA side. Does that mean training with H100 GPU in fp8 mode would use some software ecosystem that's not the existing vast existing CUDA one? Or am I just misunderstanding CUDA cores vs tensor cores? PS, as a joke, they should implement GPU fluint8 and get baked in non-linearity for th… — superkuh, 2023-05
Related videos
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- How Python was Shaped by leaky Internals, Armin Ronacher, Flask framework · Видео с мероприятий {speach! · 49:55
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- CppCon 2015: Andrei Alexandrescu “std::allocator...” · CppCon · 1:12:27
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- Preventing the Collapse of Civilization / Jonathan Blow (Thekla, Inc) · DevGAMM · 1:02:41
- "Performance Matters" by Emery Berger · Strange Loop Conference · 42:15
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- The Thirty Million Line Problem · Molly Rocket · 1:48:55
- CppCon 2017: Matt Kulukundis “Designing a Fast, Efficient, Cache-friendly Hash Table, Step by Step” · CppCon · 59:33
- Harder Drive: Hard drives we didn't want or need · suckerpinch · 36:46
- Lecture 1 | Machine Learning (Stanford) · Stanford · 1:08:40
- Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp · Erlang Solutions · 49:15
- How Crash Bandicoot Hacked The Original Playstation | War Stories | Ars Technica · Ars Technica · 31:37
- MarI/O - Machine Learning for Video Games · SethBling · 5:57