HC2023-K2: Hardware for Deep Learning
Channel:hotchipsvideos · Length:1:05:27 · Published:2023-09-13 · Views:14884
Summary
Bill Dally of NVIDIA gives a Hot Chips 2023 keynote on the challenges of building hardware for deep learning. He traces how GPUs have been adapted for machine learning, especially training.
Why HN recommends it
HN points to it for an insider view of why NVIDIA's AI gains came from design rather than process, and for data-movement power costs.
The whole talk is brilliant and worth watching. — HN, 2025
Machine learning & deep learning · Computer architecture & CPUs
Subtitles (excerpt)
Welcome to Hot Chips 2023. Keynote two. Hardware for deep learning. Uh Bill has worked uh across the stack. Um very pro-prolific and and uh successful, accomplished research scientist. He has contributed seminal works on parallel computing architecture, networks, interconnects, routing, uh so the whole stack. Uh he has received pretty much all the awards we can get as computer architects, such as Morris Wilkes and Eckert Mauchly. He is a fellow of ACM and IEEE and is advising the US president. So um I think we've all witnessed the tremendous impact Bill had, especially in the last year with ChatGPT, these technologies that would not have been able or possible without Bill's work. And uh he's also a great example for technology transfer. He basically took the work on streaming processors he did with his students at Stanford and within 15 years turned it into the sixth most valuable company in the world. So, without further ado, uh Bill, please take the podium. Thank you. IS THIS MIC WORKING? SO ASSUMING UH YEAH, THE MIC SEEMS TO BE WORKING. SO, UM there probably isn't a person in the audience who hasn't spent the better part of a weekend playing with chat GPT asking it questions, asking it to write stories
On Hacker News
Neuromorphic computation has been hyped up for ~ 20 year by now. So far it has dramatically underperformed, at least vis-a-vis the hype. The article does not distinguish between training and inference. Google Edge TPUs [LINK] each one is capable of performing 4 trillion operations per second (4 TOPS), using 2 watts of power—that's 2 TOPS per watt. So inference is already cheaper than the 20 watts… — Footpost, 2025-06
I think your biggest mistake is ignoring the amount of software that runs on top of the hardware. There were times in the past where others had a higher number of FLOPS, but it didn’t make a dent in the usage of CUDA. My standard assumption about anything in tech (and beyond) is that you don’t win on first principles alone (“parallel design is well understood”). It’s the details that make you sta… — tverbeure, 2023-10
Says the hotchips keynote - [VIDEO] - data that is moved is about 100x higher power than data that is local You have a good point about intermediate data. This would not apply to all algorithms. — gdiamos, 2024-09
Related videos
- Saving Voyager 1! - Bruce Waggoner at !!Con 2024 · !!Con · 35:25
- The Fast Fourier Transform (FFT): Most Ingenious Algorithm Ever? · Reducible · 28:23
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- 8-bit computer update · Ben Eater · 6:53
- Light Years Ahead | The 1969 Apollo Guidance Computer · TNMoC · 1:21:22
- “Hello, world” from scratch on a 6502 — Part 1 · Ben Eater · 27:24
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- Finding the Missing Memristor - R. Stanley Williams · The Qualcomm Institute · 47:52
- The Ultimate Game Boy Talk (33c3) · media.ccc.de · 1:01:06
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- The Thirty Million Line Problem · Molly Rocket · 1:48:55
- Lecture 1 | Machine Learning (Stanford) · Stanford · 1:08:40
- MarI/O - Machine Learning for Video Games · SethBling · 5:57
- Deep Dive into LLMs like ChatGPT · Andrej Karpathy · 3:31:23