Trends in Deep Learning Hardware: Bill Dally (NVIDIA)
Channel:Paul G. Allen School · Length:1:10:57 · Published:2023-11-12 · Views:32613
Summary
Bill Dally of NVIDIA gives a University of Washington lecture on trends in deep learning hardware. It covers numeric formats, specialized instructions, energy per operation and co-design of hardware and software.
Why HN recommends it
HN commenters cite it for NVIDIA's perspective on AI hardware efficiency, including logarithmic number formats and energy use.
Quite an interesting video IMO. — HN, 2023
Machine learning & deep learning · Computer architecture & CPUs · University lectures & courses
Subtitles (excerpt)
LUIS CEZE: All right, welcome everyone. It's really a pleasure here to host our distinguished lecture series with Bill Dally today. So Bill Dally is a real computer architect. He's built chips in academia and in industry. He has taught at Caltech, MIT, and then Stanford before going to NVIDIA as chief scientist and head of research there. He has done pioneering work in a bunch of techniques in parallel computing that you probably use today. And it's nice to see that with AI today, like this is what's enabled parallel-- what enables AI is parallel computing. So he's a member of National Academy-- I'm going to have to read here because this is so long so-- National Academy of Engineering, fellow of AAAS, fellow of the IEEE and the ACM, and also won all of the major computer architecture awards, including Eckert-Mauchly Award, the highest honor we have in computer architecture, the Maurice Wilkes Award, and the Seymour Cray Award. I'll keep it short so we get to hear from him, which is what you came here for. But one thing I'm not going to forget to mention is that besides being a major computer architect, super accomplished, he's also a pilot, and he flew here his own airplane. And he did not land in the water at this time, right?
On Hacker News
In a recent talk[1], Bill Dally of Nvidia suggests using actual logarithms, as they're more consistent in relative error than floats, which is especially relevant at low precision as increasingly used in AI. Multiplications then become simple additions, but then addition becomes more complicated, as in general it needs lookup tables. This is the same mechanism as a slide rule, and also reminds me… — raphlinus, 2024-01
It looks like both Pat Gelsinger and Lisa Su have no clue about software and can't even delegate this. They expect "the community" to do the software for their extremely complicated hardware. If you see the presentations by Nvidia's Bill Daly [1] it shows they've been evolving hardware and software together. The CUDA programming model was a massive bet a long time ago. And they deservedly won ML/… — alecco, 2023-12
Depends what you consider as part of chip production. If you consider new instructions and the related software as new chip production, then the performance improvements have continued quite dramatically. Single chip inference has gone 1000x over the last 10 years according to Bill Dally from Nvidia [1] at 10 minutes (probably not original source). Main gains according to the video have been diff… — huijzer, 2023-12
There's a talk by a Nvidia scientist I just saw where he talks about compute efficiency in terms of flops/joule and explicitly mentions vectorized instructions as one of the energy savers. I'll see if I can find it. Edit: found it (I think) I can't remember where in the presentation but he does mention energy efficiency as a proxy for performance. [VIDEO] Edit 2: about 26:40 he starts talking abo… — andy99, 2024-02
> Let's Fermi estimate that. I couldn't agree more. Fermi estimates are underused. > Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data. And for comparison, GPT-4 is estimated by Bill Dally (NVIDIA) at around 10^12 or 10^13 tokens [1]. Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-… — huijzer, 2023-11
Related videos
- CRDTs and the Quest for Distributed Consistency · InfoQ · 43:39
- Data Oriented GUI in Rust by Raph Levien - Bay Area Rust Meetup · Rust Programming Language · 1:05:59
- I messed up. You're using too much detergent. · Technology Connections · 48:24
- The Quantum Conspiracy: What Popularizers of QM Don't Want You to Know · Google TechTalks · 1:03:43
- Why Static Typing Came Back • Richard Feldman • GOTO 2022 · GOTO Conferences · 52:00
- America's Iron Giants - The World's Most Powerful Metalworkers · Machine Thinking · 12:48
- BlackHat USA 2011: SSL And The Future Of Authenticity · Christiaan008 · 48:01
- RSA ANIMATE: Smile or Die · Royal Society of Arts · 10:23
- How we fit an NES game into 40 Kilobytes · Morphcat Games · 12:04
- How to Speak · MIT OpenCourseWare · 1:03:42
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- 1. Introduction to Human Behavioral Biology · Stanford · 57:14
- LEADERSHIP LAB: The Craft of Writing Effectively · UChicago Social Sciences · 1:21:51
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- 8-bit computer update · Ben Eater · 6:53
- MIT's Pathway to Fusion Energy (IAP 2017) - Zach Hartwig · MIT Plasma Science and Fusion Center · 1:11:39