CS480/680 Lecture 19: Attention and Transformer Networks
Channel:Pascal Poupart · Length:1:22:38 · Published:2019-07-16 · Views:373946
Summary
A university lecture from CS480/680 by Pascal Poupart on attention mechanisms and transformer networks, worked through on a chalkboard.
Why HN recommends it
HN recommends it to newcomers as a clear explanation of attention, including the query/key/value intuition.
I would also recommend Pascal Poupart's talk [1] on Attention and Transformers. — HN, 2023
Machine learning & deep learning · LLMs & generative AI · University lectures & courses
Subtitles (excerpt)
okay so for the next set of slides I'm gonna talk about attention in more detail so we discuss already attention in the context of machine translation but attention has led to a new class of neural networks that are known as transformer networks okay and this this material that I'm going to present today is actually quite recent so it's not described in any of the textbooks that are associated with this course so for this there is a paper that was published in 2017 called attention is all you need okay so it's a very interesting title and it actually does suggest that perhaps we don't need recurrent neural networks anymore and and then a lot of the building blocks that people would design let's see let's same units gr use and and other things like this this essentially is suggesting that we don't need them all we need is really just attention ok so let's see how this works ok so it turns out that the concept of attention was was first studied in the context of computer vision and here let's say that we have some images we're going to recognize some objects so as humans let's say that we have a large image and then we're looking for a tiny object then what we will do naturally with our eyes is roughly scan the image and
On Hacker News
What made attention mechanism click to me was the video [LINK] by Sebastian Raschka. Another good video on attention that tries to give a bit more intuition on the query/key/value stuff is [VIDEO] In hindsight, what helped me most is understanding the dimensionality at each step of the attention computation (explained well in Raschka's video). Another think I simply did not get from the original … — metanonsense, 2023-06
This might be dumbing this down very very much but it all boils down to having some sort of a special lookup table and you give it a "query" and a bunch of "keys" and the "value" is the most likely next word. As you input something into the network, a network of these tables are consulted and you are given most likely next word. The novelty in this paper is this "query-key-value" relation that ge… — elankart, 2023-05
I would also recommend Pascal Poupart's talk [1] on Attention and Transformers. Doesn't seem to be cited often, but its quite well done and seeing him work out some of the details using chalk and board is very reassuring. [1] [VIDEO] — abhgh, 2023-04
> I have zero AI/ML knowledge This may make it difficult to explain and I already see many incorrect explanations here and even more lazy ones (why post the first Google result? You're just adding noise) > Steve Yegge on Medium thinks that the team behind Transformers deserves a Nobel First, Yegge needs to be able to tell me what Attention and Transformers are. More importantly, he needs to tell … — godelski, 2023-05
Related videos
- Andrew Ng: Artificial Intelligence is the New Electricity · Stanford Graduate School of Business · 1:27:44
- How to Speak · MIT OpenCourseWare · 1:03:42
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- 1. Introduction to Human Behavioral Biology · Stanford · 57:14
- LEADERSHIP LAB: The Craft of Writing Effectively · UChicago Social Sciences · 1:21:51
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- MIT's Pathway to Fusion Energy (IAP 2017) - Zach Hartwig · MIT Plasma Science and Fusion Center · 1:11:39
- Stanford's Sapolsky On Depression in U.S. (Full Lecture) · Stanford · 52:28
- Justice: What's The Right Thing To Do? Episode 01 "THE MORAL SIDE OF MURDER" · Harvard University · 54:56
- Lec 1 | MIT 18.06 Linear Algebra, Spring 2005 · MIT OpenCourseWare · 39:49
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Randy Pausch's Last Lecture: Achieving Your Childhood Dreams · Carnegie Mellon University · 1:16:27
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- Lecture 1A: Overview and Introduction to Lisp · MIT OpenCourseWare · 1:12:55
- Lecture 1A | MIT 6.001 Structure and Interpretation, 1986 · MIT OpenCourseWare · 1:12:55
- The Most IMPORTANT Video You'll Ever See (part 1 of 8) · wonderingmind42 · 9:17