CS480/680 Lecture 19: Attention and Transformer Networks

Channel:Pascal Poupart · Length:1:22:38 · Published:2019-07-16 · Views:373946

Watch on YouTube

Summary

A university lecture from CS480/680 by Pascal Poupart on attention mechanisms and transformer networks, worked through on a chalkboard.

Why HN recommends it

HN recommends it to newcomers as a clear explanation of attention, including the query/key/value intuition.

I would also recommend Pascal Poupart's talk [1] on Attention and Transformers. — HN, 2023

Machine learning & deep learning · LLMs & generative AI · University lectures & courses

Subtitles (excerpt)

okay so for the next set of slides I'm gonna talk about attention in more detail so we discuss already attention in the context of machine translation but attention has led to a new class of neural networks that are known as transformer networks okay and this this material that I'm going to present today is actually quite recent so it's not described in any of the textbooks that are associated with this course so for this there is a paper that was published in 2017 called attention is all you need okay so it's a very interesting title and it actually does suggest that perhaps we don't need recurrent neural networks anymore and and then a lot of the building blocks that people would design let's see let's same units gr use and and other things like this this essentially is suggesting that we don't need them all we need is really just attention ok so let's see how this works ok so it turns out that the concept of attention was was first studied in the context of computer vision and here let's say that we have some images we're going to recognize some objects so as humans let's say that we have a large image and then we're looking for a tiny object then what we will do naturally with our eyes is roughly scan the image and

On Hacker News

What made attention mechanism click to me was the video [LINK] by Sebastian Raschka. Another good video on attention that tries to give a bit more intuition on the query/key/value stuff is [VIDEO] In hindsight, what helped me most is understanding the dimensionality at each step of the attention computation (explained well in Raschka's video). Another think I simply did not get from the original … — metanonsense, 2023-06
This might be dumbing this down very very much but it all boils down to having some sort of a special lookup table and you give it a "query" and a bunch of "keys" and the "value" is the most likely next word. As you input something into the network, a network of these tables are consulted and you are given most likely next word. The novelty in this paper is this "query-key-value" relation that ge… — elankart, 2023-05
I would also recommend Pascal Poupart's talk [1] on Attention and Transformers. Doesn't seem to be cited often, but its quite well done and seeing him work out some of the details using chalk and board is very reassuring. [1] [VIDEO] — abhgh, 2023-04
> I have zero AI/ML knowledge This may make it difficult to explain and I already see many incorrect explanations here and even more lazy ones (why post the first Google result? You're just adding noise) > Steve Yegge on Medium thinks that the team behind Transformers deserves a Nobel First, Yegge needs to be able to tell me what Attention and Transformers are. More importantly, he needs to tell … — godelski, 2023-05

Related videos