Attention Is All You Need
Channel:Yannic Kilcher · Length:27:06 · Published:2017-11-28 · Views:817445
Summary
Yannic Kilcher walks through the Attention Is All You Need paper, which introduced the Transformer, a sequence model based solely on attention without recurrence or convolutions. He reviews the architecture and the paper's translation results.
Why HN recommends it
HN recommends it as an accessible, clear walkthrough of the transformer paper for people with some AI background.
The Yannic kilcher review is quite good. — HN, 2023
Machine learning & deep learning · LLMs & generative AI
Subtitles (excerpt)
hi there today we're looking at attention is all you need by Google just to declare I don't work for Google just because we've been looking at Google papers lately but it's just an interesting paper and we're gonna see what's the deal with it so basically what the authors are saying is we should kind of get away from basically onions so traditionally what you would do and these authors particular interested in NLP natural language processing so traditionally when you had like a language task the cat eats the mouse and you would like to translate this to say any other language like let's say German or whatever what you would do is you would try to encode this sentence into a representation and then decode it again so somehow somehow this sentence needs to all go into say one vector and then this one vector needs to somehow be transformed into the target language so these are tradition called sack to sack tasks and they have been solved so far using recurrent neural networks you might know the Alice TN networks that are very popular for these tasks what basically happens in an RNN is that you go over the say source sentence here one by one here you take the word the you kind of encode it maybe with a word vector if you know that is so
On Hacker News
YouTube channel that explains the paper in detail: [VIDEO] And subsequent follow-ups (ROME, editing transformer arch): [LINK] I find the channel amazing explaining super complex topics in simple enough terms for people who have some background in AI. — bfung, 2023-01
The Yannic kilcher review is quite good. [VIDEO] I can't ELI5 but I can ELI-junior-dev. Tl;dw: Transformers work by basically being a differentiable lookup/hash table. First your input is tokenized and (N) tokens (this constitutes the attention frame) are encoded both based on token identity and position in the attention frame. Then there is an NxN matrix that is applied to your attention frame "… — throwawaymaths, 2023-05
For accessibility I recommend Yannic Kilcher video review of "Attention Is All You Need" [VIDEO] Yannic has been making about 62 other transformer paper reviews since. You can find the usual suspects. [LINK] — visarga, 2021-12
Everyone has a different way of approaching. I like video walk-throughs of papers. If you are the same, for the theory bit you may find Yannic Kilcher's videos [0,1] helpful. On implementation side, the best project out there is [2] with good documentation. Also spacy has transformer package [3] (I personally haven't tried it), so maybe it will be easier for you to jump in if you have prior exper… — kanche, 2020-10
> I love these word-salad exaltations of LLMs - people don't even know what they're writing half the time but it sure is provocative! Well FWIW I have done an implementation of Llama LLM (3 8-70b to be specific) in non-python so I do sort of know what I'm talking about. I'm not the originator of the hash table analogy. I got it from here: [VIDEO] Hell, the vectors that are generated are called K/… — throwawaymaths, 2025-05
Related videos
- RL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning · Google DeepMind · 1:28:12
- Why Greatness Cannot Be Planned — Kenneth Stanley · Machine Learning Street Talk · 2:46:27
- High-Speed Robot Hand · Hizook · 3:00
- Neural Network Learns The Physics of Fluids and Smoke | Two Minute Papers #118 · Two Minute Papers · 5:26
- RAAIS 2017 - Simon Knowles, Co-Founder and CTO of Graphcore · The Research and Applied AI Summit - RAAIS · 35:44
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- Lecture 1 | Machine Learning (Stanford) · Stanford · 1:08:40
- MarI/O - Machine Learning for Video Games · SethBling · 5:57
- Deep Dive into LLMs like ChatGPT · Andrej Karpathy · 3:31:23
- Computer program that learns to play classic NES games · suckerpinch · 16:17
- Let's build GPT: from scratch, in code, spelled out. · Andrej Karpathy · 1:56:20
- Bay Area Vision Meeting: Unsupervised Feature Learning and Deep Learning · Google TechTalks · 48:10