LSTM is dead. Long Live Transformers!

Channel:Seattle Applied Deep Learning · Length:28:48 · Published:2019-12-03 · Views:540928

Watch on YouTube

Summary

Leo Dirac explains how transformer models have replaced LSTMs in NLP. He gives background from bag of words through RNNs and LSTMs, then a technical deep dive into multi-headed self-attention and positional encoding, with sample code.

Why HN recommends it

Recommended as a clear explanation of why RNNs fall short and how transformers solve it.

has gotten tons of compliments and keeps people watching year after year — HN, 2023

Machine learning & deep learning · LLMs & generative AI · Computer vision, speech & NLP

Subtitles (excerpt)

all right cool well thanks everybody um so I'm gonna give the second talk tonight which I'm not crazy about and and I don't want this pattern to to repeat but you know Andrew and I wanted to kick this series off and and felt like me talking twice or better than then not but we're gonna we're gonna get more diversity of folks if any of you want to give a talk yourselves you know somebody who you think might that'd be awesome but a topic that I feel is important for practitioners to understand is a real sea change in natural language processing that's you know all of like 12 months old but is one these things I think is incredibly significant in in the field and that is the advance of the Transformers so the outline for this talk is to start out with some background on natural language processing and sequence modeling and then talk about the LS TM why it's awesome and amazing but still not good enough and then go into Transformus and talk about how they work and why they're amazing so for background on natural language processing NLP I'm gonna be talking just about a subset of NLP which is the supervised learning a part of it so not structured prediction sequence prediction but where you're taking the document as some input and trying to predict

On Hacker News

I don't think you really want a 5yo's version, because that's "computers can be smart". But for an SDE1-level explanation I made this video which has gotten tons of compliments and keeps people watching year after year: [VIDEO] It's in the context of NLP, which is where transformers started of course. — leopd, 2023-05
The attention mechanism started as a simple trick to not use recurrent neural networks. Read the intro in the original paper "Attention is all you need paper" ( [LINK] ) This video explains the drawbacks to RNNs and how transformers solve that: [VIDEO] Andrej Karpathy explains attention here: [LINK] He explains how attention is seen as a communication network: [LINK] — qigtyofhp, 2023-02
Yannic Kilcher is great but this video worked better for me: "LSTM is dead. Long live transformers!" (Leo Dirac): [VIDEO] — peterfirefly, 2023-01
I've just recently been on a journey to understand transformers in and out, here are the resources I've found managed to drill it into my head: 1. Chapters 7, 9, 10 [LINK] This was really useful to really build up to the concepts of attention (although the actual attention section is still brief). 2. [LINK] This was great for visualization and understanding that attention wasn't exclusively for t… — FL33TW00D, 2020-10
I'm not sure I understand it well enough to say but watching a video on it [0] I think there were a few key points: * "Attention is all you need" introduced positional encoding which allows you to keep context of the word, allowing for more complex translation (and thus generative/chatgpt like tasks?) because words now have context relative to each other. Contrast this with "bag of words" models … — abetusk, 2023-04

Related videos