Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP
Channel:Sasha Rush · Length:39:42 · Published:2023-07-05 · Views:35431
Summary
Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.
Why HN recommends it
HN commenters recommend it for building intuition about state space models and the tradeoffs between transformer and SSM layers.
I highly recommend Sasha Rush's video on it — HN, 2024
Machine learning & deep learning · LLMs & generative AI
Subtitles (excerpt)
Today I'm going to talk about the topic I've been interested in for the last couple of years. And it boils down to this question, do we need attention? This talk is really a survey of work done by a ton of other awesome researchers. To begin, I want to give a little bit of a preface where we talk about transformers and attention. So in particular, I want to introduce this abstract representation of the goal of this project. We're going to take some input sequence in and we want to produce a final sequence out. To do this, we're going to run two repeated components. One is a feed forward layer and the other is an attention layer. Now let's start by talking about the feed forward layer. This part basically acts on each component of the sequence independently. It can be run fully in parallel and it corresponds to just running a neural network on every component of the sequence. The more interesting part is the attention part. Here we're going to basically compute the next step by taking into account every other previous time step of the sequence. And you can think about this as being a quadratic dependency where each component in the new sequence needs to look over every previous possible position.
On Hacker News
To those curious about the tradeoffs between transformer and state space model layers, I highly recommend Sasha Rush's video on it: [VIDEO] — a_wild_dandan, 2024-03
Hi! Blog author. This was an attempt a couple years ago to understand and write about this paper in a detailed way. Here is a video going through this topic as well: [VIDEO] Section 2 of the blog post is no longer very relevant. A lot of advances (DSS, S4D) simplified that part of the process. Arguably also this all should be updated for Mamba (same authors). — srush, 2024-02
Would recommend Sasha Rush's lecture [VIDEO] , it provides the intuition for state space models via linear RNNs — anomaly72, 2024-01
Related videos
- Drawing Conclusions: Is renting really a waste of money? · The Globe and Mail · 8:33
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- Lecture 1 | Machine Learning (Stanford) · Stanford · 1:08:40
- MarI/O - Machine Learning for Video Games · SethBling · 5:57
- Deep Dive into LLMs like ChatGPT · Andrej Karpathy · 3:31:23
- Computer program that learns to play classic NES games · suckerpinch · 16:17
- RL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning · Google DeepMind · 1:28:12
- Let's build GPT: from scratch, in code, spelled out. · Andrej Karpathy · 1:56:20
- Bay Area Vision Meeting: Unsupervised Feature Learning and Deep Learning · Google TechTalks · 48:10
- A Hackers' Guide to Language Models · Jeremy Howard · 1:31:13
- Andrej Karpathy - AI for Full-Self Driving at Tesla · Matroid · 30:08
- [1hr Talk] Intro to Large Language Models · Andrej Karpathy · 59:48