GPT3: An Even Bigger Language Model - Computerphile

Channel:Computerphile · Length:25:56 · Published:2020-07-01 · Views:446444

Watch on YouTube

Summary

Rob Miles discusses GPT-3 on Computerphile, covering how scaling parameters kept improving results over GPT-2 and how the model nearly learned arithmetic without anyone knowing exactly how.

Why HN recommends it

HN commenters cite it as a concise explanation of GPT-3's scaling jump and surprising abilities at its release.

Computerphile summed it up pretty well on GPT-3's release — HN, 2023

Machine learning & deep learning · LLMs & generative AI

Subtitles (excerpt)

rob welcome back to computer file in these strange times that we find ourselves recording in you've got the green screen up there we're having a few laggy problems with the communications what are you going to talk about today then uh yeah i thought uh today it would make sense to talk about gbt3 because before we had those videos about language models and transformers and gpt2 people seemed to like i think those came out quite well and now there's a new one so is it so they get better what's the what's the deal there then it is both bigger and better that's um that's the headline so like the thing about the thing about gpg2 is just that it was much bigger than anything that came before right it was more parameters and uh just a larger language model and that was kind of the point of that paper right the point it was trying to make is like people in natural language processing have spent a tremendously long time working on the all of these clever things that you can do uh getting in the nitty-gritty of the technical technical stuff for like detailed fine-tuning for these different benchmarks that they have and so on um and gpg2 was open ai just saying well what if we just made a really

On Hacker News

maybe ask your favorite language model for advice. They say it's the answer to all questions, sort of like 42. My point is here: [VIDEO] - they explain that gpt3 (almost) learned how to add and multiply, but they don't know how. To me that's the thing. (the insight comes at 12:29, 15:23) in a sense that is kind of an admission that we don't know the heck what is going on. — MichaelMoser123, 2023-02
> No one in 2018 suspected that large language models would smoothly scale up by simply increasing the number of parameters. [LINK] That was kind of the entire point of GPT-2. Computerphile summed it up pretty well on GPT-3's release: [VIDEO] Here's some quotes from that video: "The thing about gpt2 is just that it was much bigger than anything that came before. It was more parameters and was kin… — CSMastermind, 2023-01
The industry collectively sank untold billions into the blind belief that neural algorithms will somehow turn into "AI." The huge qualitative differences between GPT-2 and GPT-3 seem to suggest that they will, if you just keep adding orders of magnitude more connections and more data. [VIDEO] — stcredzero, 2021-08
yesterday I watched a youtube video about GPT3 ( [VIDEO] ), and it showed two poems. One was human made, the other was from AI trained on that human's poems. Both poems were pretty good. But one of them had a metaphor about the moon reflecting in ocean waves, being distorted and taking on monstrous forms. I figured this had to be the human one, it was a novel description (because metaphor) of a v… — tripzilch, 2020-07
Your first point is actually covered in the computerphile video on GPT-3. They discuss a paper that studies the ability of GPT-3 to do arithmetic as it's part of the learned text database Check a look [VIDEO] — ChrisRR, 2020-07

Related videos