The Zipf Mystery
Channel:Vsauce · Length:21:05 · Published:2015-09-15 · Views:28716933
Summary
Vsauce's explainer on Zipf's law, the pattern in which the frequency of a word is inversely related to its rank. It shows the pattern across many domains and discusses why it remains unexplained.
Why HN recommends it
Linked as an entertaining introduction to Zipf's law and power-law distributions.
It's actually a very deep question, called the Zipf mystery — HN, 2019
Recreational & visual math · Language & linguistics
Subtitles (excerpt)
Hey, Vsauce. Michael here. About 6 percent of everything you say and read and write is the "the" - is the most used word in the English language. About one out of every 16 words we encounter on a daily basis is "the." The top 20 most common English words in order are "the," "of," "and," "to," "a," "in," "is," "I," "that," "it," "for," "you," "was," "with," "on," "as," "have," "but," "be," "they." That's a fun fact. A piece of trivia but it's also more. You see, whether the most commonly used words are ranked across an entire language, or in just one book or article, almost every time a bizarre pattern emerges. The second most used word will appear about half as often as the most used. The third one third as often. The fourth one fourth as often. The fifth one fifth as often. The sixth one sixth as often, and so on all the way down. Seriously. For some reason, the amount of times a word is used is just proportional to one over its rank. Word frequency and ranking on a log log graph follow a nice straight line. A power-law. This phenomenon is called Zipf's Law and it doesn't only apply to English. It also applies to other languages, like, well, all of them. Even ancient languages we haven't been able to translate yet.
On Hacker News
There's an entertaining "V-Sauce" video about Zipf's law for anyone wanting to learn a bit more about it and to see other examples of where it shows up: [VIDEO] — its_ethan, 2024-07
> You need to know about 20,000 words in any language to sound fluent Are you sure about this? AFAIK the use of words in a language isn't distributed evenly, but according to a Pareto distribution. Thus, you can speak _okay_ knowing only ~500 words, I believe. It won't sound perfect but you'll speak and understand pretty decently. I once wrote a python script which read words from a .txt file ful… — WhateverHappns, 2021-04
Your shock is warranted. It's actually a very deep question, called the Zipf mystery: [VIDEO] No one really knows why it's true. It's also true across physical constants, not just artificial quantities made up by humans. The same type of distribution pops up in language, in nature, in numbers, popularity of opening chess moves, everywhere. — sillysaurusx, 2019-11
I won't say it's easy , but it's easier than you might think. Word use follows a power law distribution in all languages ( Zipf's Law, [VIDEO] is an excellent vsauce cover of the subject ), but the short of it is that within a relatively short amount of time spent studying you'll be able to pick out common words and build from there, even if you started from zero. — Baeocystin, 2019-02
The majority of seeding/downloading will be done with the (minority) most popular files, while only a minority of seeding/downloading is done for the majority of files. For example, go to TPB and look at the distribution of seeders/leechers. [VIDEO] — cowtools, 2022-10
Related videos
- How not to be ignorant about the world | Hans and Ola Rosling · TED · 19:09
- China's Bicycle sharing WENT INSANE! · serpentza · 10:02
- Testing if 60 minutes of silence drives you crazy · Veritasium · 10:32
- Is Most Published Research Wrong? · Veritasium · 12:22
- Training AI Without Writing A Reward Function, with Reward Modelling · Robert Miles AI Safety · 17:51
- The Battle of Midway 1942: Told from the Japanese Perspective (1/3) · Montemayor · 41:46
- Why Meat is the Best Worst Thing in the World 🍔 · Kurzgesagt – In a Nutshell · 7:44
- Ben Marking's Fake BBC Nuclear Attack Broadcast (Full Resolution, Uncut, Unedited) · Tooth · 55:42
- Antonin Scalia - On American Exceptionalism · AmericanRhetoric.com · 7:21
- Self: The Movie; · Jecel Assumpção Jr · 21:01
- 30 Weird Chess Algorithms: Elo World · suckerpinch · 42:35
- 4.5 Billion Years in 1 Hour · Kurzgesagt – In a Nutshell · 1:03:31
- Vectors | Chapter 1, Essence of linear algebra · 3Blue1Brown · 9:51
- But what is the Fourier Transform? A visual introduction. · 3Blue1Brown · 19:41
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- George Lakoff: Moral Politics · University of California Television (UCTV) · 58:57