The Zipf Mystery

Channel:Vsauce · Length:21:05 · Published:2015-09-15 · Views:28716933

Watch on YouTube

Summary

Vsauce's explainer on Zipf's law, the pattern in which the frequency of a word is inversely related to its rank. It shows the pattern across many domains and discusses why it remains unexplained.

Why HN recommends it

Linked as an entertaining introduction to Zipf's law and power-law distributions.

It's actually a very deep question, called the Zipf mystery — HN, 2019

Recreational & visual math · Language & linguistics

Subtitles (excerpt)

Hey, Vsauce. Michael here. About 6 percent of everything you say and read and write is the "the" - is the most used word in the English language. About one out of every 16 words we encounter on a daily basis is "the." The top 20 most common English words in order are "the," "of," "and," "to," "a," "in," "is," "I," "that," "it," "for," "you," "was," "with," "on," "as," "have," "but," "be," "they." That's a fun fact. A piece of trivia but it's also more. You see, whether the most commonly used words are ranked across an entire language, or in just one book or article, almost every time a bizarre pattern emerges. The second most used word will appear about half as often as the most used. The third one third as often. The fourth one fourth as often. The fifth one fifth as often. The sixth one sixth as often, and so on all the way down. Seriously. For some reason, the amount of times a word is used is just proportional to one over its rank. Word frequency and ranking on a log log graph follow a nice straight line. A power-law. This phenomenon is called Zipf's Law and it doesn't only apply to English. It also applies to other languages, like, well, all of them. Even ancient languages we haven't been able to translate yet.

On Hacker News

There's an entertaining "V-Sauce" video about Zipf's law for anyone wanting to learn a bit more about it and to see other examples of where it shows up: [VIDEO] — its_ethan, 2024-07
> You need to know about 20,000 words in any language to sound fluent Are you sure about this? AFAIK the use of words in a language isn't distributed evenly, but according to a Pareto distribution. Thus, you can speak _okay_ knowing only ~500 words, I believe. It won't sound perfect but you'll speak and understand pretty decently. I once wrote a python script which read words from a .txt file ful… — WhateverHappns, 2021-04
Your shock is warranted. It's actually a very deep question, called the Zipf mystery: [VIDEO] No one really knows why it's true. It's also true across physical constants, not just artificial quantities made up by humans. The same type of distribution pops up in language, in nature, in numbers, popularity of opening chess moves, everywhere. — sillysaurusx, 2019-11
I won't say it's easy , but it's easier than you might think. Word use follows a power law distribution in all languages ( Zipf's Law, [VIDEO] is an excellent vsauce cover of the subject ), but the short of it is that within a relatively short amount of time spent studying you'll be able to pick out common words and build from there, even if you started from zero. — Baeocystin, 2019-02
The majority of seeding/downloading will be done with the (minority) most popular files, while only a minority of seeding/downloading is done for the majority of files. For example, go to TPB and look at the distribution of seeders/leechers. [VIDEO] — cowtools, 2022-10

Related videos