AI can't cross this line and we don't know why.

Channel:Welch Labs · Length:24:07 · Published:2024-09-13 · Views:2530622

Watch on YouTube

Summary

A Welch Labs explainer on neural scaling laws and the compute-efficient frontier, asking why model loss seems to hit a line it cannot cross. It draws an analogy to an ideal gas law for AI.

Why HN recommends it

Recommended as a well-made, digestible introduction to scaling laws and why models cluster along a similar efficiency frontier.

see eg for a well made/easily digestable intro — HN, 2024

Machine learning & deep learning · LLMs & generative AI

Subtitles (excerpt)

AI models can't cross this boundary and we don't know why as we train an AI model its error rate generally drops off quickly and then levels off if we train a larger model it will achieve a lower error rate but requires more compute scaling to larger and larger models we end up with a family of Curves like this switching our axis to logarithmic scales a clear Trend emerges where no model can cross this line known as the compute optimal or compute efficient Frontier this trend is one of three three neural scaling laws that have been broadly observed error rate scales in a very similar way with compute model size and data set size and remarkably doesn't depend much on model architecture or other algorithmic details as long as reasonably good choices are made the interesting question from here is have we discovered some fundamental law of nature like an ideal gas law for building intelligent systems or is this transist result of the specific neural network driven approach to AI that we're taking right now now how powerful can these models become if we continue increasing the amount of data model sizing compute can we drive errors to zero or will performance level off why are data model size and compute the fundamental

On Hacker News

I'm not deep enough in the ecosystem to ingest that paper but it's title reminded me of an interesting video I saw recently: "AI can't cross this line and we don't know why" [VIDEO] — mdaniel, 2025-08
Please someone correct me if I’m wrong, but my understanding of ML/LLMs is that this kind of hand crafting has been tried, but it is easier to train/less finicky to let behavior like this emerge from more data, see [1] “Bitter Lesson” and [2] “Scaling Laws”. MAMBA as an architecture claims to have some significant gains performance wise, but to my knowledge there haven't been any really large mod… — kla-s, 2024-10
Yep we do. There is a 1 year old video on YouTube, which describes this limitation [VIDEO] Called efficient compute frontier — general1465, 2025-09
Because they are hitting Compute Efficient Frontier. Models can't be much bigger, there is no more original data on the internet, so all models will eventually cluster to similar CEF as was described in this video 10 months ago [VIDEO] — general1726, 2025-08
My attempt at a summary: authors characterize the data-compute pareto front (aka how bitter is the lesson, exactly?) For a different perspective, error vs compute, see [VIDEO] and comments (I particularly liked the one about string theorists rediscovering a fundamental theorem in GR decades too late-- rediscovering how to integrate happens in every field, it's nothing to be ashamed of :) — gsf_emergency_2, 2025-02

Related videos