WE MUST ADD STRUCTURE TO DEEP LEARNING BECAUSE...
Channel:Machine Learning Street Talk · Length:1:49:10 · Published:2024-04-01 · Views:101491
Summary
Machine Learning Street Talk interviews Paul Lessard about Categorical Deep Learning, which uses category theory and monads to make neural network architectures more interpretable and composable. They also discuss neural networks' limits in reasoning and unbounded computation.
Why HN recommends it
Mostly cited in passing, e.g. regarding claims that transformers cannot do recursion and as an example of structured deep learning work.
Functional programming · Machine learning & deep learning
Subtitles (excerpt)
So, Paul, we're, um. Where did you come from? Are you are you nearby? Tim. Uh, for the last two weeks, I've been in London. Oh, yeah? Tell Keith the story. So, um, they raised some seed capital about 18 months ago, and then they set a date in February to do so. Yeah. So what the deal is, I think sometime in 2022, George and then co-founder and his co-founder, Taylor, they were, you know, George quit Tesla because he thought that the way that they were proposing to fix autopilot, in particular, to fix the "San Francisco problem" for autopilot, was just way more data for San Francisco specifically and just, you know, throwing compute at the wall. And so it was like this. So George's thought was like, this can't be the right way to do it. So he was like, I'm not exactly sure how to do this, but I want to take a bet. I want to take a position against scale. It kind of feels like they're trying to memorize infinity. That's correct. So, um, so would you say that your experience there informed some of this skepticism? Yeah. I mean, I think if you look at the AI day presentations that effectively go into deep technical detail about how Tesla learns the world or Andre's talk at NeurIPS two years ago.
On Hacker News
Symbolica.ai | London, Australia | REMOTE, INTERNS, VISA We're trying to apply the insights of category theory, dependent type theory, and functional programming to deep learning. How do we best equip neural nets with strong inductive biases from these fields to help them reason in a structured way? Our upcoming ICML paper gives some flavor [LINK] ; you can also watch [VIDEO] ; but there is a lot… — taliesinb, 2024-06
Do you have any citation on lack of funding for foundational research? From the looks of things, it seems like there is insane amount of money in AI and while maybe majority of it goes to "projects with high probability of success", whatever that means, fringe ideas seem to generate good amount of capital as well. Even (what I would consider to be) academic projects seems to get good funding [0].… — itchyjunk, 2024-05
Interesting, I thought it was proven Transformers cannot do recursion[0] and aren't actually able to be equivalent to a turing machine due to lack of expandable memory, and are rather finite state automata. [0]: [VIDEO] — f_devd, 2024-04
Related videos
- Ruby Conf 12 - Y Not- Adventures in Functional Programming by Jim Weirich · Confreaks · 53:21
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · GOTO Conferences · 42:02
- "Simple Made Easy" - Rich Hickey (2011) · Strange Loop Conference · 1:01:38
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- "Making Impossible States Impossible" by Richard Feldman · elm-conf · 25:06
- William Byrd on "The Most Beautiful Program Ever Written" [PWL NYC] · PapersWeLove · 1:31:05
- The Value of Values with Rich Hickey · InfoQ · 31:43
- Why Isn't Functional Programming the Norm? – Richard Feldman · Metosin · 46:09
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Maybe Not - Rich Hickey · ClojureTV · 1:03:28
- Solving Problems the Clojure Way - Rafal Dittwald · Clojure/north · 1:02:26
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- Lecture 1A: Overview and Introduction to Lisp · MIT OpenCourseWare · 1:12:55
- Conway's Game Of Life in APL · Dyalog · 7:47
- Lecture 1A | MIT 6.001 Structure and Interpretation, 1986 · MIT OpenCourseWare · 1:12:55