WE MUST ADD STRUCTURE TO DEEP LEARNING BECAUSE...

Channel:Machine Learning Street Talk · Length:1:49:10 · Published:2024-04-01 · Views:101491

Watch on YouTube

Summary

Machine Learning Street Talk interviews Paul Lessard about Categorical Deep Learning, which uses category theory and monads to make neural network architectures more interpretable and composable. They also discuss neural networks' limits in reasoning and unbounded computation.

Why HN recommends it

Mostly cited in passing, e.g. regarding claims that transformers cannot do recursion and as an example of structured deep learning work.

Functional programming · Machine learning & deep learning

Subtitles (excerpt)

So, Paul, we're, um. Where did you come from? Are you are you nearby? Tim. Uh, for the last two weeks, I've been in London. Oh, yeah? Tell Keith the story. So, um, they raised some seed capital about 18 months ago, and then they set a date in February to do so. Yeah. So what the deal is, I think sometime in 2022, George and then co-founder and his co-founder, Taylor, they were, you know, George quit Tesla because he thought that the way that they were proposing to fix autopilot, in particular, to fix the "San Francisco problem" for autopilot, was just way more data for San Francisco specifically and just, you know, throwing compute at the wall. And so it was like this. So George's thought was like, this can't be the right way to do it. So he was like, I'm not exactly sure how to do this, but I want to take a bet. I want to take a position against scale. It kind of feels like they're trying to memorize infinity. That's correct. So, um, so would you say that your experience there informed some of this skepticism? Yeah. I mean, I think if you look at the AI day presentations that effectively go into deep technical detail about how Tesla learns the world or Andre's talk at NeurIPS two years ago.

On Hacker News

Symbolica.ai | London, Australia | REMOTE, INTERNS, VISA We're trying to apply the insights of category theory, dependent type theory, and functional programming to deep learning. How do we best equip neural nets with strong inductive biases from these fields to help them reason in a structured way? Our upcoming ICML paper gives some flavor [LINK] ; you can also watch [VIDEO] ; but there is a lot… — taliesinb, 2024-06
Do you have any citation on lack of funding for foundational research? From the looks of things, it seems like there is insane amount of money in AI and while maybe majority of it goes to "projects with high probability of success", whatever that means, fringe ideas seem to generate good amount of capital as well. Even (what I would consider to be) academic projects seems to get good funding [0].… — itchyjunk, 2024-05
Interesting, I thought it was proven Transformers cannot do recursion[0] and aren't actually able to be equivalent to a turing machine due to lack of expandable memory, and are rather finite state automata. [0]: [VIDEO] — f_devd, 2024-04

Related videos