The Simple Essence of Automatic Differentiation - Conal Elliott
Channel:Microsoft Research · Length:1:30:45 · Published:2018-07-26 · Views:17062
Summary
Conal Elliott presents his paper on automatic differentiation, deriving a simple, generalized AD algorithm from a natural specification. Varying the representation of derivatives yields forward and reverse mode algorithms that avoid the complexity and statefulness of backpropagation.
Why HN recommends it
Cited as the more accessible companion to the paper, and as the most interesting treatment of AD via compiling to categories.
There's a talk that Conal gives that makes the paper a bit more accessible — HN, 2022
Functional programming · Machine learning & deep learning · Pure mathematics
Subtitles (excerpt)
>> I'm Daan and I'm honored to host today Conal Elliott, former MSR colleague and a good friend of mine. Conal used to work here at Microsoft Research until 2002, and now he's a distinguished scientist at the target. But the way he was here at Microsoft research, I was so busy students and I remember being so inspired by his beautiful work on functional reactive animations, and at that time was really inspirational to me. One thing that Conal is great at is taking something really complicated. In that time like animations you will always have matrixes and frames and timers, and he kind of rephrase it in a beautiful way about the essence of animations as functions of time. I think with this work he applies the same trick again but now in terms of differentiation and machine learning, it's always about all these complicated things, and you almost don't see the essence, and this talk will hopefully shed some light on it. So, thanks a lot, Conal. >> Thanks, Don. It's great to be here. Yes, I was at Microsoft Research in the graphics group from '94 to 2002, I had a good time. So, this is my first time back giving a talk. So, anyway, we're going to talk about automatic differentiation which has
On Hacker News
The most interesting thing I've seen on AD is "The simple essence of automatic differentiation" (2018) [1]. See past discussion [2], and talk [3]. I think the main idea is that by compiling to categories and pairing up a function with its derivative, the pair becomes trivially composable in forward mode, and the whole structure is easily converted to reverse mode afterwards. [1]: [LINK] [2]: [LIN… — infogulch, 2022-04
I'm yet to give reverse mode a second try, but my attempts at forward mode (linked in parent post) were on my own without the paper's help. There's a talk that Conal gives that makes the paper a bit more accessible - [VIDEO] - but only a bit more. I know it'll take effort to wrap my head around the implementation details ... but the paper certainly helped clarify things I was stumbling around abo… — sriku, 2022-09
He also gave nice presentation on AD: [VIDEO] (surprised that you didn't already mention it, considering the venue ;-)) — noblethrasher, 2019-11
Another paper related to AD was posted last month: The Simple Essence of Automatic Differentiation [0]. Video presentation [1]. I found it pretty awesome, just in case you hadn't seen it. [0]: https//news.ycombinator.com/item?id=18306860 [1]: [VIDEO] — infogulch, 2018-11
This paper won a best paper award at ICFP 2018. Conal Elliott recently gave a longer talk about it at Microsoft Research: [VIDEO] — bmc7505, 2018-10
Related videos
- How to achieve nanometer precision in space? · Breaking Taps · 11:06
- TWO Unpickable (?) Locks for Lock Picking Lawyer! · Stuff Made Here · 14:39
- This is why we can't have nice things · Veritasium · 17:30
- The Future of Mathematics? · Microsoft Research · 1:14:48
- Fast Inverse Square Root — A Quake III Algorithm · Nemean · 20:08
- [1299] Unpickable Locks From Stuff Made Here · LockPickingLawyer · 10:35
- Why can't you multiply vectors? · Freya Holmér · 51:15
- Fourier Optics used for Optical Pattern Recognition · Huygens Optics · 18:30
- Refterm Lecture Part 1 - Philosophies of Optimization · Molly Rocket · 18:41
- MINIX 3: a Modular, Self-Healing POSIX-compatible Operating System · FOSDEM · 56:19
- 28c3: The Science of Insecurity · 28c3 · 59:23
- Netflix JavaScript Talks - Async JavaScript with Reactive Extensions · Matt Marenghi · 28:38
- Vectors | Chapter 1, Essence of linear algebra · 3Blue1Brown · 9:51
- Ruby Conf 12 - Y Not- Adventures in Functional Programming by Jim Weirich · Confreaks · 53:21
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- But what is the Fourier Transform? A visual introduction. · 3Blue1Brown · 19:41