Ray Summit 2022 - Day 1 Keynote - Fireside Chat
Channel:Anyscale · Length:18:22 · Published:2022-11-02 · Views:6013
Summary
A Ray Summit 2022 keynote fireside chat between OpenAI's Greg Brockman and Anyscale's Robert Nishihara. They discuss how OpenAI's training infrastructure evolved and its use of Ray.
Why HN recommends it
Cited as a source for the fact that OpenAI uses Ray for training, and for Brockman's views on developer speed.
OpenAI fireside chat on the evolution of their infrastructure and usage of Ray for training — HN, 2023
Distributed systems · Machine learning & deep learning · LLMs & generative AI
Subtitles (excerpt)
Greg, thank you so much for being here. It's really really a pleasure to to have you here. It's good to be here. So, you're one of the co-founders and president of Open AI. Um, can you tell us a little bit about why you started Open AI and what Open AI is working on? Yeah, in 2015, we had this dinner uh down in uh in in Palo Alto, uh not not too far from here, I guess, uh with with a bunch of of people like uh some some of my co-founders and uh we had this question of is it too late to start a lab trying to build artificial general intelligence? You know, it felt like AI is clearly going somewhere. Maybe you can even get to the point someday that you can build a machine that's as smart as as a human and it's clearly going to kind of change everything. And, you know, it was a question of do you think that people in this room could could start a company to to try to to make it go well or maybe it's just kind of already in the hands of uh of of the players that are out there. And I think we all kind of looked at each other and said, "It's not obviously impossible. Maybe it could be done." Uh and so, the next day I was full-time trying to put together a team and and make it happen. Nice. And so, that's that's that's our mission.
On Hacker News
[Author] OpenAI uses Ray, the open source version of Anyscale Platform, for training their models (video evidence: [VIDEO] ). Ray ( [LINK] ) is available for anyone to use. You can also use Aviary ( [LINK] ) to serve any of those models yourself. — waleedk, 2023-08
Consistently I see this advice from everyone who I consider has achieved outside success in their field. Greg Brockman (OpenAI Cofounder) : Developer Speed is the most important thing I care about: ( [VIDEO] ). I believe he said more about this topic but I can't find it now. Frank Slootman (Snowflake CEO): In his book Amp It Up, he stresses why pushing people to move fast is extremely important E… — sashank_1509, 2023-06
I'm one of the Ray developers, thanks for the shoutout :) If you're curious about how Ray is used for LLMs, here are some interesting examples of LLM projects using Ray! - Alpa does training and serving with 175B parameter models [LINK] - GPT-J [LINK] - Another HN thread on training LLMs with Ray (on TPUs in this case) [LINK] - OpenAI fireside chat on the evolution of their infrastructure and usa… — robertnishihara, 2023-02
Related videos
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · GOTO Conferences · 42:02
- But what is a neural network? | Deep learning chapter 1 · 3Blue1Brown · 18:40
- "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson · Strange Loop Conference · 40:20
- AlphaGo - The Movie | Full award-winning documentary · Google DeepMind · 1:30:28
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown · 27:14
- USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? · USENIX · 51:22
- Lecture 1 | Machine Learning (Stanford) · Stanford · 1:08:40
- Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp · Erlang Solutions · 49:15
- MarI/O - Machine Learning for Video Games · SethBling · 5:57
- Deep Dive into LLMs like ChatGPT · Andrej Karpathy · 3:31:23
- "Turning the database inside out with Apache Samza" by Martin Kleppmann · Strange Loop Conference · 47:42
- Computer program that learns to play classic NES games · suckerpinch · 16:17
- RL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning · Google DeepMind · 1:28:12
- The Language of the System - Rich Hickey · ClojureTV · 1:02:49
- Google I/O 2012 - Go Concurrency Patterns · Google for Developers · 51:27