Python at Massive Scale - Stephen Simmons, Neil Slinger
Channel:PyData · Length:44:45 · Published:2018-05-28 · Views:10005
Summary
A PyData London 2018 talk on how JPMorgan scaled its Athena Python platform over ten years to about 35 million lines of code and thousands of developers. It covers infrastructure, tooling and culture for analytics and visualization at that scale.
Why HN recommends it
Cited as the definitive public account of Python at massive scale, especially in discussions of monorepos and bank Python.
4500 developers making 20k commits a week. Codebase with 35m LOC. — HN, 2020
Python, Ruby & scripting languages · Software practice, open source & careers
Subtitles (excerpt)
thanks everyone so there are two names here that you'll see unfortunately Neil got six this morning and sent me progress reports all through the morning of how he was getting worse and worse and having enjoyed a birthday party for his three-year-old son he then put himself to bed so the talk today is going to be about our experience jpmorgan of running a Python infrastructure massive scale the final part of the talk was going to be a live coding demo done by Neil but he's not going to be here to do that and so I won't attempt his live coding I will give a kind of impressionistic view over the slides that he was going to present and then I'll quickly switch to questions that you might have so first of all just to introduce JP Morgan so JP Morgan is one of the biggest banks in the world not really a household name in the UK as far as banks go pretty big a couple of nice round numbers there I guess the nice system around this number of all is to revenue is a hundred billion dollars last year so that's quite large and the bank has two hundred fifty two thousand employees globally so that is that is a very large company now on the tech side most people don't think of JP Morgan as a technology company maybe for you guys because you're a lot closer to to these types
On Hacker News
As the comment by timsworkaccount says, this is definitely Athena at JPMorgan. For those interested, I gave a talk on Athena at PyData UK 2018 called "Python at Massive Scale". 4500 developers making 20k commits a week. Codebase with 35m LOC. The video is here: [VIDEO] It covers Athena's origins, what it is used for, application architecture, infrastructure, dev tooling and culture. — stevesimmons, 2020-07
>Python shines beautifully as a (somewhat) cross platform scripting language Python is much more than just a scripting language. I remember attending this talk[1] a few years about JPMorgan's 35 million LOC Python codebase. Python is being used to built seriously large software nowadays and I don't think performance is ever a minor issue. It should always be in the top 3 for any general purpose l… — Barrin92, 2022-05
See [1] for a PyData London talk I did on "Python at Massive Scale", about JPMorgan's Athena trading and risk management system, used for FX, Commodities, Credit, Equities, etc trading. At that time, Athena had 4,500 Python developers doing 20,000 commits a week. Also see one recent HN discussion on "An Oral History of Bank Python" [2]. [1] [VIDEO] [2] [LINK] — stevesimmons, 2022-01
Nice stuff! For a view of Python at JPMorgan, I did a talk at PyData London "Python at Massive Scale" - [VIDEO] — stevesimmons, 2021-11
I worked on Athena at JPMorgan for 8 years, and loved it. Seeing Python at the core of trading, risk and post-trade processing for Commodities, FX, Credit etc was such a great developer experience. By the time I left JPM, there were 4500 devs globally making 20k commits weekly into the Athena codebase. (I did a PyData presentation on this [1] for more details). The one downside was the delayed tr… — stevesimmons, 2021-11
Related videos
- How Highways Almost Destroyed Amsterdam - Plan Jokinen · Not Just Bikes · 3:25
- Ruby Conf 12 - Y Not- Adventures in Functional Programming by Jim Weirich · Confreaks · 53:21
- The spelled-out intro to neural networks and backpropagation: building micrograd · Andrej Karpathy · 2:25:52
- How Python was Shaped by leaky Internals, Armin Ronacher, Flask framework · Видео с мероприятий {speach! · 49:55
- RailsConf 2014 - All the Little Things by Sandi Metz · Confreaks · 38:47
- What the heck is the event loop anyway? | Philip Roberts | JSConf EU · JSConf · 26:52
- 🚀 TDD, Where Did It All Go Wrong (Ian Cooper) · DevTernity Conference · 1:03:55
- How To Design A Good API and Why it Matters · Google TechTalks · 1:00:19
- The Problem with Time & Timezones - Computerphile · Computerphile · 10:12
- Bret Victor The Future of Programming · Joey Reid · 32:55
- RailsConf 2016 - Turbolinks 5: I Can’t Believe It’s Not Native! by Sam Stephenson · Confreaks · 37:37
- "Uncle" Bob Martin - "The Future of Programming" · Paul Stringer's Mobile Tech · 1:18:21
- Brian Goetz - Stewardship: the Sobering Parts · ClojureTV · 1:03:30
- Ruby Midwest 2011 - Keynote: Architecture the Lost Years by Robert Martin · Confreaks · 1:06:38
- JavaScript: The Good Parts · Google TechTalks · 1:03:47
- Tim Ewald - Clojure: Programming with Hand Tools · ClojureTV · 58:39