Python at Massive Scale - Stephen Simmons, Neil Slinger

Channel:PyData · Length:44:45 · Published:2018-05-28 · Views:10005

Watch on YouTube

Summary

A PyData London 2018 talk on how JPMorgan scaled its Athena Python platform over ten years to about 35 million lines of code and thousands of developers. It covers infrastructure, tooling and culture for analytics and visualization at that scale.

Why HN recommends it

Cited as the definitive public account of Python at massive scale, especially in discussions of monorepos and bank Python.

4500 developers making 20k commits a week. Codebase with 35m LOC. — HN, 2020

Python, Ruby & scripting languages · Software practice, open source & careers

Subtitles (excerpt)

thanks everyone so there are two names here that you'll see unfortunately Neil got six this morning and sent me progress reports all through the morning of how he was getting worse and worse and having enjoyed a birthday party for his three-year-old son he then put himself to bed so the talk today is going to be about our experience jpmorgan of running a Python infrastructure massive scale the final part of the talk was going to be a live coding demo done by Neil but he's not going to be here to do that and so I won't attempt his live coding I will give a kind of impressionistic view over the slides that he was going to present and then I'll quickly switch to questions that you might have so first of all just to introduce JP Morgan so JP Morgan is one of the biggest banks in the world not really a household name in the UK as far as banks go pretty big a couple of nice round numbers there I guess the nice system around this number of all is to revenue is a hundred billion dollars last year so that's quite large and the bank has two hundred fifty two thousand employees globally so that is that is a very large company now on the tech side most people don't think of JP Morgan as a technology company maybe for you guys because you're a lot closer to to these types

On Hacker News

As the comment by timsworkaccount says, this is definitely Athena at JPMorgan. For those interested, I gave a talk on Athena at PyData UK 2018 called "Python at Massive Scale". 4500 developers making 20k commits a week. Codebase with 35m LOC. The video is here: [VIDEO] It covers Athena's origins, what it is used for, application architecture, infrastructure, dev tooling and culture. — stevesimmons, 2020-07
>Python shines beautifully as a (somewhat) cross platform scripting language Python is much more than just a scripting language. I remember attending this talk[1] a few years about JPMorgan's 35 million LOC Python codebase. Python is being used to built seriously large software nowadays and I don't think performance is ever a minor issue. It should always be in the top 3 for any general purpose l… — Barrin92, 2022-05
See [1] for a PyData London talk I did on "Python at Massive Scale", about JPMorgan's Athena trading and risk management system, used for FX, Commodities, Credit, Equities, etc trading. At that time, Athena had 4,500 Python developers doing 20,000 commits a week. Also see one recent HN discussion on "An Oral History of Bank Python" [2]. [1] [VIDEO] [2] [LINK] — stevesimmons, 2022-01
Nice stuff! For a view of Python at JPMorgan, I did a talk at PyData London "Python at Massive Scale" - [VIDEO] — stevesimmons, 2021-11
I worked on Athena at JPMorgan for 8 years, and loved it. Seeing Python at the core of trading, risk and post-trade processing for Commodities, FX, Credit etc was such a great developer experience. By the time I left JPM, there were 4500 devs globally making 20k commits weekly into the Athena codebase. (I did a PyData presentation on this [1] for more details). The one downside was the delayed tr… — stevesimmons, 2021-11

Related videos