Parsing JSON Really Quickly: Lessons Learned
Channel:InfoQ · Length:47:52 · Published:2020-01-15 · Views:85861
Summary
Daniel Lemire's QCon talk on how simdjson parses gigabytes of JSON per second. It covers SIMD vectorization, branch prediction and the lessons learned building a fast parser.
Why HN recommends it
HN points to it as the best explanation of simdjson internals and of why fast parsing is rarely disk-bound.
It's worth looking at if you're into this sort of thing. — HN, 2021
Software performance & optimization · Compilers, interpreters & program analysis · Algorithms & data structures
Subtitles (excerpt)
my name is Daniel Amir so I'm from the University of Quebec and I'm going to talk to you about parsing JSON really quickly so I'm one of the authors of what might be the fastest json parser in the world and so i'm going to try to tell you about the strategies we've used and give you some examples of the tricks that we've been building up to make this possible so i'm going to start with a relatively simple and naive question so if i to motivate my talk so if I give you a relatively large file and I ask you to read it in software and then I ask you what is your limit right so are you limited by the performance of your disk or are you limited by the performance of your processor so I'm not going to ask for a shelf and but I would guess that most people would say that you're strictly limited by your disk and I could reframe the same question by talking about network but I would argue that the story is a little bit more computer than people sometimes think so in preparing this talk I just benchmark to disk on my iMac which is just a basic stupid iMac on my desk on campus and so my desk it was rated at 2.2 gigabyte per second it's a cloud put and and of course there are better discs out there and our network adapters that are much for a sort of much faster
On Hacker News
Daniel Lemire has an interesting talk on this: [VIDEO] . They key takeaway is that the disk is rarely the bottleneck. — D4ckard, 2024-01
I find these optimizations fascinating. Anyone familiar with Lemire likely knows about this, but I listened to a podcast episode[1] with him a few days ago and learned about `simdjson`, the tool he authored that parses JSON at 25x the speed of the standard C++ library.[2][3] It's worth looking at if you're into this sort of thing. 1. [LINK] 2. [VIDEO] 3. [LINK] — Rendello, 2021-10
I can relate. I have been one of those who started writing a CSV library and went down the rabbit hole of edge cases and configurability. Looking back (and only thinking on productivity) my time would have been better spent doing something else. I did, however, learn quite a lot along the way: - how text is represented in a computer (and all its caveats), - how to identify and tune slow code, - h… — dehesa, 2020-05
Github: [LINK] It showcases an especially ingenious scheme for escaping json strings, as well as the other tokens necessary to parse json. Daniel Lemire - One of the authors at QCon 2019: [VIDEO] Another Youtube video that explains it: [LINK] — ubedan, 2026-07
The GitHub page links to a video that explains some of the internals [1]. Can someone comment on the result that they show at 14:26? My understanding is that they run a code that does 2000 branches based on a pseudo-random sequence. Over around 10 runs of that code, the CPU supposedly learns to correctly predict those 2000 branches and the performance steadily increases. Do the modern branch pred… — avian, 2020-08
Related videos
- Andrew Kelley: A Practical Guide to Applying Data Oriented Design (DoD) · ChimiChanga · 46:40
- Old-fashioned rice cookers are extremely clever · Technology Connections · 11:11
- What's a Memory Allocator Anyway? - Benjamin Feng · Zig SHOWTIME · 48:30
- Joe Armstrong & Alan Kay - Joe Armstrong interviews Alan Kay · Erlang Solutions · 1:16:55
- Keynote - Preventing, Finding, and Fixing Bugs On a Time Budget | Raymond Hettinger @ PyBay2018 · SF Python · 40:00
- 30 Weird Chess Algorithms: Elo World · suckerpinch · 42:35
- Casey Muratori – The Big OOPs: Anatomy of a Thirty-five-year Mistake – BSC 2025 · Better Software Conference · 2:27:33
- How Suburban Development Makes American Cities Poorer [ST02] · Not Just Bikes · 8:13
- The Car-Replacement Bicycle (the bakfiets) · Not Just Bikes · 10:45
- DEFCON 17: That Awesome Time I Was Sued For Two Billion Dollars · Christiaan008 · 31:27
- Surviving in the Siberian Wilderness for 70 Years (Full Length) · VICE · 35:45
- Why Canadians Can't Bike in the Winter (but Finnish people can) · Not Just Bikes · 15:13
- "The Economics of Programming Languages" by Evan Czaplicki (Strange Loop 2023) · Strange Loop Conference · 43:57
- CppCon 2016: Nicholas Ormrod “The strange details of std::string at Facebook" · CppCon · 31:18
- How Python was Shaped by leaky Internals, Armin Ronacher, Flask framework · Видео с мероприятий {speach! · 49:55
- Rich Harris - Rethinking reactivity · You Gotta Love Frontend · 36:45