Parsing JSON Really Quickly: Lessons Learned

Channel:InfoQ · Length:47:52 · Published:2020-01-15 · Views:85861

Watch on YouTube

Summary

Daniel Lemire's QCon talk on how simdjson parses gigabytes of JSON per second. It covers SIMD vectorization, branch prediction and the lessons learned building a fast parser.

Why HN recommends it

HN points to it as the best explanation of simdjson internals and of why fast parsing is rarely disk-bound.

It's worth looking at if you're into this sort of thing. — HN, 2021

Software performance & optimization · Compilers, interpreters & program analysis · Algorithms & data structures

Subtitles (excerpt)

my name is Daniel Amir so I'm from the University of Quebec and I'm going to talk to you about parsing JSON really quickly so I'm one of the authors of what might be the fastest json parser in the world and so i'm going to try to tell you about the strategies we've used and give you some examples of the tricks that we've been building up to make this possible so i'm going to start with a relatively simple and naive question so if i to motivate my talk so if I give you a relatively large file and I ask you to read it in software and then I ask you what is your limit right so are you limited by the performance of your disk or are you limited by the performance of your processor so I'm not going to ask for a shelf and but I would guess that most people would say that you're strictly limited by your disk and I could reframe the same question by talking about network but I would argue that the story is a little bit more computer than people sometimes think so in preparing this talk I just benchmark to disk on my iMac which is just a basic stupid iMac on my desk on campus and so my desk it was rated at 2.2 gigabyte per second it's a cloud put and and of course there are better discs out there and our network adapters that are much for a sort of much faster

On Hacker News

Daniel Lemire has an interesting talk on this: [VIDEO] . They key takeaway is that the disk is rarely the bottleneck. — D4ckard, 2024-01
I find these optimizations fascinating. Anyone familiar with Lemire likely knows about this, but I listened to a podcast episode[1] with him a few days ago and learned about `simdjson`, the tool he authored that parses JSON at 25x the speed of the standard C++ library.[2][3] It's worth looking at if you're into this sort of thing. 1. [LINK] 2. [VIDEO] 3. [LINK] — Rendello, 2021-10
I can relate. I have been one of those who started writing a CSV library and went down the rabbit hole of edge cases and configurability. Looking back (and only thinking on productivity) my time would have been better spent doing something else. I did, however, learn quite a lot along the way: - how text is represented in a computer (and all its caveats), - how to identify and tune slow code, - h… — dehesa, 2020-05
Github: [LINK] It showcases an especially ingenious scheme for escaping json strings, as well as the other tokens necessary to parse json. Daniel Lemire - One of the authors at QCon 2019: [VIDEO] Another Youtube video that explains it: [LINK] — ubedan, 2026-07
The GitHub page links to a video that explains some of the internals [1]. Can someone comment on the result that they show at 14:26? My understanding is that they run a code that does 2000 branches based on a pseudo-random sequence. Over around 10 runs of that code, the CPU supposedly learns to correctly predict those 2000 branches and the performance steadily increases. Do the modern branch pred… — avian, 2020-08

Related videos