What is in a Lucene index? Adrien Grand, Software Engineer, Elasticsearch
Channel:LuceneSolrRevolution · Length:1:04:30 · Published:2013-12-15 · Views:80696
Summary
Elasticsearch engineer and Lucene committer Adrien Grand explains the design decisions inside a Lucene index and what problems it can and cannot solve. The talk goes into how Lucene stores data for fast search and analytics.
Why HN recommends it
HN recommends it as an instructive, easy-to-follow look at Lucene internals from a core committer.
I can super highly recommend talks by Adrien Grand on Lucene and Elasticsearch. — HN, 2020
Databases & storage · Algorithms & data structures
Subtitles (excerpt)
[Music] we are going to talk about uh Lucin internals it's something that I found very interesting so I'm going to try to make it uh interesting for you too about me so my name is Adrian gr I come from France uh I'm a Lucan solar Comm and I I work at elastic search and one one way you can see that I'm interested in the losing internals that have changed a lot of the index file format over the last month so I started with start field but I ended up uh working with the St feed format the ter vector format and dog values as well and I'll so I'll try to give you insight into the the way it works uh with Lucin so one question maybe you had before coming to this talk is why should I learn about uh loen internals so you may think that if you need to to learn about the internals of something it means that that the API is not good enough but actually there are very good reason to learn about Lo internals the first good reason is to to know the cost of the apis so it's very important to know the cost of each API if you want to to best uh use them and to build blazing fast uh s applications uh there are a few things that you should never do with Lucin especially you should never commit all the time so doing doing a Lucin commit every
On Hacker News
I can super highly recommend talks by Adrien Grand on Lucene and Elasticsearch. They are super instructive yet easy to follow. [VIDEO] [LINK] — drej, 2020-01
This is excellent. I was recently reviewing Lucene concepts and found this video really good: [VIDEO] Also this site has a series of Lucene articles that are pretty nice. The one on Term Vectors in particular: [LINK] Based on some quick research it seems like Lucene is already using a sorted skip data structure for the posting list, so I wonder why they had to do a custom implementation? Perhaps … — stereosteve, 2020-06
This is extremely high level. A nice view of how lucene internals work by core committer Adrien Grand: [LINK] [VIDEO] — ddorian43, 2019-09
Related videos
- TypeScript types can run DOOM · Michigan TypeScript · 6:59
- C++Now 2018: Matt Godbolt “What Else Has My Compiler Done For Me Lately?” · CppNow · 1:30:03
- CppCon 2017: Matt Godbolt “What Has My Compiler Done for Me Lately? Unbolting the Compiler's Lid” · CppCon · 1:15:45
- "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson · Strange Loop Conference · 40:20
- The economics of open source by C J Silverio | JSConf EU 2019 · JSConf · 37:43
- The Value of Values with Rich Hickey · InfoQ · 31:43
- CppCon 2017: Matt Kulukundis “Designing a Fast, Efficient, Cache-friendly Hash Table, Step by Step” · CppCon · 59:33
- AWS re:Invent 2018: Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB (DAT401) · Amazon Web Services · 59:56
- "Turning the database inside out with Apache Samza" by Martin Kleppmann · Strange Loop Conference · 47:42
- The Fast Fourier Transform (FFT): Most Ingenious Algorithm Ever? · Reducible · 28:23
- Modern Dictionaries by Raymond Hettinger · SF Python · 1:07:41
- Conflict Resolution for Eventual Consistency • Martin Kleppmann • GOTO 2016 · GOTO Conferences · 51:35
- CppCon 2014: Chandler Carruth "Efficiency with Algorithms, Performance with Data Structures" · CppCon · 1:13:41
- 01 - Course Introduction & Relational Model (CMU Databases Systems / Fall 2019) · CMU Database Group · 1:06:44
- CppCon 2017: Juan Pedro Bolivar Puente “Postmodern immutable data structures” · CppCon · 1:05:59
- Sorting Algorithms: Speed Is Found In The Minds of People - Andrei Alexandrescu - CppCon 2019 · CppCon · 1:29:55