Lexical Scanning in Go - Rob Pike

Channel:GoogleFOSSSydney · Length:51:17 · Published:2011-08-30 · Views:162231

Watch on YouTube

Summary

Rob Pike shows how the Go text/template lexer is built, using state functions, goroutines and channels. The talk was given at a Google Technology User Group in Sydney in 2011.

Why HN recommends it

HN returns to it for years as an elegant introduction to writing a lexer that is easy to follow even without a parsing background.

It's far more interesting than it sounds, and even if you don't know much about lexing/parsing it's easy to follow. — HN, 2014

C, C++, Rust, Go & systems languages · Compilers, interpreters & program analysis

Subtitles (excerpt)

okay we're now recording hi everyone um again welcome to uh tonight's GTG the August Google technology User Group here in Sydney um um tonight uh Rob Pike and myself will be giving a pair of talks about the go programming language um and I'd like to welcome Rob uh first up to talk about lexical scanning and go thank you are we good Tim yep okay uh yeah I'm here to talk about let's go scanning and go excuse me um thanks everyone for coming and of course this is streaming out live and will be available I think on YouTube later yes um yes I hear okay um so how many of you are familiar with the work of Michael Jackson really quite a few a couple maybe all right well as you know Michael Jackson developed Jackson structural programming which was a model uh based uh for solving uh structural mismatch problems in Computing um and these come up a lot and he really pointed out I think correctly that many programming problems deal with having data in one form and data in another form trying to connect them up and you find that the flow of control on both sides doesn't really work very well so you've got blocks coming in you want to turn to lines and you have partial lines and partial blocks and it's all kind of a mess and a lot of programming is very difficult to get right

On Hacker News

Related, I love Rob Pike's talk about lexical Scanning in Go (2011). Educational and elegant approach. [VIDEO] — hu3, 2024-11
Lexical Scanning in Go by Rob Pike [VIDEO] I love everything about this talk. It walks you through building a lexer from scratch in a simple and elegant way, through a very interesting use of coroutines. I appreciate the bits of humor in the talk as well. — grose, 2016-10
Rob Pike did a really good talk entitled 'Lexical Scanning In Go'[0], which covers how the lexing for text/template works. It's far more interesting than it sounds, and even if you don't know much about lexing/parsing it's easy to follow. [0] [VIDEO] — TomNomNom, 2014-05
Personally, I have a penchant for writing my own pull parsers. Its a mind-expanding exercise. The neat thing about Go is that parsers can return functions that consume the next token. Rob Pike has an excellent video about this: [VIDEO] — willvarfar, 2012-06
If you enjoyed the state machine + parsing part and also I recommend the older "Lexical Scanning in Go" talk that Rob Pike gave: [VIDEO] It's in Go, but you can apply most of it easily in other languages. I'm so happy that modern languages like Rust, Zig, and Go support unicode/runes/graphemes natively. So many problem just disappear compared to Java, .net, C++, or scripting languages. — Xeoncross, 2025-06

Related videos