Unfortunately I've never written docs for the format. The folks over at Couchbase did port this library to Go though, and in the course of doing it, they sketched out a format in writing: https://github.com/couchbase/vellum/blob/master/docs/format.md --- I haven't combed over it detail though, so I don't know how accurate it is. And there have been some minor changes (like support for CRC32) since that document was written.

Overall, the format itself was my own devising, but took a lot of inspiration from the various papers linked in the docs with respect to compressing finite automata. An FST is a "compressed data structure," meaning that its native form is itself also compressed. So for example, most states inside an FST are represented by a single byte.