Hello there!

I’ve been working for a while on a lexer/parser generator, which I’m already using in a couple of projects. I took the time to write some documentation so that others could benefit from it, though it still requires some work before I consider the project stable enough for a version 1.0: code documentation, refactoring, improved error messages, and parser optimizations. I must also make sure the unsupported grammar constructs are all detected.

Here are the main links:

• the project repository, on GitHub.
• the documentation: intro, tutorial, reference (my thanks to the mdbook team!)
• the crate of the executable (the project is split into 4 crates)

The main features are:

• predictive, non-recursive LL(1) parser
• automatic transformation of non-LL(1) elements: left-recursive and ambiguous rules, left factorization
• extended grammar language with * and + repetitions, ?, …
• listener pattern to avoid mixing grammar and Rust code, and to give more flexibility
• “low-latency” callbacks and ability to parser continuous streams.

The generated code includes a listener trait that you implement to interact with the parser, so it’s quite straightforward. To get an idea of what it looks like, see the README of the binary crate or the tutorial. There are a few examples in the repository, too.

If you’re curious about the grammar transformations, jump to this section of the reference. And for those who aren’t too familiar with parsers, I made a small introduction.

I hope it can be useful to someone else. Don’t hesitate to ask questions or give your feedback!

I’m planning to add support for LALR parsers at some point (I’ve already done a few experiments), and I’m also toying with the idea of making a little GUI app to test grammars. Not sure yet in which order or when that’ll happen.

For the little story:

I used to work with ANTLR in other languages, like Java and Kotlin, and I was missing the flexibility of the listener pattern. There was a port of that generator, but it was somewhat abandoned at the time, and the others I found didn’t really suit my needs.

I’m also interested in parsers and compilers in general, so I decided to write a lexer/parser generator in Rust that produces Rust code, and which has an approach similar to ANTLR but with a few key differences. The format of the lexicon and the grammar should look familiar to those who used that tool.

I saw in a recent video made by @jonhoo that the port mentioned above had been taken over by another team not long ago, so hopefully it’s active again (I think it moved elsewhere). Still, I had different requirements, so it doesn’t invalidate what I did.

1 post - 1 participant

Read full topic