I have been enjoying using Chumsky, but I am missing having a lexer pass because it is easier to handle newlines and padding here. I assume there are methods to accomplish this, but I haven't found any/come up with any I like.

My first thought was to create a first chumsky pass by using an enum of all the lexer tokens, and then parse the &'src str input into a Vec. Once I have the vec of tokens, I would then use the Vec as an input (instead of &'src str) to a second pass that would actually parse my language. This doesn't seem to work well, because I can't figure out a good way to discriminate between the enum variants besides "validate".

My second idea was to have individual parser rules for each lexer token, and then use these from a parser. I don't like this idea to much, but it should work. Priorities of lexer tokens would need to be managed at each call rather than once during the lexer pass.

Are there better methods for performing a lexer pass in Chumsky?

1 post - 1 participant

Read full topic