NimbleParsec - a simple and fast parser combinator for Elixir

[tmbb]

@josevalim

We don’t give the byte location. We do give UTF-8 line and column.

Good enough for me. Actually, for my purposes there are no parse errors (Makeup lexers should consume every binary passed as an argument)

@josevalim

I am not quite sure what you mean but if you annotate each token properly, you have all of the information that you need that then can be reduced into a final result with reduce/3.

This is definitely not the same thing. You can’t parse context-sensitive languages without storing the context somewhere. You’d have to change the grammar into something that’s context-free either in a preprocessing stage or at a later stage. For example, how would you, using nimble write a parser that would detect when a line is less indented than the previous line? For example, this bit of python code:

def f(x):
    if x > 0:
        return x
    else:
        return -x

Blocks are dictated by indentation, so you need to store the indentation stack somewhere and compare it with the current indentation level. With ExSpirit, it’s trivial to write indent() and dedent() parsers that recognize this (so that you can match a block as indent() |> block() |> dedent() or something like that). AFAIK, you can’t do this if you don’t store the context somewhere. This is not a showstopper for many applications, but when you need these kinds of context-dependent parser, you really need them.

(Actually the Python parser is context-free; During a preprocessing stage, the lexer introduces some special INDENT and DEDENT tokens which can be matched by the parser as if they were matching parenthesis)

So yeah, if you apply arbitrary transformations to your string before or after parsing you don’t need the context, but in some cases it might be simpler to just keep a context.

@josevalim

Btw, I have one question. It annoys me that the tokens emitted by the parser do not contain the line+column information.

What API do you propose for that? Doing this kind of thing by default is kinda hard… Which tokens would you want to tag? what is a token?

@josevalim

But none of the parser combinator libraries seem to do it by default. How do you tackle this in ex_spirit?

You don’t. ExSpirit doesn’t handle this by default. It’s trivial to define a parser combinator that does it, though. You just take the old context (before running the inner parser) and the new context (after running the inner parser), and this will give you old_position and new_position with which you can annotate the token if you wish. In ExSpirit you can do it like this: ExSpirit.Parser – ex_spirit v0.4.0

I think this does require first-class access to the context map (in your case a tuple, although a record might be even better).