# Leex regex problem

**URL:** <https://forum.elixirforum.com/t/leex-regex-problem/13601>\
**Category:** Questions\
**Tags:** leex, regex\
**Created:** [April 10, 2018, 9:01pm UTC](https://forum.elixirforum.com/t/leex-regex-problem/13601 "2018-04-10T21:01:43Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![kokolegorille](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kokolegorille/32/4784_2.png) [@kokolegorille](https://forum.elixirforum.com/u/kokolegorille)\
**Post date:** [April 10, 2018, 9:01pm UTC](https://forum.elixirforum.com/t/leex-regex-problem/13601/1 "2018-04-10T21:01:43Z")

</div>

Hello everyone,

I am having some trouble understanding leex behaviour. To give some context, I would like to create a parser for pgn files.

Here is some code to explain my problem.

I created the lexer file pgn\_lexer.xrl with

```elixir
% -- Definitions.

Definitions.

TAG = (\[[^\]]*\])+
COMMENT = ({[^}]*})+
WHITESPACE = [\s\t\n\r]

Rules.

{TAG} : {token, {tag, TokenLine, TokenChars}}.
{COMMENT} : {token, {comment, TokenLine, TokenChars}}.
{WHITESPACE}+ : skip_token.

Erlang code.

```

The important rule is for TAG

```elixir
( \[[^\]] * \] )+

```

I cannot understand why

```elixir
iex> :pgn_lexer.string '[Tag1 "Value1"] [Tag2 "Value2"]{comment}'                                                        
{:ok,
 [
   {:tag, 1, '[Tag1 "Value1"]'},
   {:tag, 2, '[Tag2 "Value2"]'},
   {:comment, 2, '{comment}'}
 ], 2}
# This works has expected, it detects 2 tags

iex> :pgn_lexer.string '[Tag1 "Value1"][Tag2 "Value2"]{comment}'  
{:ok, [{:tag, 1, '[Tag1 "Value1"][Tag2 "Value2"]'}, {:comment, 1, '{comment}'}],
 1}
# This does not work, tag1 and tag2 are merged

```

Why do the tags need to be separated by a space?

Thanks for enlightments.

---

<div class="post-metadata">

**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\
**Post date:** [April 10, 2018, 9:22pm UTC](https://forum.elixirforum.com/t/leex-regex-problem/13601/2 "2018-04-10T21:22:35Z")

</div>

The reason is that regexes in Leex are greedy. Your regex in this case matches the longest string possible. Here’s a example using your regex and data in IEx:

```elixir
iex> r = ~r/(\[[^\]]*\])+/
~r/(\[[^\]]*\])+/

iex> String.split "[Tag1 \"Value1\"] [Tag2 \"Value2\"]{comment}", r, include_captures: true
["", "[Tag1 \"Value1\"]", " ", "[Tag2 \"Value2\"]", "{comment}"]

iex> String.split "[Tag1 \"Value1\"][Tag2 \"Value2\"]{comment}", r, include_captures: true 
["", "[Tag1 \"Value1\"][Tag2 \"Value2\"]", "{comment}"]

```

In general I find these issues become prevalent when you’re using Leex to _parse_ instead of just _tokenize_. (I know this because I’ve spent hours on similar cases myself). My learning is to use Leex to tokenise and Yecc to parse.

Or look at [nimble\_parsec](https://hex.pm/packages/nimble_parsec) or [ex\_spirit](https://hex.pm/packages/ex_spirit)

---

<div class="post-metadata">

**Author:** ![kokolegorille](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kokolegorille/32/4784_2.png) [@kokolegorille](https://forum.elixirforum.com/u/kokolegorille)\
**Post date:** [April 10, 2018, 9:25pm UTC](https://forum.elixirforum.com/t/leex-regex-problem/13601/3 "2018-04-10T21:25:37Z")

</div>

Thank You for the answer, now I know why, and I know why I am spending hours too 🙂

---

<div class="post-metadata">

**Author:** ![NobbZ](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/nobbz/32/27235_2.png) [@NobbZ](https://forum.elixirforum.com/u/NobbZ)\
**Post date:** [April 10, 2018, 9:48pm UTC](https://forum.elixirforum.com/t/leex-regex-problem/13601/4 "2018-04-10T21:48:55Z")

</div>

> [@kip](#):
>
> The reason is that regexes in Leex are greedy

Yes, they are greedy, but thats not the cause here.

`TAG` is defined as `(\[[^\]]*\])+` and thus we demand at least one but allow many repitions of square-bracket-pairs-with-stuff-inbetween and consolidate them into one token.

If the second example is expected to spit out 2 `tag` tokens, then `TAG` should be just `(\[[^\]]*\])` (without the `+`), now we should even be able to remove the grouping parens without changing anything.

---

<div class="post-metadata">

**Author:** ![kokolegorille](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kokolegorille/32/4784_2.png) [@kokolegorille](https://forum.elixirforum.com/u/kokolegorille)\
**Post date:** [April 10, 2018, 9:52pm UTC](https://forum.elixirforum.com/t/leex-regex-problem/13601/5 "2018-04-10T21:52:26Z")

</div>

That did the trick, thank You very much.

Here is the modified version…

```elixir
TAG = (\[[^\]]*\])

iex> :pgn_lexer.string '[Tag1 "Value1"][Tag2 "Value2"]{comment}' 
{:ok,
 [
   {:tag, 1, '[Tag1 "Value1"]'},
   {:tag, 1, '[Tag2 "Value2"]'},
   {:comment, 1, '{comment}'}
 ], 1}

```

And as mentionned, the short version works as well

```elixir
TAG = \[[^\]]*\]

```

---

<div class="post-metadata">

**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\
**Post date:** [April 10, 2018, 10:00pm UTC](https://forum.elixirforum.com/t/leex-regex-problem/13601/6 "2018-04-10T22:00:43Z")

</div>

Arrrggghhhhh sorry for my misleading reply.