# Is there a complete Elixir AST reference?

**URL:** <https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923>\
**Category:** Discussions\
**Tags:** ast\
**Created:** [April 13, 2021, 1:54pm UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923 "2021-04-13T13:54:57Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![dorgan](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dorgan/32/22107_2.png) [@dorgan](https://forum.elixirforum.com/u/dorgan)\
**Post date:** [April 13, 2021, 1:54pm UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923/1 "2021-04-13T13:54:57Z")

</div>

Lately I’ve been digging into the elixir AST for learning purposes. There is a section in the elixir docs([Syntax reference — Elixir v1.20.2](https://hexdocs.pm/elixir/syntax-reference.html#the-elixir-ast)) that describes the ast and gives some examples of it.

It mentions that the ast is made of:

> - atoms - such as `:foo`
> - integers - such as `42`
> - floats - such as `13.1`
> - strings - such as `"hello"`
> - lists - such as `[1, 2, 3]`
> - tuples with two elements - such as `{"hello", :world}`
> - tuples with three elements, representing calls or variables

Regarding the last point, it then says:

> the first element is an atom ( **or another tuple** ), the second element is a list of two-element tuples with metadata (such as line numbers) and the third is a list of arguments.

What I didn’t find so far is what is meant by “or another tuple”. Moreover, when that element is a tuple there seem to be some cases where elixir gives them a special meaning.

For example, consider the “dot syntax”:

```elixir
quote do
  foo.bar(:baz)
end

```

```elixir
{{:., [], [{:foo, [], Elixir}, :bar]}, [], [:baz]}

```

The first element there is the tuple `{:., [], [{:foo, [], Elixir}, :bar]}` that represents `foo . :bar`, so here the dot is a binary operator. If that is used as the first element in another three-tuple ast node, then it turns into `foo.bar()`, and any arguments will be put inside the parens, in this case `foo.bar(:baz)`.

This does not happen with other operators. For example, this node:

```elixir
{{:+, [], [{:foo, [], Elixir}, :bar]}, [], []}

```

Gets turned into `foo + :bar()`. So this tells me the dot expression is a special case. This in itself is not a surprise, but how many other special cases do exist?

By playing around I found another special case with the multi alias syntax. This expression:

```elixir
Foo.{Bar, Baz}

```

is represented by this ast:

```elixir
{{:., [],
  [
    {: __aliases__ , [], [:Foo]},
    :{}
  ]}, [],
 [
   {: __aliases__ , [], [:Bar]},
   {: __aliases__ , [], [:Baz]}
 ]}

```

The first element is again the dot operator tuple:

```elixir
{:., [], [{: __aliases__ , [], [:Foo]}, :{}]}

```

which in this case means `Foo . :{}`. The difference is that now if that tuple is used as the first element in another three-element tuple node, the arguments are put _inside of the curly braces_:

```elixir
Foo.{Bar, Baz}

```

No matter how bizarre the arguments are:

```elixir
{{:., [], [{: __aliases__ , [], [:Foo]}, :{} ]}, [], [{: __aliases__ , [], [:Bar]}, :baz, {:+, [], [1, 2]} ]}
#=> Foo.{Baz, :baz, 1 + 2}

```

So again, the dot operator when the right hand side is the `:{}` atom has special meaning.

I’m finding all of this by playing around, since the elixir docs don’t cover these cases.

The question is: is there a reference of the Elixir AST, or some section/comments/issue that documents all the cases where ast nodes are given special meaning?

---

<div class="post-metadata">

**Author:** ![marciol](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/marciol/32/4223_2.png) [@marciol](https://forum.elixirforum.com/u/marciol)\
**Post date:** [April 13, 2021, 5:31pm UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923/2 "2021-04-13T17:31:11Z")

</div>

I’m subscribing this topic. It’d be interesting to know if such documentation exists.

---

<div class="post-metadata">

**Author:** ![al2o3cr](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/al2o3cr/32/3457_2.png) [@al2o3cr](https://forum.elixirforum.com/u/al2o3cr)\
**Post date:** [April 14, 2021, 12:29am UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923/3 "2021-04-14T00:29:26Z")

</div>

The parser to handle `alias Foo.{Bar, Baz}` is:  
[https://github.com/elixir-lang/elixir/blob/b498c3790e570e341f24ae8e7b73b0d45eae9279/lib/elixir/src/elixir\_parser.yrl#L457-L460](https://github.com/elixir-lang/elixir/blob/b498c3790e570e341f24ae8e7b73b0d45eae9279/lib/elixir/src/elixir_parser.yrl#L457-L460)

`build_dot_alias` handles actually creating the AST structure.

The code to handle that AST is:  
[https://github.com/elixir-lang/elixir/blob/b498c3790e570e341f24ae8e7b73b0d45eae9279/lib/elixir/src/elixir\_expand.erl#L50-L62](https://github.com/elixir-lang/elixir/blob/b498c3790e570e341f24ae8e7b73b0d45eae9279/lib/elixir/src/elixir_expand.erl#L50-L62)

---

<div class="post-metadata">

**Author:** ![dorgan](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dorgan/32/22107_2.png) [@dorgan](https://forum.elixirforum.com/u/dorgan)\
**Post date:** [April 14, 2021, 1:50am UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923/4 "2021-04-14T01:50:46Z")

</div>

Thanks for the pointer! I was reading both the elixir .yrl file and the [Macro.to\_string/2 source](https://github.com/elixir-lang/elixir/blob/v1.11.4/lib/elixir/lib/macro.ex#L809) with the hopes of finding some useful info, and I got some.

It seems there’s only three cases in which a tuple is used as the first element, and all involve the dot operator in some way:

**Access syntax**  
The access syntax is represented as a `{:., [], [Access, :get]}` call with two arguments. The first is an expression node, and the second is the `key` used by `Access.get`:

```elixir
quote do
  foo[:bar]
end

#=>
{
  {:., [], [Access, :get]},
  [],
  [{:foo, [], Elixir}, :bar]
}

```

The dot is nowhere to be seen in the syntax, but I guess it’s used in the ast because it somewhat means “it’s a member of”. There’s very explicit checks for the `[Access, :get]` expression, and the ast node is being explicitly generated that way [by the parser](https://github.com/elixir-lang/elixir/blob/b498c3790e570e341f24ae8e7b73b0d45eae9279/lib/elixir/src/elixir_parser.yrl#L887).

**Dot alias**  
The dot alias(`foo.{:bar, :baz}` or `Foo.{Bar, Baz}`) is represented as a `{:., [], [left_hand_side, :{}]}` call, where the arguments are the elements inside the curly brackets:

```elixir
quote do
  foo.{:bar, 1 + 2}
end

#=>
{
  {:., [], [{:foo, [], Elixir}, :{}]},
  [],
  [:bar, {:+, [], [1, 2]}]
}

```

Note: this example will raise as it’s an invalid expression, but it’s allowed by the ast and it may be possible to use it as a very funky syntax for some ugly macro.  
This node is built in [this line](https://github.com/elixir-lang/elixir/blob/b498c3790e570e341f24ae8e7b73b0d45eae9279/lib/elixir/src/elixir_parser.yrl#L830).

**Generic calls**  
Every other kind of call(except “simple” calls like `foo(:bar)`) is represented as a `{:., [], [before_dot, after_dot?]}` call, where the arguments are the elements inside the parenthesis. In that tuple, the `before_dot` expression is the left hand side of the dot and is always present, but the `after_dot?` expression is optional, as in the case of `foo.(:bar)` (this case is covered by the Elixir syntax reference):

```elixir
quote do
  foo.bar(1, 2, 3)
end
#=> {{:., [], [{:foo, [], Elixir}, :bar]}, [], [1, 2, 3]}

quote do
  foo.(1, 2, 3)
end
#=> {{:., [], [{:foo, [], Elixir}]}, [], [1, 2, 3]}

```

* * *

Those seem to be all the cases, from what I could gather so far. If what I found is correct, I’d gladly submit a PR to extend the elixir docs on this, as it’s pretty much undocumented.

---

<div class="post-metadata">

**Author:** ![ityonemo](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ityonemo/32/11341_2.png) [@ityonemo](https://forum.elixirforum.com/u/ityonemo)\
**Post date:** [April 14, 2021, 2:15am UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923/5 "2021-04-14T02:15:24Z")

</div>

_Everything_, not just calls, except for what’s in that list (atom string float int string list 2-tuple) is a three element tuple. For example, map literals, and non-2-ary tuples are neither calls nor variables but they are also represented as a triple. Iirc the reason why 2-ary tuples are not represented as triples is because it makes parsing options keyword lists passed into a macro a nightmare.

E.g.

```elixir
  quote do {1, 2, 3} end

```

---

<div class="post-metadata">

**Author:** ![dorgan](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dorgan/32/22107_2.png) [@dorgan](https://forum.elixirforum.com/u/dorgan)\
**Post date:** [April 14, 2021, 2:26am UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923/6 "2021-04-14T02:26:29Z")

</div>

> [@ityonemo](#):
>
> _Everything_, not just calls, except for what’s in that list (atom string float int string list tuple) is a three element tuple.

According to the docs, three element tuples are “representing calls or variables”, so for instance `{:{}, [], [1, 2, 3]}` can be thought of as a call to the tuple constructor `{}/1`(which is a special form, like `%{}/1`, ` __MODULE__ /0` or `with/1`).  
My interest however was in the cases where the first element in that 3-tuple is another 3-tuple. I was writing a little code style checker like credo as an exercise and stumbled upon the “weird” case of the dot alias ast and thought “what else am I missing?”.

> [@ityonemo](#):
>
> Iirc the reason why 2-ary tuples are not represented as triples is because it makes parsing options keyword lists passed into a macro a nightmare.

Yes, I remember reading that is the reason but I can’t remember exactly where I read that.

---

<div class="post-metadata">

**Author:** ![AndyL](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/andyl/32/3659_2.png) [@AndyL](https://forum.elixirforum.com/u/AndyL)\
**Post date:** [April 14, 2021, 5:12am UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923/7 "2021-04-14T05:12:51Z")

</div>

Is there a way to generate an AST that preserves comments? How does the Elixir formatter do it?

---

<div class="post-metadata">

**Author:** ![hauleth](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/hauleth/32/18942_2.png) [@hauleth](https://forum.elixirforum.com/u/hauleth)\
**Post date:** [April 14, 2021, 7:49am UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923/8 "2021-04-14T07:49:12Z")

</div>

> [@dorgan](#):
>
> My interest however was in the cases where the first element in that 3-tuple is another 3-tuple.

Yes, in any situation where you have call on the left hand of `()` (call) operator. There aren’t many of these, only 2 to be exact:

1. `.` operator, like `mod.func()`
2. another call, like `foo()()`

Of course 2nd option is possible in only 1 situation, `unquote(atom)()`, and will fail to compile in any other case, however it is possible to have such construct in the `quote`.

```elixir
quote do: foo()()
# => {{:foo, [], []}, [], []}

```

To see how it work when `unquote/1` is used you need to disable unquoting first:

```elixir
quote [unquote: false], do: unquote(:foo)()
# => {{:unquote, [], [:foo]}, [], []}

```

---

<div class="post-metadata">

**Author:** ![dallagi](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dallagi/32/21521_2.png) [@dallagi](https://forum.elixirforum.com/u/dallagi)\
**Post date:** [April 14, 2021, 12:30pm UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923/9 "2021-04-14T12:30:46Z")

</div>

Looking at the code, the formatter does not put comments in an AST.  
Rather it seems to pass a callback to the tokenizer to process comments (see [here](https://github.com/elixir-lang/elixir/blob/3f4ee226dfcc0e4dcba00426ad10fc49ebfdb424/lib/elixir/lib/code/formatter.ex#L209)) and then it generates an algebra document from the comments and the AST.

The formatting is then done on the algebra document, not on the AST.

---

<div class="post-metadata">

**Author:** ![dorgan](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dorgan/32/22107_2.png) [@dorgan](https://forum.elixirforum.com/u/dorgan)\
**Post date:** [April 14, 2021, 1:23pm UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923/10 "2021-04-14T13:23:55Z")

</div>

This is correct, the formatter works with an algebra document since quite a lot of data is lost when parsing to the AST  
With [https://ast.ninja/](https://ast.ninja/) you can play around and see both the AST and the albegra document for any given elixir code(click `Add` and then `Code.Formatter.to_algebra/2`)

---

<div class="post-metadata">

**Author:** ![dorgan](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dorgan/32/22107_2.png) [@dorgan](https://forum.elixirforum.com/u/dorgan)\
**Post date:** [April 16, 2021, 6:49pm UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923/11 "2021-04-16T18:49:05Z")

</div>

If you’re still interested, I wrote a blog post with everything I learned so far about the AST:

> **[A deep dive into the Elixir AST: Introduction](https://dorgan.netlify.app/posts/2021/04/the_elixir_ast/)**
>
> ドラゴン

I’ll try and see if I can make a PR to the elixir docs to fill the missing parts in the syntax reference

---

<div class="post-metadata">

**Author:** ![arjan](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/arjan/32/2737_2.png) [@arjan](https://forum.elixirforum.com/u/arjan)\
**Post date:** [April 20, 2021, 9:27am UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923/12 "2021-04-20T09:27:55Z")

</div>

> [@dorgan](#):
>
> With [https://ast.ninja/](https://ast.ninja/) you can play around and see both the AST and the albegra document for any given elixir code(click `Add` and then `Code.Formatter.to_algebra/2`)

Was just about to post the link to my AST Ninja but I see somebody already beat me 🙂

---

<div class="post-metadata">

**Author:** ![dorgan](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dorgan/32/22107_2.png) [@dorgan](https://forum.elixirforum.com/u/dorgan)\
**Post date:** [April 20, 2021, 12:24pm UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923/13 "2021-04-20T12:24:59Z")

</div>

It’s a really helpful tool, thanks for it 🙂

---

<div class="post-metadata">

**Author:** ![Rich\_Morin](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/rich_morin/32/2020_2.png) [@Rich\_Morin](https://forum.elixirforum.com/u/Rich_Morin)\
**Post date:** [April 25, 2021, 6:00pm UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923/14 "2021-04-25T18:00:39Z")

</div>

As I just posted in [José announces Livebook - a web application for writing interactive and collaborative code notebooks](https://forum.elixirforum.com/t/jose-announces-livebook-a-web-application-for-writing-interactive-and-collaborative-code-notebooks/38932), I think that Livebook could be a great way to generate this sort of reference documentation (e.g., with executable examples).

-r

---

<div class="post-metadata">

**Author:** ![dorgan](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dorgan/32/22107_2.png) [@dorgan](https://forum.elixirforum.com/u/dorgan)\
**Post date:** [April 27, 2021, 8:11pm UTC](https://forum.elixirforum.com/t/is-there-a-complete-elixir-ast-reference/38923/15 "2021-04-27T20:11:48Z")

</div>

I’ve seen a couple people already using Livebook to try stuff or document code from existing codebases, it seems to be great at that. I’m not sure it would be a good idea for public documentation like ex\_doc(because of remote code execution), but for internal documentation it’s great.

If making it safe to use for public facing documentation is desired, I guess that Livebook having pluggable “interpreters” for the snippets could enable that. A “safe” interpreter could do some ast whitelisting, something like what [Sand](https://forum.elixirforum.com/t/sand-an-elixir-sandbox/35742) or [tryelixir](https://tryelixir.net/) do.