Testing Ash – share your design and best practices

Questions & Help Questions
, , ,
2

This is a very well written post. Thank you :bowing_man:. As part of the documentation overhaul, we will be adding some testing guidelines. I believe that testing is very situational, but there are a few things that make testing Ash code a bit unique.

Here are some things, some in response to your questions, some things worth knowing while testing, in no particular order :slight_smile:

Actions vs Seeds

While I often suggest using your resource actions to create data when you can, you may also need to reach for seeds. This can often make large differences in the speed of your tests. Ash has tools for this in Ash.Seed. For example:

Ash.Seed.seed!(%Ticket{})`

Additionally, @jimsynz has created a layer on top of Ash.Seed with great ergonomics that many folks enjoy called Smokestack.

Generators and StreamData

Ash has utilities built in that work with StreamData to allow for generating valid values (and in the future invalid values). This can be used with Ash.Generator, for example:

Ash.Generator.many_changesets(%Ticket{}, :update, 10)

And it can also be used for property testing:

check all(input <- Ash.Generator.action_input(Resource, :create) do
  ...
end

Handling metadata

You can solve this one of two ways:

  1. use pattern matching or more specific assertions, not ==. This doesn’t work everywhere, but can be useful.

For example:

assert [%{id: ^ticket_id}] = 

assert [result] = Ash.Query.for_read(Ticket, :read) |> Support.read!()
assert result.id == ticket.id
  1. Use Ash.Test.strip_metadata/

There is a tool for this, called Ash.Test.strip_metadata/1. This can let you make your assertions, i.e

assert strip_metadata(Ash.Query.for_read(Ticket, :read) |> Support.read!()) == strip_metadata([ticket])

You can call it on result tuples or lists, and it will strip the metadata of any contained records.

Testing APIs

Honestly, it isn’t generally possible for Ash to produce something that doesn’t match the schema it defines. I personally would be testing specific behavioral things. (and I’d use AshJsonApi.Test to do it). For example:

  import AshJsonApi.Test
  
  @tag :phoenix
  test "lists all tickets", %{conn: conn, ticket: ticket} do
    MyApp.Support
   # get and assert on status
    |> get("/api/v1/support/tickets", status: 200) 
    |> assert_data_equals([%{...}, %{...}])
  end

I think there are some potentially missing assertions for AshJsonApi that would be really nice, like assert_data_matches where you provide a pattern, and so on. But you could copy the pattern of our assertions (and maybe PR some to ash_json_api, as they assert and return the original conn. (Actually, I found a few assertions that weren’t doing this while checking this out, and have pushed dup a fix for them :slight_smile: )

But ultimately, the point is that with Ash you don’t really have to test “can it turn an instance of my resource into a valid response”, you need to test the behavior of the action. i.e that all existing tickets are contained in the response, or whatever other action-specific things you have going.

Unit Testing in Ash is different

For example, you can unit test a calculation directly, i.e

calculate :full_name, :string, expr(first_name <> " " <> last_name)

assert Ash.calculate!(User, :full_name, refs: [first_name: "Zach", last_name: "Daniel"]) == "Zach Daniel"

When testing this calculation, I would likely do it at the resource level, not the api level. This same thing is true for a lot of cases for me. I would likely have most of my tests at the action/Resource level.

Conclusion

So, that was a lot, and I’m not sure I’ve truly answered all your questions, but its all the time I have left right now, so I’ll leave it at that for the time being :laughing:. Let me know your thoughts and I will respond tomorrow :bowing_man: