Integration testing is hard

[LostKobrakai]

I have a few principles I try to follow when testing processes / sets of processes:

• Try to start processes in the test
  • isolated from anything else
  • no singletons or dependency on global state
• Application started processes are global state (best not to be depended on, unless they provide isolation on their own (e.g. ecto))
• Do not assert on processes internal state. Don’t look into the black box.
• Assert on things observable to the outside world using public APIs of the processes or interactions with other public resources
• Strongly prefer waiting on being messaged over sleeps.
• Don’t forget that the test is an isolated process, which can receive messages.
• If you want to test implemenation details used within a process, extract to a function and unit test the function.
• Consider stateful property tests. They’re involved, but also powerful. Can the the right tool depending on the context.
• Rearchitecting to aid testing is not a bad idea.

@hkrutzer

Mocking the repo or something near the database insert? Most of the mocking libraries are kind of iffy and can cause problems or don’t work async etc. Then there is Mox, but it only works with behaviors. So I’ve seen people add a @callback to a module, just so it can be mocked, there isn’t even a behaviour. I don’t like that because now we’re creating half of a behaviour just for tests. Additionally you need to put stuff in the Application env, causing clutter. That is two aspects of mocking with Mox that require adding test-only code in the regular (non-test) codebase. So I don’t want to do this because in most cases, other than Mox requiring it, there is no reason for adding a behaviour.

Imo this is a fallacy. The usecase for behaviours is to provide some level of interface where multiple implementations are to be used. If you have one implementation running, that’s hard to test, and a different one meant to aid in testing that’s multiple implementations. Even if an implementation happens to only be used in testing it’s just as bad if the implementation starts to drift apart with what is expected from the implementation as it would be for one used in production.

@hkrutzer

That’s why I try to avoid it, only test a single process / module as much as possible. But how do you convince other people to avoid these kinds of tests? In some cases they are not that hard to write, but you pay the price later anyway when you rewrite them and they are no longer easy.

I think the general arguments about unit tests vs integration tests apply here as well.