It depends on the app purpose. If I’m splitting a larger system into multiple top-level OTP apps, then sure. But here, we’re discussing a client library. I don’t think that such library should try to be smart and host my connection processes. It’s my system, so I want to decide when to open the connection, and where to host it in my supervision tree.
But the biggest sin of KafkaEx is IMO the fact that it seems to open this connection during its startup (or at least it requires connection parameters during its startup), and to do that, it requires a global config (app env). That means we can only connect to one server, which seems quite limiting. Even if the library offers an opt-out through some app env, that approach is still clumsy and global. I can still see potential for conflicts, for example if one client app wants to use the default startup connection while another one wants to disable it.
Also, that approach complicates runtime configuration, as we’ve seen in this thread. We’re now discussing included applications or setting start type to :load simply because KafkaEx is overly eager about fetching its options and starting as soon as possible.
There’s a lot of needless clumsiness here IMO. Just give me a start_link function which takes its options as a combo of required and optional arguments. I can use standard OTP mechanisms to organize my code and runtime in arbitrary ways:
- I can start the connection when/where I want to.
- I can easily fetch secrets from wherever.
- I can still use app env to vary dev/test/prod parameters.
- I can easily separate concerns myself through umbrella apps.
There’s a lot of flexibility here, and the decisions are left to me, the end-user of the library.
Another nice benefit is that such interface can clearly specify which parameters are mandatory (e.g. host address), and which are optional (e.g. port). And with using typespec it can also specify type requirements for each parameters.
