I would say go with Phoenix unless it is a very simple API that won’t grow in the future (which does not seem to be the case). The overhead of phoenix is negligible but makes many things easier.
Just exclude everything you don’t need:
mix phx.new my_api --no-assets --no-gettext --no-html --no-mailer
