If you open a shell with iex -S mix run --no-start, you can load the Bumblebee models manually and the downloads will be cached for future runs.
EMLX is also not currently stable for usage as a compiler, so I recommend you use just the backend, or maybe even EXLA with the backend and compiler set
