I’ve recently been interested in trying out Nx with Bumblebee so have been following along with the fantastic dockyard article on creating a basic RAG. However, i’m seemingly falling at the first hurdle with just getting the predict which uses Nx.Serving. I have the following: defmodule Rag.Embed…

Does the behavior change if you let the shell run for a while before requesting a prediction? Checking for the process will tell you if it’s started, but loading models etc happens after that. Also observe the machine’s CPU/GPU load while waiting: is there work being done?

If you open a shell with iex -S mix run --no-start, you can load the Bumblebee models manually and the downloads will be cached for future runs. EMLX is also not currently stable for usage as a compiler, so I recommend you use just the backend, or maybe even EXLA with the backend and compiler set

For what it’s worth: This Nx issue aims to help in debugging why EMLX is not on par with EXLA w.r.t. results. I have someone working on this now. Hopefully by next week we’ll have something out. https://github.com/elixir-nx/nx/issues/1606 And there are a few issues on the EMLX tracker related to …

How did you manage to get this working?

Elixir Programming Language Forum

Basic Bumblee text embedding run seemingly stalling

Questions & Help Questions

georgealexanderday May 7, 2025, 3:44pm 5

Managed to get it working, thanks for the help here folks!