I’m actually not using LLMs for the initial matching! I’m running embedding models via Bumblebee… which made self-hosting on a 3080 pretty feasible. Cost was definitely a factor, but honestly I also just wanted an excuse to dig into the Bumblebee/EXLA stack… it’s been fun to work with.
For Fly… Just a single instance right now. Traffic is low enough that geo-distribution isn’t necessary yet. :
