I am trying to use Meta’s Llama 2 with Bumblebee, but I am getting a 401 error when I try to load it. I have been granted access to the repo on Hugging Face, but I think I need to provide an access token when loading the model, and I am not sure how to do so. {:ok, model} = Bumblebee.load_model({:…

Hey @djaouen , you can specify it in repository options: {:hf, "meta-llama/Llama-2-7b-chat-hf", auth_token: "..."} : )

I don’t think Bumblebee supports that for now? Does it? @seanmor5 EDIT: Correction, there seems to be auth_token but for cached downloads https://github.com/elixir-nx/bumblebee/blob/main/lib/bumblebee/huggingface/hub.ex#L41

Thanks for looking into this for me. Is there a way to utilize this with load_model, load_tokenizer, and load_generation_config?

Thanks, @regex.sh and @jonatanklosko !

Not yet. I was waiting for Bumblebee to get upgraded so that I could use Bumblebee.Text.Llama.

Looks like v0.3.1 has the official support for Bumblebee.Text.Llama. Is that not working for you?

Actually, I don’t know what version of Bumblebee I was using, as it’s been long enough that Hugging Face deleted my notebook. But I will certainly look into that if I build out a similar project in the future. Thanks for the info!

I was able to get “NousResearch/Llama-2-7b-hf” working using 0.3.1. Does anyone know if we can use GGML locally so i don’t have to wait for 5minute inference for two sentences on my M2?

You can go ahead and do the bindings for llama.cpp yourself. Check out Erlang NIFs. Otherwise, docs for LLaMA have been added here: https://hexdocs.pm/bumblebee/llama.html

In case someone still needs Llama doc, it’s not available on the latest doc and can be found here https://hexdocs.pm/bumblebee/0.4.2/llama.html

Elixir Programming Language Forum

Using Llama 2 With Bumblebee

Questions & Help Questions

haavars August 30, 2023, 4:09pm 6

Did you ever get i to work?