It is definitely possible. A RAG system has three components:
- Models for generating embeddings
- An index
- A LLM
You will find support for generating embeddings in Bumblebee. You need to pick a model though and sbert is a starting point: https://www.sbert.net/
Indexes is the area we have least developed on. There are both ExFAISS and hnswlib bindings on GitHub. We want to officially release the latter at some point. Alternatively, you can pick a vector database or even PG with pg_vector for this step, which I would recommend.
Then you need to pick a LLM, either with Bumblebee or off the shelf.
Here is a post, a bit dated, that gives you more pointers: Semantic Search with Phoenix, Axon, Bumblebee, and ExFaiss - DockYard
Honestly, implementing this has both technical moving parts but business building parts. What is the best model for your use case? Best embeddings? How to generate embeddings for your documents? Etc. my suggestion would be to pick an off the shelf solution to evaluate the results and build a prototype, and only then evaluate what makes sense to bring in-house for performance, value, security reasons.
In case it matters, I am speaking both as a library author and as someone who has built more than one proof of concept RAG system. 
