Regarding req_llm, they are honestly very similar. Both libraries attempt to solve the same problem and approach it in very similar ways. Both libraries are clearly influenced by the Vercel AI SDK, but in Omni I’ve taken a lot of inspiration from the Pi codebase, which I think steers it in a slightly leaner direction.
- Both have a almost identical top-level API:
generate_text(model, context, inference_opts)andstream_text(model, context, inference_opts) - Both attempt to establish a canonical data model that works across all LLM provider APIs. There are some minor differences in the shape of those data models, but essentially they’re doing the same thing.
- Both source model data from the models.dev API.
- Both can generate text, structured objects, can stream responses, track usage tokens and costs.
- Both use Req under the hood.
Some gaps (some of these will narrow over time):
- req_llm supports 45 providers with ~665 models - omni supports 6 providers with ~300 models
- req_llm supports image generation, omni does not
- Omni takes a slightly pragmatic view with more obscure inference options like top_p, logprobs etc and doesn’t currently attempt to support every option for every provider - req_llm appears to be have a bit more complete coverage
- Omni provides a simple `Omni.Agent` genserver as a building block for agents - req_llm does not have anything like this (but is part of the wider Jido ecosystem)
Lower level differences:
- Omni splits a provider into two behaviours - the Provider and the Dialect (the wire format). This should make adding new providers much quicker and easier to maintain, as most providers in the wild share dialects. This also makes it easy for users to create their own providers.
- Omni is streaming first - so even
generate_text/3is a streaming request that is accumulated in one call. This means a dialect only needs to care about streaming requests - resulting in simpler implementations.
