I wanted to use tiktoken for splitting up some text into tokens before sending to OpenAI’s APIs, and I ended up writing the program as a Python script, and interfacing with it from Elixir using a Port.
Depending on your deployment this does complicate things a fair bit since you’ve got the extra language/tools to think about, so unless you really really need something from Python, doing it in Elixir would be easier.
