# Nx vs. Python performance for sentence-transformer encoding

**URL:** <https://forum.elixirforum.com/t/nx-vs-python-performance-for-sentence-transformer-encoding/56502>\
**Category:** Questions\
**Tags:** benchmark, nx\
**Created:** [June 14, 2023, 3:53pm UTC](https://forum.elixirforum.com/t/nx-vs-python-performance-for-sentence-transformer-encoding/56502 "2023-06-14T15:53:14Z")\
**Posts on this page:** 1\
**Showing post:** 22

<div class="post-metadata">

**Author:** ![jonatanklosko](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jonatanklosko/32/37940_2.png) [@jonatanklosko](https://forum.elixirforum.com/u/jonatanklosko)\
**Post date:** [June 28, 2023, 10:36am UTC](https://forum.elixirforum.com/t/nx-vs-python-performance-for-sentence-transformer-encoding/56502/22 "2023-06-28T10:36:15Z")

</div>

> This leads me to believe that even though I had multiple servings the embedding creation is still sequential.

Only one computation can be running at a time on the given device, so spawning multiple servings is not going to help (unless you have a cluster of multiple nodes, each with its own serving).

Just to be sure, you are using `Nx.Serving.batched_run/2` and not `Nx.Serving.run/2` right?

> This is slower, it takes 76 seconds.

Just the computation or everything including application boot? Note that with `compiler: EXLA` the serving is going to compile everything into an efficient computation, which may take a bit, but then the computations themselves are faster.

> How do I determine the batch size and the sequence\_length?

`sequence_length` is used to pad/trim the input, usually you want to set it to the longest sequence the model supports, unless you know your inputs are always short and it can be reduced. `batch_size` depends on several factors, when running on a CPU increasing the `batch_size` at some point makes the computation take linearly as long since there is no more room for for parallelisation, when running on a GPU it depends on how much computation you can fit into the GPU memory; and in both cases depends on the expected number of requests (if it’s small then it doesn’t make sense to use large batch\_size).

---

_[View the full topic](https://forum.elixirforum.com/t/nx-vs-python-performance-for-sentence-transformer-encoding/56502)._
