Well I’m not really aligned with your conclusions. If we want to implement a tool that is not fast (it searches through database or runs network calls to an Elastic search instance), then all other users will have to wait in line.
Typical handler execution: 10-5000ms (dominated by I/O)
5 seconds is a lot. If you have 3 users using the server at the same time, one user will wait 15 seconds before the calling llm displays anything.
