Wow, thanks for looking into this so thoroughly!
Is this a fair summary?
- ~600 MB memory is expected, because of overly large default cache size
- However, it did balloon above that when killing all Repo processes. Seems like on kill we do not cleanly close the DB connection, which would explain it.
- Our NIF memory usage does not show in observer. Imo this may be due to the fact that SQLite3 does its own memory management and does not use the NIF alloc functions (see below).
This is done in the ecto_sqlite3 library defaults. We have that opinionated default in the documentation, but it was decided somewhat arbitrarily. See this issue. I created a new issue to consider lowering it, it definitely seems like 2MB is a fine enough default. I created an issue for that, here.
If you could share snippets of your benchmarking code or just the more specific process or calls you do in the IEX or whatever (for those with no experience in these things, like myself), that would be greatly appreciated! I could use that info to try debugging further.
Yes, this could perhaps be done. As mentioned above, I think the fact SQLite3 uses its own malloc is why this is. It may be possible to provide the erlang memory management functions at DB init time, which could incidentally fix the memory leak issue we saw, perhaps.
See this SQLite3 documentation. Not sure if the NIF memory functions provided can fit this interface entirely, though.
