I stumbled upon this: https://medium.com/@kaisersly/storing-any-elixir-data-in-postgres-e8f93367f473 I am looking for a way to store transient data in PostgreSQL. Basically, LLMChains that can grow big in size. Is there any better way to do it ? The article suggests using :binary and Term. Or is …

Maybe, just maybe, you can add two columns, one is for the key (indexed), and the other is for the arbitrary Erlang/Elixir data structure encoded in binary.

How big is big? Past a certain point you probably want an object store. Before that though binary and term are quite useful. If you want want to be able to search it from within the DB though consider :jsonb

If size is an issue you either use term_to_binary with compression or just use a full-blown object store as @benwilson512 suggested.

@benwilson512 What do you mean by object store?

Amazon S3 or Minio are examples of object stores. They’re also often called blob storage.

I need to look them up by a key, so yes. The data can get big, since it is a chain of AI messages

I think it all comes down to how big the data for each item will be. Yes, if it gets big, then maybe yes, this way in a standard PostgreSQL DB, not memory, mnesia or ETS.

[image] Aetherus: and the other is for the arbitrary Erlang/Elixir data structure encoded in binary. Can it be stored compressed? The binary data?

External Term Format — OTP 29.0.2 (erts 17.0.2) seems to say that the data is zlib-compressed.

:erlang.term_to_binary(term) compresses the data by default. :erlang.term_to_binary(term, compressed: 9) compress the data at highest level. :erlang.term_to_binary(term, compressed: false) turns off the compression.

Elixir Programming Language Forum

Storing any type of Elixir data in PostgreSQL

Questions & Help Questions

Aetherus December 9, 2024, 4:39am 5

Do you need to index them?