# :erlang.term\_to\_binary and postgresql =\> invalid byte sequence for encoding "UTF8" **URL:** \ **Category:** Questions\ **Tags:** ecto\ **Created:** [November 21, 2023, 6:49am UTC](https://forum.elixirforum.com/t/erlang-term-to-binary-and-postgresql-invalid-byte-sequence-for-encoding-utf8/59862 "2023-11-21T06:49:09Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Terbium-135](https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/t/5f9b8f/32.png) [@Terbium-135](https://forum.elixirforum.com/u/Terbium-135)\ **Post date:** [November 21, 2023, 6:49am UTC](https://forum.elixirforum.com/t/erlang-term-to-binary-and-postgresql-invalid-byte-sequence-for-encoding-utf8/59862/1 "2023-11-21T06:49:09Z")
After reading this: > [@Acceptance of Erlang's 'term\_to\_binary' and vice versa in Elixir](https://forum.elixirforum.com/t/acceptance-of-erlangs-term-to-binary-and-vice-versa-in-elixir/11034): > > I’m playing with Elixir - It’s fun. I think @rvirding does give Elixir courses these days. Re: files and database - when I given Erlang courses I say the three best things about erlang are processes, links and term\_to\_binary and the same should be true of Elixir (since these have nothing to do with the surface syntax and are properties of the beam VM) I may be wrong but I don’t seem much use of term\_to\_binary in Elixir - this is the great way to store anything on disk or communicate between s… I tried to use term\_to\_binary together with postgresql: Migration: ```elixir [...] add :owner, :text add :version, :text add :state, :bytea [...] ``` Schema: ```elixir [...] field :state, :binary [...] ``` Code: ```elixir update_or_insert(%State{owner: owner, state: :erlang.term_to_binary(state)}) def update_or_insert(state) do state |> Repo.insert( on_conflict: {:replace_all_except, [:id, :owner, :created_at]}, conflict_target: [:owner], set: [updated_at: DateTime.utc_now()] ) end ``` Error: ```elixir 07:11:53.186 [error] GenServer #PID<0.15770.0> terminating ** (Postgrex.Error) ERROR 22021 (character_not_in_repertoire) invalid byte sequence for encoding "UTF8": 0x83 ``` How to solve this? Change encoding for the resulting binary? Leave term\_to\_binary alone and use JSON encoding/decoding? I was also looking at the docs for erlang term\_to\_binary but there are only compression levels mentioned > **[erlang — OTP 29.0.2 (erts 17.0.2)](https://www.erlang.org/doc/apps/erts/erlang.html)** ---
**Author:** ![jhogberg](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jhogberg/32/24176_2.png) [@jhogberg](https://forum.elixirforum.com/u/jhogberg)\ **Post date:** [November 21, 2023, 8:28am UTC](https://forum.elixirforum.com/t/erlang-term-to-binary-and-postgresql-invalid-byte-sequence-for-encoding-utf8/59862/2 "2023-11-21T08:28:49Z")
`term_to_binary/1,2` produces a binary in the external term format, not a UTF-8-encoded binary, so this seems like a problem with the schema (by the looks of it on the database server side). Try looking into why `field :state, :binary` implies UTF-8 encoding. ---
**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\ **Post date:** [November 21, 2023, 8:50am UTC](https://forum.elixirforum.com/t/erlang-term-to-binary-and-postgresql-invalid-byte-sequence-for-encoding-utf8/59862/3 "2023-11-21T08:50:16Z")
My guess is that its resulting from an encoding error in either the `owner` or `version` fields since the Postgres `text` type maps to Elixir `String.t` type - which is UTF-8. Therefore one possible cause is that that either the Postgres database or the database connection string are set to LATIN-9 or some other non-UTF-8 encoding. The `bytea` type maps to Elixir/erlang `binary` as you have configured. ---
**Author:** ![rvirding](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/rvirding/32/1409_2.png) [@rvirding](https://forum.elixirforum.com/u/rvirding)\ **Post date:** [November 21, 2023, 5:55pm UTC](https://forum.elixirforum.com/t/erlang-term-to-binary-and-postgresql-invalid-byte-sequence-for-encoding-utf8/59862/4 "2023-11-21T17:55:55Z")
I still mainly do Erlang courses but I play with Elixir as well. 😄 ---
**Author:** ![rvirding](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/rvirding/32/1409_2.png) [@rvirding](https://forum.elixirforum.com/u/rvirding)\ **Post date:** [November 21, 2023, 5:57pm UTC](https://forum.elixirforum.com/t/erlang-term-to-binary-and-postgresql-invalid-byte-sequence-for-encoding-utf8/59862/5 "2023-11-21T17:57:18Z")
Yes, the ETF is an internal format and is definitely not UTF-8 encoded. ---
**Author:** ![Terbium-135](https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/t/5f9b8f/32.png) [@Terbium-135](https://forum.elixirforum.com/u/Terbium-135)\ **Post date:** [November 22, 2023, 7:06pm UTC](https://forum.elixirforum.com/t/erlang-term-to-binary-and-postgresql-invalid-byte-sequence-for-encoding-utf8/59862/6 "2023-11-22T19:06:56Z")
Thanks for chiming in @jhogberg and @kip! This was interesting to solve. The DB is set to UTF8: ![2023-11-22 08_41_28-sudo -Hiu postgres psql](https://forum.elixirforum.com/uploads/default/original/3X/7/2/72dfc2812c32b088e4d959b0cfbc92e3622cc96b.png) While I was trying to narrow down the problem I got a different error message: ```elixir 09:44:01.258 [error] GenServer #PID<0.7186.0> terminating ** (Postgrex.Error) ERROR 22023 (invalid_parameter_value) payload string too long (ecto_sql 3.11.0) lib/ecto/adapters/sql.ex:1054: Ecto.Adapters.SQL.raise_sql_call_error/1 ``` Which leads to the real problem The table has a trigger attached - and that trigger is converting the updated/inserted table row to JSON to send a message using `pg_notify` =\> trying to convert the binary data… And I was able to verify it as soon as I had a look at the postgresql log files **Never thought of this** 🤪