# (Syntax Error) invalid or reserved Unicode code point

**URL:** <https://forum.elixirforum.com/t/syntax-error-invalid-or-reserved-unicode-code-point/41604>\
**Category:** Questions\
**Tags:** iex, httpoison\
**Created:** [August 11, 2021, 5:31pm UTC](https://forum.elixirforum.com/t/syntax-error-invalid-or-reserved-unicode-code-point/41604 "2021-08-11T17:31:34Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Doerge](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/doerge/32/19632_2.png) [@Doerge](https://forum.elixirforum.com/u/Doerge)\
**Post date:** [August 11, 2021, 5:31pm UTC](https://forum.elixirforum.com/t/syntax-error-invalid-or-reserved-unicode-code-point/41604/1 "2021-08-11T17:31:34Z")

</div>

Hey everybody,

I get a response from a (pretty funky) API, and am trying to decode it with HTTPoison.  
It fails with something like this:

```elixir
{:error,
 %Poison.ParseError{
   data: " { \"config\": ...",
   skip: 86293,
   value: "\\ud83d"
 }}

```

Toying around with it in IEx also doesn’t work:

```
iex(1)> tmp = "\ud83d"
** (SyntaxError) iex:1:8: invalid or reserved Unicode code point \u{d83d}. Syntax error after: \u

```

I’m not really sure how to proceed from here.  
How can I clean up my string from invalid code points before decoding it? It’s ok for me to just drop the invalid ones, but I would like to keep the valid ones.

---

<div class="post-metadata">

**Author:** ![03juan](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/03juan/32/23111_2.png) [@03juan](https://forum.elixirforum.com/u/03juan)\
**Post date:** [August 11, 2021, 5:38pm UTC](https://forum.elixirforum.com/t/syntax-error-invalid-or-reserved-unicode-code-point/41604/2 "2021-08-11T17:38:24Z")

</div>

You could sanitise the raw response with `Enum.reduce` before trying to decode it, dropping each value that results in an incorrect codepoint.

---

<div class="post-metadata">

**Author:** ![Doerge](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/doerge/32/19632_2.png) [@Doerge](https://forum.elixirforum.com/u/Doerge)\
**Post date:** [August 11, 2021, 5:56pm UTC](https://forum.elixirforum.com/t/syntax-error-invalid-or-reserved-unicode-code-point/41604/3 "2021-08-11T17:56:41Z")

</div>

I don’t know which ones are invalid, so I don’t really know which ones to drop?  
Is there an allow-list of valid ones builtin somewhere?

---

<div class="post-metadata">

**Author:** ![Doerge](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/doerge/32/19632_2.png) [@Doerge](https://forum.elixirforum.com/u/Doerge)\
**Post date:** [August 11, 2021, 6:01pm UTC](https://forum.elixirforum.com/t/syntax-error-invalid-or-reserved-unicode-code-point/41604/4 "2021-08-11T18:01:57Z")

</div>

I realize that a single code point might not be meaningful in itself. Here is an actual value from the API response that also fails in IEx:

```
iex(1)> "\ud83d\udccd FooBar"
** (SyntaxError) iex:1:2: invalid or reserved Unicode code point \u{d83d}. Syntax error after: \u

```

The response is a Javascript block. The above should be this emoji:  
[https://charbase.com/1f4cd-unicode-round-pushpin](https://charbase.com/1f4cd-unicode-round-pushpin)

I guess Javascript uses a different codepoint set, than Elixir? How do I tell Elixir or Poison to use Javascript’s?

---

<div class="post-metadata">

**Author:** ![03juan](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/03juan/32/23111_2.png) [@03juan](https://forum.elixirforum.com/u/03juan)\
**Post date:** [August 11, 2021, 6:07pm UTC](https://forum.elixirforum.com/t/syntax-error-invalid-or-reserved-unicode-code-point/41604/5 "2021-08-11T18:07:55Z")

</div>

That is a good question. Looks like [String.chunk/2](https://hexdocs.pm/elixir/String.html#chunk/2) is a good place to start to weed out invalid UTF8 characters. This section of the docs gives some more information. [String — Elixir v1.20.2](https://hexdocs.pm/elixir/String.html#module-code-points-and-grapheme-cluster)

As for emoji code points, there might be a library that can help with that but unfortunately this is as far as my knowledge extends on the subject.

I don’t think emoji fall into the official unicode spec, that may be part of the issue.

edit:

> [@Doerge](#):
>
> Is there an allow-list of valid ones builtin somewhere?

If you don’t want to keep the emoji at all then a simple `Enum.filter(raw, &String.valid?/1)` should do the trick?

---

<div class="post-metadata">

**Author:** ![al2o3cr](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/al2o3cr/32/3457_2.png) [@al2o3cr](https://forum.elixirforum.com/u/al2o3cr)\
**Post date:** [August 11, 2021, 6:10pm UTC](https://forum.elixirforum.com/t/syntax-error-invalid-or-reserved-unicode-code-point/41604/6 "2021-08-11T18:10:49Z")

</div>

`\ud83d` is a Unicode “surrogate pair” character; it’s normally followed by another character of the `\uDxxx` variety to represent a character above `\uFFFF` in UTF-16 systems. If there’s one in the text by itself (a “lone surrogate”) that string can’t be represented in UTF-8 _at all_ and is invalid.

`Jason` can parse these (when they are paired):

```elixir
iex(livebook_ky4n2p4p@Matts-MacBook-Pro-2)22> {:ok, decoded} = Jason.decode("{\"foo\":\"\\uD83D\\uDE04\"}")
{:ok, %{"foo" => "😄"}}

```

But oddly, doesn’t _produce_ them:

```elixir
iex(livebook_ky4n2p4p@Matts-MacBook-Pro-2)23> {:ok, encoded} = Jason.encode(decoded)
{:ok, "{\"foo\":\"😄\"}"}
iex(livebook_ky4n2p4p@Matts-MacBook-Pro-2)24> String.to_charlist(encoded)
[123, 34, 102, 111, 111, 34, 58, 34, 128516, 34, 125]

```

---

<div class="post-metadata">

**Author:** ![Doerge](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/doerge/32/19632_2.png) [@Doerge](https://forum.elixirforum.com/u/Doerge)\
**Post date:** [August 11, 2021, 6:18pm UTC](https://forum.elixirforum.com/t/syntax-error-invalid-or-reserved-unicode-code-point/41604/7 "2021-08-11T18:18:45Z")

</div>

Thanks for explaining, and pointing me to Jason! It works perfectly!

---

<div class="post-metadata">

**Author:** ![kip](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kip/32/1440_2.png) [@kip](https://forum.elixirforum.com/u/kip)\
**Post date:** [August 11, 2021, 10:24pm UTC](https://forum.elixirforum.com/t/syntax-error-invalid-or-reserved-unicode-code-point/41604/8 "2021-08-11T22:24:49Z")

</div>

> [@03juan](#):
>
> I don’t think emoji fall into the official unicode spec, that may be part of the issue.

Emoji are definitely part of the [Unicode specification](http://unicode.org/reports/tr51).

---

<div class="post-metadata">

**Author:** ![03juan](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/03juan/32/23111_2.png) [@03juan](https://forum.elixirforum.com/u/03juan)\
**Post date:** [August 12, 2021, 7:20am UTC](https://forum.elixirforum.com/t/syntax-error-invalid-or-reserved-unicode-code-point/41604/9 "2021-08-12T07:20:40Z")

</div>

Yes that was an ignorant statement from my part from lack of research. Thanks for the correction.

---

<div class="post-metadata">

**Author:** ![DidactMacros](https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/d/41988e/32.png) [@DidactMacros](https://forum.elixirforum.com/u/DidactMacros)\
**Post date:** [March 5, 2024, 10:45am UTC](https://forum.elixirforum.com/t/syntax-error-invalid-or-reserved-unicode-code-point/41604/10 "2024-03-05T10:45:44Z")

</div>

Old thread, but I encountered a somewhat adjacent issue with my file names. Character encodings over 1418 aren’t represented as their string forms, and you get a boxed question mark instead or a boxed question mark with a space, and character encodings over 553295 (553\_295) cause a codepoint error with string conversion attempts.

(Maybe this is due to something that needs to be installed/configured on my computer?)

I had to filter out character lists with such encodings (ones over 553295) when passing :file.dir\_list output elements to File.dir? because this function first converts character lists to strings, however lower encodings were still usable, even though they didn’t have distinct string representation.

---

<div class="post-metadata">

**Author:** ![al2o3cr](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/al2o3cr/32/3457_2.png) [@al2o3cr](https://forum.elixirforum.com/u/al2o3cr)\
**Post date:** [March 5, 2024, 5:42pm UTC](https://forum.elixirforum.com/t/syntax-error-invalid-or-reserved-unicode-code-point/41604/11 "2024-03-05T17:42:55Z")

</div>

> [@DidactMacros](#):
>
> Character encodings over 1418 aren’t represented as their string forms, and you get a boxed question mark instead or a boxed question mark with a space

It’s difficult to narrow down what could be causing this because there are a lot of suspects:

- the font your terminal program uses
- the terminal program
- the compiler options used to build the BEAM
- the shell’ls character set settings
- the operating system’s character set settings
- the operating system’s implementation of the filesystem APIs

---

<div class="post-metadata">

**Author:** ![DidactMacros](https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/d/41988e/32.png) [@DidactMacros](https://forum.elixirforum.com/u/DidactMacros)\
**Post date:** [March 5, 2024, 6:27pm UTC](https://forum.elixirforum.com/t/syntax-error-invalid-or-reserved-unicode-code-point/41604/12 "2024-03-05T18:27:17Z")

</div>

Ah I thought this may just be a general occurrence.

I’ll check if the same happens on my Linux system.

edit: The boxed question mark occurs only for characters over 43008 while the codepoint error still has the same trigger point.

I don’t think it’s the operating system’s character set since I can see emojis in my file names. It’s likely the terminal.