# Common data transform patterns - please add yours! **URL:** \ **Category:** Guides/Tuts/Tips/Info\ **Tags:** lists, maps\ **Created:** [December 1, 2018, 5:25pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381 "2018-12-01T17:25:17Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![9mm](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/9mm/32/7475_2.png) [@9mm](https://forum.elixirforum.com/u/9mm)\ **Post date:** [December 1, 2018, 5:25pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/1 "2018-12-01T17:25:17Z")
So I’m really loving elixir. BY FAR the most excruciating piece of learning a functional language for me is having to “transform” all my input data so that it includes ALL desired output data. In a mutable language like ruby you can have loops and iterators that just update variables (maps, counters, etc) outside the scope. This makes it super easy to wrangle complex data structures… but in Elixir it’s taking me quite awhile to deal with all this. I wanted to make a little “database” in this thread that I can constantly come back to and reference of all the best ways you might do simple/medium/advanced data transformations. The advanced ones are too many to include in this thread but I feel like there are a lot of simple/medium ones that you can count on 2 hands that you use over and over and over. I will keep adding to this thread but here are a few to start off… **A. Transform all values inside a map** input: `%{a: "hello", b: "world"}` output: `%{a: "HELLO", b: "WORLD"}` Answers: ``` x |> Enum.reduce(%{}, fn {k, v}, acc -> Map.put(acc, k, String.upcase(v)) end) for({k, v} <- x, into: %{}, do: {k, String.upcase(v)}) :maps.map(fn (_k,v) -> String.upcase(v) end, i) ``` **A2. Transform value inside a map for a specific key** input: `%{a: "hello", b: "world"}` output: `%{a: "hello", b: "WORLD"}` **A3. Transform value for specific key inside a nested map** input: `%{person: %{first: "Jose", last: "valim"}, age: 100}` input: `%{person: %{first: "JOSE", last: "VALIM"}, age: 100}` Answers: ``` x |> update_in([:person, :last], &String.upcase(&1)) ``` **B. Transform all values inside a list of map** input: `[%{a: "hello", b: "world"}, %{a: "fizz", b: "buzz}]` output: `[%{a: "HELLO", b: "WORLD"}, %{a: "WORLD", b: "BUZZ}]` **B2. Transform one value inside a list of map for a specific key** input: `[%{a: "hello", b: "world"}, %{a: "fizz", b: "buzz}]` output: `[%{a: "hello", b: "WORLD"}, %{a: "fizz", b: "BUZZ}]` Answers: ``` x |> Enum.map(fn m -> Map.update(m, :b, nil, &String.upcase(&1)) end) ``` **C. Push to a list in a map** input: `%{list: []}` output: `%{list: [1]}` Answers: ``` x |> update_in([:list], &[1 | &1]) ``` I will add more as I go, feel free to add your own ---
**Author:** ![kokolegorille](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kokolegorille/32/4784_2.png) [@kokolegorille](https://forum.elixirforum.com/u/kokolegorille)\ **Post date:** [December 1, 2018, 5:35pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/2 "2018-12-01T17:35:45Z")
My small contrib for A 🙂 ```elixir iex> i = %{a: "hello", b: "world"} iex> o = %{i | a: "HELLO", b: "WORLD"} # or iex> o = i |> Enum.reduce(%{}, fn {k, v}, acc -> Map.put(acc, k, String.upcase(v)) end) ``` ---
**Author:** ![9mm](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/9mm/32/7475_2.png) [@9mm](https://forum.elixirforum.com/u/9mm)\ **Post date:** [December 1, 2018, 5:39pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/3 "2018-12-01T17:39:59Z")
Amazing, I already like that more than `Enum.map` + `Enum.into` blah blah blah. thanks! ---
**Author:** ![peerreynders](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/peerreynders/32/5826_2.png) [@peerreynders](https://forum.elixirforum.com/u/peerreynders)\ **Post date:** [December 1, 2018, 5:48pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/4 "2018-12-01T17:48:18Z")
```elixir input = %{list: []} output = update_in(input, [:list], &[1 | &1]) IO.inspect(output) ``` ```plaintext iex(1)> input = %{list: []} %{list: []} iex(2)> output = update_in(input, [:list], &[1 | &1]) %{list: [1]} iex(3)> IO.inspect(output) %{list: [1]} %{list: [1]} iex(4)> ``` ---
**Author:** ![peerreynders](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/peerreynders/32/5826_2.png) [@peerreynders](https://forum.elixirforum.com/u/peerreynders)\ **Post date:** [December 1, 2018, 5:55pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/5 "2018-12-01T17:55:41Z")
```elixir input = %{person: %{first: "Jose", last: "valim"}, age: 100} output = update_in(input, [:person, :last], &String.upcase(&1)) IO.inspect(output) ``` ```plaintext iex(1)> input = %{person: %{first: "Jose", last: "valim"}, age: 100} %{age: 100, person: %{first: "Jose", last: "valim"}} iex(2)> output = update_in(input, [:person, :last], &String.upcase(&1)) %{age: 100, person: %{first: "Jose", last: "VALIM"}} iex(3)> IO.inspect(output) %{age: 100, person: %{first: "Jose", last: "VALIM"}} %{age: 100, person: %{first: "Jose", last: "VALIM"}} iex(4)> ``` ---
**Author:** ![peerreynders](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/peerreynders/32/5826_2.png) [@peerreynders](https://forum.elixirforum.com/u/peerreynders)\ **Post date:** [December 1, 2018, 6:05pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/6 "2018-12-01T18:05:27Z")
```elixir input = %{a: "hello", b: "world"} output = Map.update(input, :b, "THERE", &String.upcase(&1)) IO.inspect(output) ``` ```plaintext iex(1)> input = %{a: "hello", b: "world"} %{a: "hello", b: "world"} iex(2)> output = Map.update(input, :b, "THERE", &String.upcase(&1)) %{a: "hello", b: "WORLD"} iex(3)> IO.inspect(output) %{a: "hello", b: "WORLD"} %{a: "hello", b: "WORLD"} iex(4)> ``` ---
**Author:** ![kokolegorille](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kokolegorille/32/4784_2.png) [@kokolegorille](https://forum.elixirforum.com/u/kokolegorille)\ **Post date:** [December 1, 2018, 6:10pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/7 "2018-12-01T18:10:15Z")
For B2… ```elixir iex> input = [%{a: "hello", b: "world"}, %{a: "fizz", b: "buzz"}] iex> input |> Enum.map(fn m -> Map.update(m, :b, nil, &String.upcase(&1)) end) ``` ---
**Author:** ![peerreynders](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/peerreynders/32/5826_2.png) [@peerreynders](https://forum.elixirforum.com/u/peerreynders)\ **Post date:** [December 1, 2018, 6:12pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/8 "2018-12-01T18:12:52Z")
```elixir input = %{a: "hello", b: "world"} output = for({k, v} <- input, do: {k, String.upcase(v)}) |> Map.new() ``` ```plaintext iex(1)> input = %{a: "hello", b: "world"} %{a: "hello", b: "world"} iex(2)> output = for({k, v} <- input, do: {k, String.upcase(v)}) |> Map.new() %{a: "HELLO", b: "WORLD"} iex(3)> ``` ---
**Author:** ![kokolegorille](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/kokolegorille/32/4784_2.png) [@kokolegorille](https://forum.elixirforum.com/u/kokolegorille)\ **Post date:** [December 1, 2018, 6:15pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/9 "2018-12-01T18:15:58Z")
> [@peerreynders](#): > > output = for({k, v} \<- input, do: {k, String.upcase(v)}) |\> Map.new() You might also use the into parameters… ```elixir iex> output = for({k, v} <- input, into: %{}, do: {k, String.upcase(v)}) ``` ---
**Author:** ![peerreynders](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/peerreynders/32/5826_2.png) [@peerreynders](https://forum.elixirforum.com/u/peerreynders)\ **Post date:** [December 1, 2018, 6:32pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/10 "2018-12-01T18:32:01Z")
```elixir input = [%{a: "hello", b: "world"}, %{a: "fizz", b: "buzz"}] output = for(m <- input, do: for({k, v} <- m, into: %{}, do: {k, String.upcase(v)})) ``` ```plaintext iex(1)> input = [%{a: "hello", b: "world"}, %{a: "fizz", b: "buzz"}] [%{a: "hello", b: "world"}, %{a: "fizz", b: "buzz"}] iex(2)> output = for(m <- input, do: for({k, v} <- m, into: %{}, do: {k, String.upcase(v)})) [%{a: "HELLO", b: "WORLD"}, %{a: "FIZZ", b: "BUZZ"}] iex(3)> ``` `into` is OK as long as you stay with [`Keyword`](https://hexdocs.pm/elixir/Keyword.html) while you are still transforming the data - i.e. not constantly creating Maps only to turn them back to keywords again. ```elixir input = [%{a: "hello", b: "world"}, %{a: "fizz", b: "buzz"}] output = Enum.map(input, &(Enum.map(&1, fn {k, v} -> {k, String.upcase(v)} end) |> Map.new())) ``` ```plaintext iex(1)> input = [%{a: "hello", b: "world"}, %{a: "fizz", b: "buzz"}] [%{a: "hello", b: "world"}, %{a: "fizz", b: "buzz"}] iex(2)> output = Enum.map(input, &(Enum.map(&1, fn {k, v} -> {k, String.upcase(v)} end) |> Map.new())) [%{a: "HELLO", b: "WORLD"}, %{a: "FIZZ", b: "BUZZ"}] iex(3)> ``` ---
**Author:** ![peerreynders](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/peerreynders/32/5826_2.png) [@peerreynders](https://forum.elixirforum.com/u/peerreynders)\ **Post date:** [December 1, 2018, 6:46pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/11 "2018-12-01T18:46:21Z")
```elixir input = [%{a: "hello", b: "world"}, %{a: "fizz", b: "buzz"}] output = for(m <- input, do: update_in(m, [:b], &String.upcase(&1))) ``` ```plaintext iex(1)> input = [%{a: "hello", b: "world"}, %{a: "fizz", b: "buzz"}] [%{a: "hello", b: "world"}, %{a: "fizz", b: "buzz"}] iex(2)> output = for(m <- input, do: update_in(m, [:b], &String.upcase(&1))) [%{a: "hello", b: "WORLD"}, %{a: "fizz", b: "BUZZ"}] iex(3)> ``` ```elixir input = [%{a: "hello", b: "world"}, %{a: "fizz", b: "buzz"}] output = input |> Enum.map(fn m -> update_in(m, [:b], &String.upcase(&1)) end) ``` ```plaintext iex(1)> input = [%{a: "hello", b: "world"}, %{a: "fizz", b: "buzz"}] [%{a: "hello", b: "world"}, %{a: "fizz", b: "buzz"}] iex(2)> output = input |> Enum.map(fn m -> update_in(m, [:b], &String.upcase(&1)) end) [%{a: "hello", b: "WORLD"}, %{a: "fizz", b: "BUZZ"}] iex(3)> ``` ---
**Author:** ![rvirding](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/rvirding/32/1409_2.png) [@rvirding](https://forum.elixirforum.com/u/rvirding)\ **Post date:** [December 1, 2018, 7:00pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/12 "2018-12-01T19:00:26Z")
Another solution for **A** is ```elixir %{a: "hello", b: "world"} iex(7)> o = :maps.map(fn (_k,v) -> String.upcase(v) end, i) %{a: "HELLO", b: "WORLD"} ``` I don’t know why the `Map` module doesn’t include a `map` function. Yes, you can do it with `Enum.map` but it **always** returns a list and not a result of the same type as the input. Which I feel is a bit off. Another solution would be to use `for`. ---
**Author:** ![peerreynders](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/peerreynders/32/5826_2.png) [@peerreynders](https://forum.elixirforum.com/u/peerreynders)\ **Post date:** [December 1, 2018, 7:55pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/13 "2018-12-01T19:55:22Z")
> [@rvirding](#): > > but it **always** returns a list and not a result of the same type as the input. Just me hypothesizing: - Even [`maps.map/2`](http://erlang.org/doc/man/maps.html#map-2) generates a [list of keywords](https://github.com/erlang/otp/blob/master/lib/stdlib/src/maps.erl#L249-L255) as an intermediate result. - For `Enum` the list is a kind of universal output (iterable) data structure. - Elixir has the pipe (`|>`) operator and `Enum` functions are used in pipelines quite heavily. So once something is a list, it should stay a list throughout the pipeline and not be “rehydrated” until the end of the pipeline ([`Map.new/1`](https://hexdocs.pm/elixir/Map.html#new/1)). - By _not_ offering a `Map.map/2` there is a speed bump to _mindlessly_ putting `Map.map/2` at the beginning (or in the middle) of a pipeline (where `map` is typically used) - i.e. creating a map that is just going to be converted to a list again. ---
**Author:** ![rvirding](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/rvirding/32/1409_2.png) [@rvirding](https://forum.elixirforum.com/u/rvirding)\ **Post date:** [December 1, 2018, 8:28pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/14 "2018-12-01T20:28:33Z")
What I meant was that I think `Enum` should go `type -> type` and not `type -> list`. It feels wrong imao. ---
**Author:** ![LostKobrakai](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lostkobrakai/32/3072_2.png) [@LostKobrakai](https://forum.elixirforum.com/u/LostKobrakai)\ **Post date:** [December 1, 2018, 9:03pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/15 "2018-12-01T21:03:34Z")
The reasoning behind that is explained a bit here: [Collectable — Elixir v1.20.2](https://hexdocs.pm/elixir/Collectable.html#module-why-collectable) ---
**Author:** ![peerreynders](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/peerreynders/32/5826_2.png) [@peerreynders](https://forum.elixirforum.com/u/peerreynders)\ **Post date:** [December 1, 2018, 10:05pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/16 "2018-12-01T22:05:42Z")
> [@rvirding](#): > > `Enum` should go `type -> type` and not `type -> list` . Example: `m |> Enum.map(f1) |> Enum.reduce(init,f2)` If `m` is a Map, it would also be a Map emerging from `map/2` which would then have to be converted to a list again before entering `reduce/3` - which introduces an unnecessary `type->list` conversion. It makes more sense to simply leave it as a list and have [`Collectable`](https://hexdocs.pm/elixir/Collectable.html) reconstitute everything (as desired) at the end of the pipe. ---
**Author:** ![rvirding](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/rvirding/32/1409_2.png) [@rvirding](https://forum.elixirforum.com/u/rvirding)\ **Post date:** [December 2, 2018, 11:44pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/17 "2018-12-02T23:44:32Z")
> [@peerreynders](#): > > Example: > > `m |> Enum.map(f1) |> Enum.reduce(init,f2)` > > If `m` is a Map, it would also be a Map emerging from `map/2` which would then have to be converted to a list again before entering `reduce/3` - which introduces an unnecessary `type->list` conversion. It makes more sense to simply leave it as a list and have [`Collectable`](https://hexdocs.pm/elixir/Collectable.html) reconstitute everything (as desired) at the end of the pipe. `Enum.reduce/3` can already work directly on a map so I don’t see the problem here. I just think it should be enumerable in and not, list out, but enumerable out. It makes it cleaner, imao. ---
**Author:** ![peerreynders](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/peerreynders/32/5826_2.png) [@peerreynders](https://forum.elixirforum.com/u/peerreynders)\ **Post date:** [December 3, 2018, 1:56am UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/18 "2018-12-03T01:56:47Z")
> [@rvirding](#): > > I just think it should be enumerable in and not, list out, but enumerable out. While I don’t have an issue with the implementation compromise for efficiency sake, it’s actually the name `map` that from my perspective fails the “principle of least astonishment”. When I see a `map` function I expect it to produce the same shape of data that it consumed - perhaps with a different contained type. That’s because I encountered `map` in Haskell _consciously_ for the first time and that is how it behaves. So in a way `Enum.map/2` sets expectations that it doesn’t deliver on - `Enum.map_to_list/2` would be more accurate. Once I got over my initial “astonishment” I simply reclassified the `Enum` module as a set of functions that are capable of consuming a range of collections but (mostly) produce output collections as lists (i.e. I took the `_to_list` suffix as implied). ---
**Author:** ![LostKobrakai](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lostkobrakai/32/3072_2.png) [@LostKobrakai](https://forum.elixirforum.com/u/LostKobrakai)\ **Post date:** [December 3, 2018, 5:18pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/19 "2018-12-03T17:18:44Z")
There are cases, where the input type == output type doesn’t work though: ```elixir range = Date.range(~D[1999-01-01], ~D[2000-01-01]) result = Enum.map(range, fn date -> date.day end) # or set = MapSet.new([1, 2, 3, 4, 5, 6]) result = Enum.map(set, fn number -> rem(number, 2) end) ``` ---
**Author:** ![peerreynders](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/peerreynders/32/5826_2.png) [@peerreynders](https://forum.elixirforum.com/u/peerreynders)\ **Post date:** [December 3, 2018, 6:11pm UTC](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381/20 "2018-12-03T18:11:00Z")
> [@LostKobrakai](#): > > where the input type == output type doesn’t work though Which is why in Haskell the `fmap` function is tied to the `type` or more accurately `typeclass` - so neither `Range` nor `MapSet` would be a member of the `Functor typeclass`. The original complaint was that there should be a `Map.map/2` that produces a `Map` (while `Enum.map_to_list` can do whatever it needs to do). [Next page](https://forum.elixirforum.com/t/common-data-transform-patterns-please-add-yours/18381.md?page=2)