Babel - Data transformations made easy

[Eiji]

@Eiji

It would be perfect if you could give us also some example data and expected output

@wolf4earth

As I mentioned before, I’m working on a LiveBook example.

This one was only for the example you shared as you said it’s more complex one it’s used in production, so I have tried to re-produce it.

@wolf4earth

Doing these non-trivial transformations with standard library mechanisms is feasible but if you don’t want to raise and instead collect errors and be able to explain them, it becomes very complex - and also very hard to read - very quickly.

I didn’t have a problem with that at all, but who am I to speak about it when Gentoo was my first Linux distribution.

Perhaps you mean cases like Map.fetch/2 returning :error atom? It could be solved with a simple wrapper function.

def map_fetch(map, key) do
  case Map.fetch(map, key) do
    {:ok, result} -> {:ok, result}
    :error -> {:error, %Wrapper.MapKeyError{key: key, map: map}}
  end
end

All Elixir core features including debugging is more than enough for me and I rarely need to write any wrapper. As said if Elixir core is not enough at most I have used ecto.

@wolf4earth

Babel does not raise unless you call apply!/2 (which calls apply/2 under the hood and raises in case it returns an error). At no point during the transformation exceptions are involved. It’s all results that get accumulated.

Ignore that, I got something wrong when quickly browsing documentation.

@wolf4earth

Your example is actually a good one to compare against because it’s largely optimized for the happy path. Defaulting to nil values is not something that flies in production.

That’s a good point … for a Rust developers.

I’m writing pipelines in pure Elixir supporting not single, but all “happy paths” and either “let it default” to nil in case some part of data is expected to be nullable or let it fail when it’s not supported. Especially the “let it fail” is Erlang and Elixir concept, so speaking generally about “happy paths” have not sense in general context in such languages. Therefore at the very end the code is very specific to your needs. I believe that your package may therefore be or not be used even in very similar cases. Just speaking it’s a good for “Rust” was 50-50 for fun.

@wolf4earth

A comparable solution would check at every step if an error occurs, collect it, and return the accumulated error. It would be able to explain why that step failed, what the step was, and what the input data was.

Sounds like latest Elixir improvements like dbg macro, improved error messages and using part of code in error output. I guess that I personally would be more interested in such package before mentioned improvements, but at the end I would stop use said package after that improvements would be added.

Again who am I to speak about that when in many answers I gave few possible solutions where the biggest one is based only on the pattern-matching. I have no idea, but for some reason I like how things works now.

@wolf4earth

I’d like to refer to the initial error handling example to point out how much information Babel provides in error cases.

Oh, at first I did not expanded it, so I did not saw it. Hmm …

Too long! Shorten it into 5 words!

More seriously … it’s really long and maybe that’s because it somehow reminds me a terrible long stacktrace in JVM-based languages. I’m exhausted by just seeing and XX more at the bottom of already too long stacktrace. Again I like how things works now, so I’m not sure if I can see a real-world usage in my personal case.

@wolf4earth

Ecto is excellent when you have control over the shape of the incoming data.

Why you mention control? You have only control if your app is generating the data, right? As said for all those years I have used pure Elixir and at most ecto to cover almost all if not all cases of casting a data from for example JSON API response.

@wolf4earth

If you don’t, you will have to write a non-trivial amount of code to transform the data into a shape ecto understands.

What? You are developing the “shape” as same as you are creating structs. Regardless if data have proper format or not you simply pass "audioCollection" value to changeset function and in worst case you use some mapper like the default one supported by ecto i.e. @field_source_mapper mentioned previously.

@wolf4earth

But Ecto and Babel do not cover the same use-cases.

Yeah, I guess that I did not get non-ecto use cases.

I can see that posting:

Data transformations made easy.

without explaining what exactly use-cases you mean is really confusing. That makes me only more interested to hear “your story”.

Maybe answering such questions could help you describe your point of view more easily:

1. Could you please describe uncommon data sources from your use cases?
2. Could you please describe by example where casting data by ecto starts causing trouble?
3. Why things like @field_source_mapper are not enough for a data with different structure?

---

Edit: When writing all of this I got some idea. If it’s not “something like” JSON API that have a well known structure then the source could have a dynamic structure. Ok, but how to parse a dynamic structure? How about creating a complex html form for dynamically querying data of unknown for the app structure (but known by user)? That’s interesting, but how such data would be used? How about using dynamic rules to fetch dynamic data in order to present it in a predictable format like a graph? Am I going into the right direction or by any change I have messed it up completely? Maybe I think about it too much and there is much simpler use case …