Just putting this out there, but… What I REALLY want is something similar to Ecto for JSON/XML…
Meaning I find myself needing to consume some JSON structure. eg:
{
....,
"firewall": {
"dnat": {
"nat-in": [
{
"dest": "loc:192.168.111.4",
"dport": "7",
"proto": "tcp",
"source": "net"
},
{
"dest": "loc:192.168.111.4:80",
"dport": "8080",
"origdest": "&ppp0",
"proto": "tcp",
"source": "net"
}
]
}
},
....,
}
So this is a map of maps of maps, which contains an array of maps.
Now this snippet is part of a much larger JSON structure which has config for other stuff, ie there are other keys at the top level with their own trees under.
Now I want to parse chunks of this into Elixir structures, check that it’s valid before starting, present those to the user as some kind of phoenix/Liveview form, accept back the updated params and validate them (so that I can do instant errors on screen). Finally I want to be able to diff what changed from the original and re-apply it to the current JSON structure
Whilst I’ve been a little over specific on some of my own use case, I don’t think this is so different from a use case you likely have in mind: consume some API end point, present the details to the user, allow them to edit stuff, send the changes back to the API end point?
Things which might not be obvious from the above:
- I need to validate fields in combination with each other, so certain params may only be valid if the :action is something specific
- It’s REALLY boring mapping a map of maps of lists of maps into Ecto format… Ecto can only cope with representing database tables, so your map of maps ends up needing to become a list structure where you copy the keys in and out (think how you would represent it in an SQL database). It would be SOOO much easier if Ecto could understand something like a map structure in it’s “has_many” fields (yes you can do custom data types like {:map, string}, but then you lose the ability to use schemas and changesets on those fields)
- I need to also validate the keys of the maps. They need to be sanitised and controlled for length, etc (as they may map to UI elements, etc)
- I need some level of round trip ability
- I need to “diff” the changes (so I can apply only the changes back upstream - I want to be granular if simultaneous changes were made to separate parts of the document. However, ideally I want to be able to re-run my “is it valid” after re-apply those changes as two edits might individually not clash, but there might be dependencies between the key values, eg we might have a section for the IP range of the local network and another section for the DHCP parameters, but we have a schema validation between the two as we enforce that one is within the same range as the other.
I solve this at the moment using ecto changesets. The shape of a changeset can cross chunks of the whole json document if needed to enforce cross schema changes. ie a changeset “plucks out” a bunch of fields from the JSON input, and kind of flattens them into the structures allowed within ecto (lists). Then we can run our nested validations, etc. Then unfortunately this needs another function to reverse this process as it’s not necessarily purely mechanical to reverse the original extraction. It’s also painful to represent maps of maps as these need flattening into lists with an id column to represent the map key names (and this reversing later)
What I desire is something like a JSON parser, coupled to a generic structure validator. Which in turn can be used in phoenix forms with functioning error handling (the phoenix error function is something you define, so it can work with any library which produces a validation output including some per field error term)
Does this sound like a direction you are heading in?
