I think this is the root point of misalignment between us. I believe that in practice this particular explicitness only makes development more expensive, and makes code more complex and harder to support without real benefits. But there cannot be a theoretical proof for this.
Let’s wait for more feedback then!
Meanwhile I’m curious, what would be you idea of implementing dynamic default with Ecto? Patching all the functions that uses schema or can have this schema module as an argument? Or do I miss something more clever?
Find all the places, which need that behaviour and make them call a function setting those dynamic defaults. Anything short of that is “hope for the best / that nothing broke”. Introducing that new behaviour is no different to any other change you add to your codebase. That you consider those dynamic field values “defaults” imo doesn’t change that.
So, if we’re talking about a huge codebase, it means:
do it in dozens or potentially hundreds of places (with Ecto virtual fields we need also patch each place where we read from the storage).
do not forget to include this functionality in any new functions that create such struct (isn’t it error prone in huge projects?)
Does still look acceptable? =)
I do not agree that “introducing dynamic default” is like any other change. There is a concept of a default value. When you think about it from general design perspective “is it dynamic or not?” is an implementation detail. What I see as a problem is that static and dynamic ones has very different ways of being implemented.
Looks like you ok with the idea that “dynamic defaults should not be part of a struct definition”. It makes dynamic default an opt-in behavior, not a guaranteed one. So, we have the following inconsistency:
Static defaults:
part of a struct definition
guaranteed to be applied
introducing the first static default does not require refactoring
Dynamic defaults:
separate from a struct definition
has no application guarantees (it’s up to developer to find all the places where to inject them)
introducing the first dynamic default requires refactoring and searching for workarounds when 3rd party libraries involved.
This inconsistency is more destructive from my point of view than accepting %A{} != %A{}. If you can hint a way that will resolve this inconsistency without violating %A{} == %A{} that would be awesome.
I personally like Elixir for easiness of refactoring of business code. So argument “yes, refactoring is expensive just accept it” goes against things I love Elixir for. =)
I don’t think we’ll find any alignment here. I still stand by the fact that structs should behave like native datatypes, while you seem to want to make them even more a special case to what they already are and partially out of necessity. I see %Struct{} as a notation to define data and data doesn’t execute code. Yes one can change the definition of the struct, but generally people try to avoid having the before and after of those definitions meet.
Execution is btw another footgun to your proposal. %Struct{} executing code means different runtime behaviour. It’s no longer “O(1)”, but could start an http request, fail somewhere on the path and raise some arbitrary exception crashing a whole process – just from %Struct{}.
What about having a slightly different data-type then: dynamically created struct. It will work just like normal structs, support struct!(...), protocols, etc. The only difference is that you define it using defdynstruct and cannot create them as literal, you must use something like %MyStruct() and %MyStruct(collectable) to instantiate them?
Also, %MyStruct(...) should work for normal structs also.
(Yes, we already have struct! and can use this instead of %MyStruct(...), but it feels like worse UX to me).
Then:
Static defaults via defstruct:
part of a struct definition
guaranteed to be applied
introducing the first static default does not require refactoring
Dynamic defaults via defdynstruct:
part of a struct definition
guaranteed to be applied (you cannot do %MyStruct{})
introducing the first dynamic default requires refactoring, but more trivial one:
in the struct it’s a mere replacement of defstruct with defdynstruct
in the code you own it’s a mere replacement of %MyStruct{...} to %MyStruct(...)
in libraries: you only need to switch to using struct!(...) in places where you work with user-provided structs and avoid do %unquote(struct_mod){} hacks in macro.
What is the use-case around not having a dynamic values auto-generated just before insertion? I also don’t understand why would you want to have dynamically generated default values populated on read.
It would be interesting to hear motivation beyond the DateTime one. As dates go, I’m having a hard time imaging a case where you would ever want any other time than at insertion. The only thing I can think of is “when did the user start filling out this form,” though I think that could only reliably be tracked on the client.
When a field is a subject for validation. Or part of a condition of a conditional validation. Validation happens before insertion.
When a field is used to calculate another field value before insertion.
When there is no persistence (virtual fields, when schema is used only for validation, like in Oban jobs).
We have a schema that resembles “booking restrictions config”. It has “current allowed dates period” virtual field. By default it should be from beginning of the current year to the end of it. Even if you read a saved one from DB.
Meanwhile, they’re cases when we need to override it. In such cases we override this virtual field.
This virtual field is used in validations. It used during rendering things for JSON responses, etc. It used in many places when the entity is used. It is very convenient to have it as a virtual field. Otherwise we need to add one additional argument/option to too many functions.
Consider that I’ve significantly simplified related business logic.
While adding this additional argument may look a right way to do, on practice it will make implementation uglier and more fragile. It still possible, but I asked myself - why I have to do such ugly things in my favorite language? (When in Rails or Golang it would be a one-liner).
I realized that many times I needed a dynamic default, but each time instead of using this simple concept I was forced to do a workaround. Sometimes spending significant time on a thing that should be simple. But it was… acceptable. Then I found this case where I wasn’t able to fix it even after couple of hours. And I realized that it’s probably easier to create a library that adds dynamic defaults to Elixir. And here we’re. =)
Just thinking out lout here and the whole thing is a matter of year or even two. But could this maybe be solved (at least partially) by the future typed structs? If you would declare your field as being DateTime (for example). You would then be forced everywhere you’re trying to create this struct to provide that value? The ergonomics wouldn’t be ideal, but at least there won’t be any missed case.
Sorry I don’t want to pile too much onto the detractor side but I think this in a nutshell is the problem. Again I would have to see code but I don’t know why something like would be ugly or why it would have to change in so many places no matter how big the project. Unless you are doing something like Repo.get(Struct, 1) everywhere (web layer, api layer) instead of going through a domain function
Hmm. Interesting! It will work for many cases but I’m afraid not for Ecto virtual fields: Ecto creates struct instances inside their helper (MyStruct.__schema__(:loaded)). The helper is generated by schema do ... end macro. It will try to create a record with defaults from field/3 definitions and I haven’t found any documented way to override it.
Might be I need to be more explicit of my goal here: It’s not about finding a way how to reliably refactor in order to introduce a dynamic default.
I want to find a way to avoid refactoring when I introduce a dynamic default to an existing struct in a big (500k+ lines) project. And find a way to define static and dynamic defaults using similar syntax, both in Elixir stdlib and libraries.
That’s a right question. For simplicity let me provide one related example:
We have shared queries - because domains want to preload things defined in other domain. Or sql-join with them. While subquery is defined inside domain A, it becomes part of the query in domain B and now domain B should also take care of initializing virtual field defaults. As result I need to put dynamic defaults in every domain that generates my struct indirectly via query.
And we can easily have dozens of functions that creates your struct directly or indirectly in a single domain of our big monolith.
Could throw together a custom credo check that warns on %Struct{...} when Struct.new/build is detected, perhaps. A community consensus on new vs build could be handy as well.
Yes. For me the split I’m observing comes down to people who feel that code which calls functions should be a function call, and this proposal where doing literal syntax is actually sometimes a way to call functions dynamically. You can tell from this framing which side I’m on I suppose lol.
But more seriously some of the objections to using new such as the time required to convert code base don’t hold water. Sure we are talking about dynamic data now but what about invariants? Should struct literal syntax also run validators on the values? The answer is: no, we already crossed this bridge with changesets. (And no the type system won’t solve this, there are invariants not amenable to types).
If we had a convention around new or build then this would be a non issue. You wouldn’t have 100 spots in your code to refactor you’d have 100 function calls and you make one change to the function. This is already what you have when using changesets.
I agree with everyone who has said that “it takes to long to refactor” is not a very compelling argument. It would be nice if there was a more “blessed” constructor pattern though. It would be great to know that if I am initializing a struct I can/should always call `new` or whatever. Even better if it came with a way to prevent “private” fields from being set somehow…
These are exactly my thoughts. This entire thread wouldn’t be needed if constructors were the default way to create structs.
I’m convinced that it’s perfectly fine to have constructor functions that only pass the data into a struct. Some will become more complex later, others will stay trivial, but the changes will never spread outside of the module.
FWIW, the aforementioned estructura provides coerce/3 which calculates (also coerces and validates btw) everything because it goes through Access which is amending the calculated fields. Example.
There are none, if we use dependent types. Sorry for nitpicking, but inventing new wheels for a wheelchair, like ad-hoc validations. would never cover SMT-solving theories. Ad-hoc validations are extremely hard to validate in the wild, unlike a proof in languages like Idris. That’s basically why I started Cure effort.
I feel you. Such refactoring wasn’t painful for me until I started working on 2M+ LOC Elixir codebase. It’s one of the biggest (if not the biggest) production monolith written in Elixir. In any other codebase I touched, this problem was in “meh, annoying!” category, not “it’s too long to refactor”.
100% percent agree! That’s the problem I’m trying to actually solve! Bear with me:
The reality I see is that %Schema{} syntax is used to “create struct with defaults”. My strongest proof that even Ecto does it and proposes it. If Ecto were enforcing using Schema.build(...) as the idiomatic way to create a schema instance - it would have dynamic defaults already.
I believe that allowing static defaults in struct definition was a design mistake (that was done for Erlang records, structs merely repeat it). If setting static default were require making a build/new function - Ecto would have build/new functions enforced.
It made me think about making %Schema{} a full-featured constructor. But it violates %A{} == %A{}, so, not a good way.
My latest proposal is not about %MakeMeDynamic{} anymore.
Not exactly. What I’m fighting with right now is the idea that static and dynamic defaults should be implemented in different ways. Ideally, all my problems would be solved by:
Remove ability to set static defaults. Any kind of defaults should require implementing new/build function.
But it’s too late for this proposal.
Should I make a proposal to Ecto then? To make Ecto create build function when defining schema and suggest people to use it over %Schema{} syntax if they need any kind of defaults? It’s a breaking change. How would you estimate chances that people agree on this? (Considering that in the most cases lack of dynamic defaults are not super painful)
Not every Elixir developer uses Ecto. And the design mistake of not enforcing usage of build/new when a struct can have dynamic defaults has a good chance to be repeated in other libraries.
That’s why I want to solve this problem on the language syntax level. It’s more bulletproof solution than relying on traditions and conventions.
You sparked a really good discussion. Thanks for that.
The fact that default: DateTime.utc_today() is resolved at compile time actually also a surprise of a similar level.
True, but that is really part of the language. It’s like the @tag that also gets evaluated at compile time.
You are right that you could learn this too. The difference is that, once you have learned ot, you don’t get surprises, whereas for your structs I would need to check the focumentation every time. This is the kind of surprise I’m talking about.
As a language you need to be conservative when adding new features to the language (consider the Change Event Horizon). Hence we ahould ask ourselves how common your problem is, and whether it could be solved on a different level before solving it on the language. Here some options I could see:
Solve the problem on a database level, e.g. at database insertion/retrieval time (and I think that would be cleanest, because you solve the problem locally)
Add a feature to Ecto that allows the initialization of default/calculated values. Maybe an _init: &myinit/1 field in your struct?
Even though I might be against your proposal, I very much appreciate you taking the time to raise it. As more use cases surface, the more likely it is that a solution might happen on a language level.
Side note:
If even creators of Ecto (and ExMachina) fell into the trap
It’s up to them to decide that. As demonstrated above there are plenty of ways on how you can extend keep things extendable. And at the end it’s always a) a balancing act and b) a learning process
