Hey @dimitarvp,
In the article (and professionnally) I do not use of any code comments, I use doctests and moduledoc.
I use doctests to document a case what I do not want to handle right now, the doctest itself will fail loudly as soon as the edge case is fixed (which is the behaviour that I want).
So I guess I could return the question back to you, what do you suggest to use when we do not want to handle an edge case? How can we communicate these decisions to LLMs beyond the sessions during which we’ve made these decisions?
- Writing a code comment? As you pointed out, those will become stale very quickly and fail silently
- Updating the code to recognise the edge case and maybe return an error? Now you are diving into implementation territory, you need to make decisions (should I error, return nil, raise, etc ..) it’s another opportunity for the LLM to get something wrong; or maybe you don’t even want to update the function’s public API for fear of having to do many changes across the codebase …
- A separate product specification / document that must be updated everytime the underlying code changes?
- something else?
To me doctests seems the best compromise for that (they allow me to use prose to explains stuff AND I can encode assertions that fail loudly when they aren’t true anymore); though I’m very happy to hear from how other teams are handling those cases!
I use moduledoc to document the high level product and business processes that are less likely to go stale any time soon. Look at how the moduledoc does not contain:
- any API
- any structs
- any function calls
- etc ..
Of course you could have a separate .md file, a separate skill, but to me, putting those high level product processes near where they matter most seems also the best compromise.
I have to be honest too, I do not have empirical evidence or benchmark that this drastically improves LLMs ability to recall the correct rules when doing something. I only have a preference, an educated guess / gut feeling.
