Getting back to your original question about how well LLMs do with Elixir, what follows might be a useful example. In this case I decided to try something I said I never do: use the LLM as a brainstorming aid. I’m doing some broad refactoring across my codebase and some questions (admittedly of a bike-shedding quality) are vexing me a bit. It’s a short interaction, but I think it can give some hints about how well these tools work with Elixir. I’m putting these in screenshot style because it’s a bit easier to see who’s talking and some graphical elements which show the UX for the interaction. (My apologies if this looks like crap on your side, devices, etc.)
My prompts are in the white areas immediately after the “First/Second Interaction” headings. The LLM responses are in gray-backed areas when using a speaking voice and in white-backed areas when showing code examples.
First Interaction:
Second Interaction (referenced code: mscmp_syst_error.ex):
From an a basic Elixir language standpoint, it seems competent. I don’t immediately see any basic issues like syntax issues. it handles modules, exceptions, typespecs, typical representation of functions with their arity (i.e. get_root_cause/1) and other constructs/conventions correctly. The code I gave it for context appeared to produce relevant results for the subsequent responses.
I think it missed the more conceptual/architectural level of interaction I was looking for; that could well be an issue with my prompting though I do think it would have gone to implementation details quickly either way. Also I think there are a couple stray things in its suggestions… for example the MscmpSystError.ErrorCodes module doesn’t appear to really have any point the way its expressed. I think defining those recommended lists as types would be better and the one place such a MscmpSystError.ErrorCodes module could pay off, in the suggested “new” functions as validation of the code values on new exception creation, isn’t part of the recommended solution. Finally, I have to admit seeing the closing paragraph made me smile a bit.
So I’ll reiterate, it has this junior quality about it, but it’s not giving me wildly stupid answers either.
