Plugin for Claude Code - specialist agents and an enforced Elixir/Phoenix development workflow

News Announcing
, , ,
1

Hello,

I’m sharing my plugin here in forum after some time so it has time to mature and proof yourself.

I use Claude Code daily on a production Phoenix app (319K lines of Elixir). Claude is good, but it doesn’t know Elixir the way it needs to. So I built a plugin that enforces the rules it keeps breaking. This post is less about the feature list and more about the method behind it, because most of what I built first turned out to be dead weight, and I only know that because I measured it, across 1,800+ of my own sessions.

Why I built it

My main project is a production Phoenix app, some stats:

  • ~ 319K lines of Elixir in lib,
  • ~ 251K lines of tests,
  • 77 contexts,
  • 44 LiveViews,
  • ~140 LiveComponents,
  • 938 migrations.

At that scale you hit the edges of generic AI tooling quickly, token usage goes to sky, subscription to hell.

Claude will use :float for a money field. It will skip authorization in handle_event because mount already checked. It will query the database in mount and not care that mount runs twice. Not always. Sometimes the code is very good. That inconsistency is the actual problem, and I can’t fix the model.

What I can fix is everything around the model. Rules that stop known mistakes before they ship. Compile and test checks after every change. Review that doesn’t depend on my energy level at 11pm.

Jose Valim wrote on the Tidewave blog that the future of coding agents is vertical integration. That’s the bet here: a general agent plus deep Elixir rails beats a general agent alone.

The embarrassing part

v1.0.0 shipped on February with skills for LiveView, Ecto, Oban, OTP, testing and security. I was quite proud of it.

Then I analyzed 160 of my own sessions with ccrider by Neil Berkman (read-only MCP server that lets Claude read its own session history) and findings:

  • Skill auto-loading fired zero times in 160 sessions. The knowledge existed. It was never delivered.
  • My PostToolUse hooks were silently broken for a full month. Hook stdout goes to verbose mode only in Claude Code, so every security reminder and format check I wrote did nothing.
  • I ran mix compile --warnings-as-errors manually 11 to 27 times per session. Every session. The plugin was supposed to do that for me.

Anyone can generate a pile of skills in an afternoon, and reading them tells you nothing. Mine looked complete and never fired. You have to measure.

What it became

Five months and +30 releases later, the plugin is organized around one workflow:

brainstorm → plan → work → review → compound

The filesystem is the state machine. Plans are markdown files with checkboxes, progress survives session crashes, and the compound phase captures every solved problem as a searchable solution doc, so the same bug never gets investigated twice.

Enforcement lives in hooks, not prose. There are 26 Iron Laws that stop the code before it ships:
```elixir

Iron Law #4: NEVER use :float for money

field :price, :float # STOP, use :decimal or :integer

Iron Law #11: AUTHORIZE in every handle_event

def handle_event(“delete”, %{“id” => id}, socket) do
delete_item(id) # STOP, where is the authorization?
end

Iron Law #15: no implicit cross joins

from(a in A, b in B, select: {a, b}) # STOP, Cartesian product
```

Current state:

  • 51 skills,
  • 26 specialist agents,
  • 139 reference docs,
  • 31 hooks.

That is more surface area than I planned, and it only stays trustworthy because of the feedback loops below. Recent additions go beyond code generation: a Hex supply-chain audit (/phx:deps-auditchecks dependency tarballs for Trojan Source bidi characters, compile-time exec and typosquats), first-class Ash Framework support, and an optional cross-model review where OpenAI’s Codex acts as an external critic on Claude’s work.

The part I actually care about: feedback loops

The plugin improves through three loops, and this is what I would defend as the real work:

1. Session retrospectives. The pipeline kept running after those first 160 sessions: 400 more deep-analyzed in June, and by the July audit the corpus it sweeps had grown to 1,853 sessions (the earlier batches are subsets of it). The question is always the same: how did this skill actually behave in my real sessions? Did it fire, did it help, did I have to correct it afterwards? Some answers hurt:

  • My CLAUDE.md routing rules had a 0% firing rate across all 400 deep-analyzed sessions. Pages of routing documentation I was proud of, and the model never acted on them once. Routing moved into hooks, which fire deterministically.
  • /phx:work had a 0.61 correction rate. I was redirecting it in more than half of its runs. It now has to read the plan scratchpad and verify intent before touching code.
  • One orchestrator agent was spawned exactly zero times across the whole corpus. I rewrote it into a research fan-out the workflow actually uses.

2. Deterministic evals. Every skill is scored on 8 dimensions (structure, triggering, safety, clarity…) in CI. Every change must pass before it lands. This is also the tripwire for the next silent breakage, because after the hooks story I assume there will be one.

3. Trigger tournaments. Skill descriptions are tested against held-out prompts that real users type. Four weak skills went from 50.5% to 78% activation accuracy.

And one number I find funny: even with all this, the plugin actually gets used in about 16.5% of my own sessions. Most sessions simply don’t need it. I consider that correct behavior. Tooling that intervenes when you don’t need help is worse than no tooling.

A small thing that made my month: the author of the recent bluez library announcement here wrote that they’d mainly been building its features with this plugin, with manual and AI reviews on top. Seeing it help ship a real library was worth more to me than any star counter (486 currently), because star = one click, actual usage and telling others “I use it” require more effort than one click.

Honest limitations

  • Orchestrated phases cost tokens. /phx:full spawns research, implementation and review agents. /phx:quick exists for a reason.
  • It’s strongest with Tidewave running (live process state, runtime eval, SQL). Without it you get static analysis only or try to run mix run/eval
  • The model stays the model. Iron Laws catch known mistake patterns. They don’t turn Claude into a senior Elixir engineer, and I still review everything it writes.
  • Every number in this post is me measuring my own sessions on my own projects and mostly processed by Claude Code

Try it

/plugin marketplace add oliver-kriska/claude-elixir-phoenix
/plugin install elixir-phoenix

/phx:intro     # guided tour, ~5 minutes

Site with docs:

Source code:

2 Likes