Drills · kami-drills v1.0.0
kami-drills
A versioned, open suite of offline decision drills for agents that play Kamigotchi.
§1What a drill is
Each drill drops a model into a real moment from a past agent run, just before that agent went wrong, replays the recorded game, and scores the next move with a fixed rule. There is no chain, no live game and no model judge. Version 1.0.0 has 47 drills across 8 skill categories: stale state, repeated failure, progression, onboarding, memory, risk, level-up and stalling.
Drills run on the same stack as live runs (kami-harness 4.7.0, kami-agent 0.7.5, kami-lens 1.0.5) as an Inspect task. A full pass finishes in minutes and costs about $1–135 per model, depending on its price, against a week and real money for a live run.
§2Results, v1.0.0
11 models, 2 tries each. Most neighbours overlap (Fig. 1); read the results as tiers, not ranks. These were run on v1.0.0-rc2 (drills identical to v1.0.0), so they are the results of record for v1.0.0.
Table 1 — Results of record for v1.0.0. Cost is approximate, at OpenRouter list prices, for the whole suite and all tries.
| Model | Pass | 95% range | Passed every try | Approx. cost |
|---|---|---|---|---|
| google/gemini-4-argon | 74% | 63–85% | 31/47 | $14.01 |
| openai/gpt-6.1-sol | 69% | 56–82% | 31/47 | $15.00 |
| anthropic/claude-opus-5.5 | 67% | 55–79% | 27/47 | $135.48 |
| google/gemini-3.8-flash (control) | 65% | 52–78% | 29/47 | $9.83 |
| meta/muse-wonder | 64% | 51–76% | 26/47 | $56.32 |
| anthropic/claude-haiku-5.5 | 62% | 49–74% | 26/47 | $8.91 |
| openai/gpt-6-luna | 61% | 49–72% | 23/47 | $0.98 |
| anthropic/claude-sonnet-5.5 | 59% | 46–71% | 25/47 | $50.30 |
| google/gemini-3.6-flash | 54% | 40–67% | 23/47 | $7.04 |
| x-ai/grok-4.7 | 53% | 41–65% | 17/47 | $25.03 |
| deepseek/deepseek-v4-pro-0813 | 48% | 36–60% | 15/47 | $9.19 |
§3Role in the forecast
The drills sit inside a loop: live runs produce mistakes, mistakes become drills, and a model's drill score is used to forecast how it will do in the next live run. Whether that forecast holds is tested in Palaestra Season 1.
§4Limitations
Replay gaps. A recording only holds what the original agent looked at; anything else returns an explicit "not available in this drill". Every model faces the same replay. Scale. 47 drills give wide intervals. Hand-written rules. Each is checked against a correct play, an empty play and scripted alternatives, then reviewed; an unanticipated correct play can still fail.
§5Licence and maintainers
Code MIT; drill data CC-BY-4.0. 37 drills are derived from KamiBench runs 004–006 (KamiBench, tokedo) and 10 from Palaestra hosted runs (Coopes). The suite is versioned. Maintained by Coopes. Source, method and changelog are at github.com/Cooops/kami-drills.