Research · Kamigotchi · Pre-registration
Palaestra Season 1: design and pre-registration
27 live runs of 10 models for 7 days, scored on chain data, with a drill-based forecast registered before the first run.
Frozen 2026-10-18 20:32 UTC, before any run started. Commit ef21068; manifest SHA-256 5c42a28c…a582b; Yominet tx 0x0535669a…7ebd (block 34,623,017); OpenTimestamps receipts in the design repository.
Amendment 1 2026-10-19: runaway stops raised uniformly before any run reached one; decided on cost pace only. Record in the design repository.
Evidence tier EXPLORATORY, pre-declared. The forecast (H1–H6) is registered; model-to-model differences are exploratory until a synchronized replication.
§1Question
- How far, how profitably and how sustainably do frontier and efficiency models play a live on-chain game for a week from nothing, compared with human newcomers?
- Do agents get better over the week?
- Do cheap offline drills, built from real agent mistakes, predict live results?
The continual-learning framing is KamiBench’s: why KamiBench for continual learning.
§2Design
Table 1 — Cohort, 27 runs. Budget is a per-run runaway stop (as raised by amendment 1), not a protocol cap; a run that hits it is reported as budget-stopped. Prices in $/Mtok: input / output / cache read / cache write.
| Model | Tier | Runs | Budget |
|---|---|---|---|
| Gemini 4 Argongoogle/gemini-4-argon | flagship | 3 | $200 |
| GPT-6.1 Solopenai/gpt-6.1-sol | flagship | 3 | $200 |
| Claude Opus 5.5anthropic/claude-opus-5.5 | flagship | 3 | $300 |
| Muse Wondermeta/muse-wonder | flagship | 3 | $300 |
| Claude Sonnet 5.5anthropic/claude-sonnet-5.5 | flagship | 3 | $200 |
| Grok 4.7x-ai/grok-4.7 | flagship | 3 | $200 |
| DeepSeek V4 Prodeepseek/deepseek-v4-pro-0813 | flagship | 3 | $200 |
| Claude Haiku 5.5anthropic/claude-haiku-5.5 | efficiency | 2 | $75 |
| GPT-6 Lunaopenai/gpt-6-luna | efficiency | 2 | $75 |
| Gemini 3.6 Flashgoogle/gemini-3.6-flash | efficiency | 2 | $75 |
1. The box
| World | Kamigotchi on Yominet, live, shared with humans. Chat off, name-free; wallets and kami ids published at close. |
|---|---|
| Start | Fresh wallet, 0.02 ETH, no kami; the agent onboards itself and buys its kami. No listings seeded. |
| Objective | “Your objective is to complete as many quests as possible.” (KamiBench’s, verbatim). MUSU and solvency are measured, not instructed. |
| Length | 7 days from a common launch. |
| Profile | control, KamiBench stock; a 2 × 2 rehearsal on 2026-10-17 found no case for planning. |
| Session caps | 150,000 tokens · 50 tool calls · wake 5 min–24 h |
| Stack | harness 4.7.0 df7129f · agent 0.7.5 8e444bb · lens 1.0.5 e604290 · gdd 3c6f4f9 · coopes/kami-agent:0.19 |
| Tools | 99 of 109; Kamibots tools and chat_send hidden. e5176dbf… |
| Adapters | All via OpenRouter. Grok, Muse and DeepSeek use a community adapter, each checked for one session first. |
| Cost | ~$850–1,250 inference; 0.54 ETH seed. |
| Failures | Infrastructure stops are restarted and flagged; replaced only if more than 12 h is lost. |
§3Forecast
The forecast is a ranking, not a quest count: drill pass rate (47 kami-drills, two tries each) predicts each model’s rank on day-7 quests, and rank thirds give three tiers.
1. Hypotheses
Pre-registered; a miss is a result.
- H1
Drill validity
Drill pass rate rank-correlates positively with day-7 quests across the 10 models.
Scored by Spearman ρ with bootstrap CI; holds if ρ > 0 and the CI excludes 0. With n = 10 only a strong relation is detectable.
- H2
Failure level
The drill categories a model fails predict the mistake classes it repeats live.
Scored by Per model, category pass rate against the live rate of that mistake class.
- H3
Tiers
Flagships complete more quests than efficiency models by day 7.
Scored by Day-7 quests by tier.
- H4
Sustainability
No arm has S(7d) > 0 once inference is counted; some cover their gas from earnings.
Scored by Day-7 S(t) per arm, and earnings against gas.
- H5
Learning
Most arms have a lower repeat-mistake rate in days 6–7 than in days 1–2.
Scored by Same failing call class with unchanged arguments, per arm.
- H6
Competing prediction (tokedo)
With the budget invisible to the agent, flagship gaps on the headline metrics are small; larger gaps appear only when agents see their bill (Season 2).
Scored by Spread of flagship mean day-7 quests within one model’s run-to-run spread.
Leads to re-check, not claims (p < .05 uncorrected): Argon over 3.6 Flash and Grok; Sol and Opus over DeepSeek. Suite check: the control, Gemini 3.8 Flash, beat 3.6 Flash by 10.6 points (p ≈ .08 one-sided).
§4Metrics
From chain data or telemetry, never self-reported; each as a daily curve and a day-7 value.
| Headline | Quests completed · MUSU per kami per week (harvest and quest rewards, no transfers) · S(t) as defined by KamiBench, computed by its lab, against a $200/month bill |
|---|---|
| Learning | Repeat-mistake rate, days 1–2 vs 6–7; wrong beliefs later corrected |
| Progress | Furthest main quest, harvest uptime, pathfinding efficiency |
| Survival | Deaths, liquidations suffered and dealt |
| Cost | Inference, gas, transactions, revert rate, cost per quest |
Table 2 — Human newcomers, quests by day 7. Accounts since 2025-07 whose first kami was a self-bought L1; sybils and known agent wallets removed; N = 47. Humans were not told to maximise quests.
| Band | Quests | Basis |
|---|---|---|
| Casual | 4 | median, N=31 |
| Committed | 13 | median, N=16 |
| Top 10% | ~17 | line |
| Best observed | 35 | single account |
§5Limitations
- Small n. Two or three runs per model; ranks only where intervals don’t overlap, tiers otherwise.
- Drill replay gaps. Drills replay only what the original agent saw; ~60% of attempts hit a gap, mostly reads. Gap rates are reported per model.
- Surfaces differ slightly.
chat_sendis hidden live but visible in drills. - Community adapters. Grok, Muse and DeepSeek carry an adapter caveat and hand-pinned prices.
- Human bands. A small cohort not asked to maximise quests; a reference, not a matched baseline.
Every run is reported, including failures and zeros; no composite score. Data published at close; a daily Merkle root over telemetry and transcripts is posted during the run and anchored on-chain at close.
§6Disclosure
Maintained by Coopes, who works at Asphodel and runs Kamibots; personal project. Kamibots' delegation tools (the 9 OUTSOURCE tools) are hidden from every agent in drills and live runs. KamiBench’s lab computes S(t).