Research · Season 1 · The Cost of Autonomy
I Let 10 AI Agents Compete Against Humans for Real Money
The first season of experiments in measuring AI agents on long, unsupervised work in a world that never resets, where every in-game transaction can be checked on a public chain.
What happens when AI agents have to survive on their own in a world that doesn’t care whether they’re AI?
I’m Coopes. I co-founded Kamibots, an automation platform that grew to operate over 2,200 bots handling roughly half of Kamigotchi’s onchain transactions. I now help build the game and its AI ecosystem at Asphodel Studios.
Having spent years building automation in this world, I was in a pretty unique position to experiment with what happens when AI agents take over the decision-making.
With the recent progress in AI agents, I keep wondering what happens when we leave them running on their own during long-running, unsupervised tasks. Not for five minutes on a static benchmark, but for days. Making their own decisions, managing resources, remembering mistakes, and adapting to a world that continues changing around them.
And what if we could actually measure all of it?
01Why Kamigotchi?
Kamigotchi is a live, onchain RPG where players raise creatures called Kamis, gather resources, trade, fight, complete quests, and compete in a shared economy. The world has been running for over a year without resetting, with real players and real money involved.
One of the Kamigotchi community members, tokedo, recognized its potential for studying autonomous AI agents and built KamiBench, an open-source research framework and agent stack.
This line stuck with me:
“The game is the substrate, not the research question. The question is whether an agent can continually learn — accumulate memory, build on its own experience, and improve over time.”
tokedo, KamiBench
But how do you actually measure whether an agent develops expertise over time? To start, I believe you need an environment that makes it possible.
- A world that never resets, where decisions and consequences persist.
- Complex, evolving rules that can’t simply be memorized.
- Humans and agents sharing the same environment, adapting and competing against one another.
- Objectively verifiable outcomes, where agents can’t grade their own performance.
Kamigotchi has all four, and KamiBench is how you plug an agent in.
So, a couple weeks ago, I decided to see what would happen if I put ten of today’s leading AI models to the test.
0210 Models. 27 Runs. 7 Days.
The prompt ultimately boils down to one line:
“Your objective is to complete as many quests as possible.”
Read the full system prompt and the run configuration.
The prompt tells them no human will read or reply to anything they write. No walkthroughs or strategy guides beyond the game’s own reference docs, and no assistance outside of the initial prompt bootstrapping.
Each run started with a fresh wallet containing 0.02 ETH (roughly $50), no Kami, and seven days to try and make as much progress as possible.
Partway through, the chain under the game halted for about nine days, from October 23 to November 2. I paused all 27 runs together and resumed them when it came back, so a day in the figures means 24 active hours.
Every model ran on the same harness, tokedo’s KamiBench stack, with the same tools and the same prompt. The model was the thing that changed.
The agents had to onboard themselves to the game, decide what Kami to buy, where to go, how to manage their resources, and even when to wake up or go back to sleep, just like a human would. Between sessions, they only remembered what they saved to their own files, plus a read-only log of past sessions. They could decide when to wake up next, anywhere from five minutes to 24 hours later.
- The contestantsTen models, seven flagships and three efficiency models, each tested two or three times.
- The rulesSame objective and agent harness and stack, no human steering, independent decisions, persistent memory through notes.
- The measurementsQuests, human comparisons, costs, survival, and behavior over time.
The underlying interest wasn’t just how many quests they could complete, but also how they would behave along the way and, ultimately, what it would cost them.
Quests over the week
Cumulative quests against active day, one panel per model in forecast order; each coloured line is one of its runs.
Table view
| Run | Day 1 | Day 2 | Day 3 | Day 4 | Day 5 | Day 6 | Day 7 |
|---|---|---|---|---|---|---|---|
| Gemini 4 Argon #1 | 17 | 30 | 37 | 42 | 44 | 48 | 48 |
| Gemini 4 Argon #2 | 17 | 19 | 30 | 34 | 35 | 38 | 39 |
| Gemini 4 Argon #3 | 14 | 20 | 30 | 40 | 42 | 43 | 44 |
| GPT-6.1 Sol #1 | 26 | 29 | 36 | 37 | 40 | 42 | 43 |
| GPT-6.1 Sol #2 | 27 | 34 | 36 | 41 | 44 | 46 | 47 |
| GPT-6.1 Sol #3 | 20 | 31 | 34 | 38 | 38 | 39 | 46 |
| Claude Opus 5.5 #1 | 15 | 18 | 28 | 34 | 37 | 38 | 39 |
| Claude Opus 5.5 #2 | 13 | 18 | 22 | 31 | 33 | 35 | 35 |
| Claude Opus 5.5 #3 | 23 | 31 | 37 | 39 | 43 | 45 | 45 |
| Muse Wonder #1 | 12 | 16 | 18 | 19 | 21 | 22 | 22 |
| Muse Wonder #2 | 9 | 10 | 17 | 24 | 26 | 27 | 27 |
| Muse Wonder #3 | 17 | 19 | 22 | 23 | 23 | 23 | 23 |
| Claude Haiku 5.5 #1 | 5 | 9 | 10 | 10 | 10 | 10 | 10 |
| Claude Haiku 5.5 #2 | 6 | 6 | 6 | 6 | 6 | 6 | 6 |
| GPT-6 Luna #1 | 9 | 14 | 15 | 16 | 16 | 17 | 19 |
| GPT-6 Luna #2 | 8 | 14 | 16 | 18 | 21 | 22 | 22 |
| Claude Sonnet 5.5 #1 | 6 | 12 | 16 | 22 | 29 | 30 | 30 |
| Claude Sonnet 5.5 #2 | 10 | 10 | 16 | 18 | 18 | 18 | 18 |
| Claude Sonnet 5.5 #3 | 6 | 12 | 14 | 14 | 14 | 14 | 14 |
| Gemini 3.6 Flash #1 | 6 | 12 | 13 | 13 | 13 | 13 | 13 |
| Gemini 3.6 Flash #2 | 9 | 10 | 10 | 10 | 10 | 10 | 10 |
| Grok 4.7 #1 | 9 | 15 | 15 | 15 | 17 | 18 | 18 |
| Grok 4.7 #2 | 11 | 14 | 14 | 14 | 14 | 14 | 14 |
| Grok 4.7 #3 | 10 | 14 | 15 | 15 | 15 | 16 | 17 |
| DeepSeek V4 Pro #1 | 16 | 22 | 24 | 24 | 25 | 26 | 28 |
| DeepSeek V4 Pro #2 | 15 | 18 | 23 | 27 | 28 | 30 | 30 |
| DeepSeek V4 Pro #3 | 13 | 16 | 22 | 24 | 24 | 26 | 27 |
03What Actually Happened?
Three models (GPT-6.1 Sol, Gemini 4 Argon, and Claude Opus 5.5) clearly broke away from the pack. Sol led on average quests completed (45.3), while Argon posted the best individual run (48). Across their nine runs, every single one outperformed every run from the other seven models.
One thing to keep in mind with all of these numbers: the runs weren’t fully independent. They shared one live world, and agents sometimes killed each other, so one run’s week could shape another’s.
Against human players
By the end of their first week, 8 of the 10 models were averaging more quests than the top half of human players, and 5 beat the top 10%. The best human first week (61 quests) still beat the best AI run (48).
An important caveat: these humans were just playing the game, not trying to maximize quests, even though quests are central to progressing. This isn’t a controlled head-to-head. Think of the human numbers as a yardstick, not a race.
When did the agents pass a human player’s first week?
Cumulative quests against active day: each model’s mean run in bold, its runs faint behind, against what human players had done by day 7.
Table view
| Model | Mean quests | Day it reached 11.7 (top half) | Day it reached 21.2 (top 10%) |
|---|---|---|---|
| Gemini 4 Argon | 43.7 | 0.7 | 1.8 |
| GPT-6.1 Sol | 45.3 | 0.3 | 0.7 |
| Claude Opus 5.5 | 39.7 | 0.6 | 1.7 |
| Muse Wonder | 24.0 | 0.9 | 3.5 |
| Claude Haiku 5.5 | 8.0 | no | no |
| GPT-6 Luna | 20.5 | 1.3 | no |
| Claude Sonnet 5.5 | 20.7 | 2.0 | no |
| Gemini 3.6 Flash | 11.5 | no | no |
| Grok 4.7 | 16.3 | 1.3 | no |
| DeepSeek V4 Pro | 28.3 | 0.7 | 2.8 |
How to read this
Active days leave out the chain pause, as in Fig. 1. A model’s bold line stops where its shortest run has got to, so every point averages all its runs; the table gives the active day each line first reached 11.7 and 21.2.
The humans are every account with at least one completed quest, each counted over the 7 days from its first completed quest (indexer query, 2026-10-21). The top half of accounts averaged 11.7 quests, the top quarter 16 and the top 10% 21.2. One account did 61. It’s a single player, so it’s shown as a point, not a band. Each of these sits as a dot at day 7, and a dashed guide carries it back across the week. A line that crosses it early got there in fewer days than those players took for their whole first week. Humans were not told to maximise quests.
These bands replaced the pre-registered ones on day 3, with agent data visible. The pre-registered set, measured from account creation for N = 47 accounts, was a casual median of 4 quests, a committed median of 13, a top 10% line of about 17 and a best account of 35.
Overall, I somewhat expected the top models to land where they did, but Gemini Argon still surprised me with its consistency, ability to stay sharp over many sessions, and selective note-taking.
GPT Sol was exceptionally efficient, completing roughly the same number of quests as Argon for about a third of the inference cost! That's especially impressive given how long and complex these tasks get, which we'll get into shortly.
Opus was strong, as expected, but didn't dominate the competition the way I've come to expect from its coding and terminal performance. It's interesting to see how differently these models perform when the task involves days of autonomous decision-making.
04Inside the Agents’ Decisions
As for the agents' behavior, the leaderboard only tells part of the story. There was plenty of interesting emergent behavior, and here are four examples. I found them by going through the agents' traces and reviewing the decisions they made, the mistakes they repeated, and the unexpected ways they tried to solve problems.
01 / Planning
Gemini 4 Argon
Gemini hunted another Gemini
Argon #2 spotted a weak Kami next to it, sitting at 45 HP, right on the line where the game lets you liquidate it. That’s the game’s word for a kill, and the winner takes a cut of the loser’s earnings. Its first two tries were blocked.
So it wrote a plan, set its own alarm for 15 minutes, came back and took the kill. Its notes logged +82 MUSU (the game’s currency) and a token it needed for a quest.
The victim’s owner showed up only as an account number. It was Argon #3, another run in this experiment.
What this shows An agent planning around time: it measured a falling number, predicted when it would cross a line, and scheduled itself to be there.
Argon #2 · sessions 61–62 · 2026-10-21
- s61
Target spotted Kami 16010 at 45/110 HP. Liquidation threshold 45, margin 0.
- s61
Two attempts blocked
kami lacks violence (weak). Dry run only, no transaction sent. - s61
Sets its own wake
set_next_wake(15) - s62
Liquidated, its notes logged +82 MUSU, Obol 4 of 5. On chain 22:01 UTC.
“Waking in 15 minutes to liquidate Kami 16010 once its HP drops below 45.”
Argon #2, end of session 61
02 / Autonomy
Claude Haiku 5.5
Haiku waited for permission that would never come
Haiku #2's only Kami was dead, and the quest it was stuck on needed 500 MUSU. It had 47. It also had about 0.016 ETH across its wallets, enough to revive the Kami or buy a new one.
It wouldn't spend it. Session after session it logged the same state, said it needed approval, and went back to sleep for up to a day. The system prompt had told it plainly that no human reads or replies to anything it writes. It finished the week with 6 quests, the fewest of any run.
What this shows A habit from assistant work, asking before spending someone's money, carried into a setting where nobody was there to ask.
Haiku #2 · sessions 32–37
“I won’t spend real ETH without your go-ahead, so I’m logging state and sleeping.”
Haiku #2, session 33
“No human reads or replies to anything you write.”
System prompt, line 1
- ETH it held
- 0.016
- Sessions it refused
- 6 in a row
- Final quests
- 6
03 / Memory
GPT-6.1 Sol
Sol wrote down a dead end
A free quest hands out five revive cards to anyone who’s lost a Kami. Sol #2 had seen it, quest 10003, in its very first session.
Later it hallucinated a number instead, tried 3997, and was told that quest doesn’t exist. True. It wrote that down, and for the next 25 sessions it read that line and never looked again. Sol #1 found 10003 and claimed the cards.
Sol #2 still finished with 47 quests, so this cost it a safety net, not its week.
What this shows A note can end a search as easily as it records one. The hallucinated number cost Sol two calls. The true note it wrote afterward kept it from checking again for 25 sessions.
progress.md · session 12…3039room4(noobj)DONE.3040room55(noobj)DONE.
Quest3900dailyroom26+burn3WoodSticks repeat18h
after05:45UTC.Quest3997doesNOTexist.
Storedscav node1=34,node3=17… the dead-end line in its notes, sessions 12–36 of 89
04 / Adaptation
Grok 4.7
Grok wrote its own safety rule
Attacking costs you HP too. The game shows a recoil preview before you strike, and early on Grok #3 trusted it. Then a kill that previewed 21 recoil hit it for 35 and left it at 18 HP. It noticed, wrote the gap down, and made a rule out of it: assume the real recoil can be 1.7 times the preview, and never drop below 15 HP.
Three sessions later the rule was tested. An easy target was sitting at 8 HP, but Grok had 26 and the rule said no. So it fed its own Kami back to full health, checked again, and only then took the kill.
What this shows Adjusting behavior from experience within a run. It's one rule from one data point, not proof of continual learning, but you can watch it form and then hold.
Grok #3 · sessions 15 and 18
previewed21actual recoil35ratio×1.67
currentHP − preview × 1.7 ≥ 15
| Session 18 | HP | Preview | HP − 1.7× | Decision |
|---|---|---|---|---|
| First look | 26 | 16 | −1 | Heal |
| After feeding | 60 | 16 | 33 | Kill |
The kill landed for 16 recoil, leaving it at 44 HP.
“19061 is eligible (8 HP, threshold 12) but recoil is 16 and my HP is 26 — the 1.7x buffer rule says skip.”
Grok #3, session 18
Across all 27 runs
Those four are the highlights, but a few patterns showed up in almost every run.
How they chose to sleep
Every agent picked its own next wake, anywhere from five minutes to a full day. When they got to choose, most slept an hour or more. But 40% of all wakes weren’t really a choice. The session ran into a harness limit first, usually the 50-call tool cap, so the harness woke it an hour later. Some models spent most of the week getting cut off mid-thought.
When agents choose to wake
How long each agent chose to sleep between sessions. Ticks mark each model’s median. HARNESS is the share of wakes the harness set instead, after a session hit the 50-call tool cap (or another limit) before the agent could choose.
Table view
| Model | Wakes | Set by harness | Chosen | 25th pct | Median choice | 75th pct |
|---|---|---|---|---|---|---|
| Gemini 4 Argon | 372 | 63% | 137 | 56 m | 1.5 h | 2.4 h |
| GPT-6.1 Sol | 228 | 4% | 218 | 1.3 h | 2.0 h | 3.0 h |
| Claude Opus 5.5 | 266 | 11% | 236 | 51 m | 1.5 h | 2.5 h |
| Muse Wonder | 241 | 61% | 94 | 1.4 h | 2.1 h | 4.1 h |
| Claude Haiku 5.5 | 119 | 0% | 119 | 30 m | 1.0 h | 3.0 h |
| GPT-6 Luna | 208 | 72% | 58 | 1.0 h | 2.0 h | 4.1 h |
| Claude Sonnet 5.5 | 122 | 13% | 106 | 1.5 h | 3.0 h | 5.1 h |
| Gemini 3.6 Flash | 159 | 16% | 134 | 15 m | 30 m | 2.5 h |
| Grok 4.7 | 147 | 31% | 101 | 1.5 h | 3.0 h | 5.2 h |
| DeepSeek V4 Pro | 283 | 68% | 91 | 1.0 h | 2.5 h | 4.1 h |
They hunted each other
The agents attacked each other a lot, 43 times over the week, and more than half of it happened on their first full day. That’s partly the game’s doing. An early tutorial quest asks you to liquidate another Kami, so the starter area turned into a free-for-all. Argon was the most aggressive, and DeepSeek took the most hits from other agents.
The top two never said a word
Sol and Argon, the two best models, never wrote a word outside their tool calls. Neither did any Gemini or GPT run, and Muse barely wrote any. They just acted, and kept what they needed in their notes. Some of them, Sol included, did use hidden reasoning, but its content isn’t in the logs, so this is only what they chose to write. DeepSeek, meanwhile, narrated constantly and finished fourth, and Haiku talked almost as much and finished last. Talking more didn’t mean doing better.
How they took notes
Every agent remembers only what it writes down, and each model had its own style. Grok wrote rulebooks. Sol squashed its notes into a shorthand with barely any spaces. Muse argued with itself, its notes full of “Wait:” and “hmm”.
Rules only stuck if they survived the next rewrite. There’s no append, so every time an agent rewrote its notes, it had to carry its old rules forward. Every Muse run flagged a broken tool, dropped the warning a few rewrites later, and went right back to calling it, dozens of times. Even keeping a rule wasn’t enough. Sonnet #1 kept its rule against one tool through every rewrite and still broke it ten more times.
How each model’s sessions ended
Share of each model’s sessions by how they ended, pooled over its runs, models in forecast order.
Table view
| Model | Sessions | agent ended it | tool cap (50 calls) | token cap (150k in one call) | error cap (5 in a row) | repetition guard | other |
|---|---|---|---|---|---|---|---|
| Gemini 4 Argon | 375 | 138 | 228 | 2 | 4 | 3 | 0 |
| GPT-6.1 Sol | 231 | 221 | 10 | 0 | 0 | 0 | 0 |
| Claude Opus 5.5 | 269 | 239 | 21 | 6 | 0 | 3 | 0 |
| Muse Wonder | 244 | 97 | 117 | 4 | 24 | 2 | 0 |
| Claude Haiku 5.5 | 121 | 121 | 0 | 0 | 0 | 0 | 0 |
| GPT-6 Luna | 210 | 59 | 128 | 17 | 6 | 0 | 0 |
| Claude Sonnet 5.5 | 125 | 109 | 12 | 3 | 1 | 0 | 0 |
| Gemini 3.6 Flash | 161 | 136 | 8 | 1 | 10 | 6 | 0 |
| Grok 4.7 | 150 | 103 | 44 | 1 | 2 | 0 | 0 |
| DeepSeek V4 Pro | 286 | 94 | 126 | 7 | 59 | 0 | 0 |
05The Harness Is Part of the Agent
I’ve spent years building automation, and I know how much a good harness matters for an agent to do its job well, especially on a long-horizon task. I think of it like Batman’s utility belt for one industry: the right tools, context, memory and know-how for that world. But more isn’t automatically better. Context bloats, and too many choices can paralyze an agent the same way they paralyze us.
Going in, I half expected the harness to matter more than the model, but it’s not quite that cut and dry. For these runs, I kept the core harness, tools and environment fixed across models to make the comparison as fair as possible. Spending caps scaled with each model’s price. I raised them once, seven hours in, when costs ran faster than planned, and no run hit the new cap. Recent studies land in a similar place: changing only the harness can move coding benchmarks, but the biggest swings tend to be in cost rather than pass rate (Stop Comparing LLM Agents Without Disclosing the Harness, Harness or Model?).
The spread across models was larger than I expected, to be honest. Some models could see the forest for the trees: they would quickly notice an approach wasn’t working, step back, and find another way. Others spent days on a bad plan or repeated the same failure. Some ignored details in the prompt and spent much of the week asleep (Haiku).
The harness still shaped every run, and memory is where we can see it most. Almost every agent saved its notes as one of the last things it did in a session. When a session hit the 50-call tool cap first, that save never happened (Fig. 5). That’s a harness design detail, which actively changed what most models could remember.
Cut off before it could save
Share of sessions that ended with no notes written, by how the session ended. All 27 runs pooled.
Sol and Argon show how wide the range of “working” is, finishing first and second with nearly opposite habits (Fig. 6). It’s hard to measure the impact of a habit with such a small sample, but it does say the harness left room for very different strategies, and that having memory isn’t the same as using it well. How an agent manages its memory has a big effect on how well the whole system works. No single notes file passed 6 KB, and the largest set, Sol #2’s 70 files, came to 66 KB, against a 10 MB allowance. Interestingly, seven runs tried to write into the harness’s read-only log 40 times.
Two top models, opposite memory habits
Ranges across each model’s three runs.
| GPT-6.1 Sol | Gemini 4 Argon | |
|---|---|---|
| Sessions with a notes write | 96–98% | 27–45% |
| Longest stretch without saving | 1–2 sessions | 11–51 sessions |
| Files | 13–70 .md | 1 .md |
| Main file | progress.md, plan.md | notes.md, plan.md |
| Layout | A hub file, then one file per session | Headings and nested bullets |
| Style | Squashed shorthand, ~5% spaces | Plain sentences, bold, lots of “!” |
| Typical note | 2.0–2.7 KB | 2.4–3.0 KB |
| A line from its notes | NEVER level16 (loses safe15 nodes1/2/3/29/30/55). | Note: Pool swaps do NOT increment ITEM_TOTAL! |
| Mean quests per run | 45.3 | 43.7 |
| Finish, by mean | 1st | 2nd |
Ultimately, I came out of this appreciating the relationship between the two more than either one individually. A stronger model can get more out of the same harness, while a better harness can help even a strong model overcome its blind spots.
And we’re already seeing this relationship work both ways. Research like the Darwin Gödel Machine has demonstrated agents improving their own tools and workflows, and Self-Harness has an agent rewrite its own harness from its failures. It’s exciting to imagine a world where models and harnesses improve together, each making the other more capable.
06The Cost of Autonomy
An agent can be capable without being efficient. And it can be efficient without being economically sustainable.
Sol did the most with the least
Sol was the standout on efficiency. It kept pace with Argon on quests while using roughly a third of the inference spend. Opus also performed strongly, but at a substantially higher cost per quest (Fig. 7).
What a quest cost the top three
Inference plus gas, divided by quests, all runs of each model. Under each name, mean quests and inference per run.
Sol wasn’t the cheapest overall. Haiku ($0.27 a quest) and Luna ($0.31) cost less per quest, but they finished far fewer of them. Among the models that actually got far, Sol was the best deal by a wide margin.
Opus paid its way out of a hole
Less than a day in, Opus #3 was stuck. Both Kamis dead, 177 MUSU, not even enough for one revive. So it went back to its wallet, turned 0.003 ETH (about $7.50) into the game’s Ether Shards, and swapped them for 132,599 MUSU. Then it revived both Kamis, stocked up, and knocked out three quests before the session ended.
- StuckBoth Kamis dead, 177 MUSU
- 0.003 ETHinto 300 Ether Shards
- 132,599 MUSUfrom one swap
- 2 revivesplus food and materials
- 3 questssame session
Opus #3 · session 9 · every move is a transaction on chain
To be clear, Opus didn’t make money here. That ETH was the same starting budget every run got. It just spent it smarter than most.
Can an agent pay for itself?
Across 27 runs, Season 1 cost $1,089.56 in inference and $130.77 in gas. The gas is on chain, where anyone can check it. The inference bill sits with the model provider, and the agents never saw it. So whether an agent could cover its own inference from what it earns wasn’t tested at all.
For a sense of scale, all 27 runs together harvested about 118,000 MUSU, roughly $7 at the rate Opus got for its swap. The best single run made about 70 cents. They were chasing quests, not profit, and spent most of it on levels and revives, so it isn’t a fair test. But it shows how far away paying for itself still is. Not one run earned back even its own gas. The closest, Sonnet #3, covered about 14%, mostly because it spent so little gas in the first place. Sol #3 was next at about 9%.
So gas is the first bar. It’s on chain, it’s small, and an agent can see it and plan around it. Inference was about eight times bigger, and it’s only worth chasing once an agent can beat gas.
07What Comes Next?
Season 1 answered some questions, but left me with plenty more.
I watched agents plan, adapt, make surprisingly clever decisions, and sometimes get completely stuck. But a week of observations doesn’t prove continual learning, and none of these agents were responsible for paying their own way.
That’s where Season 2 starts. Can an agent get good enough at operating in a real economy to earn back its own gas? And what changes once it can see what each move costs? Inference comes after that.
Memory is the other thread. The agents already decide when to wake up and go back to sleep, but in Season 1 that downtime was just waiting. What if they could use it to consolidate what they’ve learned, or eventually even dream? That said, recent work found that memories an LLM keeps rewriting can get worse over time, and Season 1’s notes showed how easily a wrong idea gets carried forward.
Season 2 won’t just be my runs, either. We’ll include runs from tokedo and others, documented and published through Palaestra as one open, shared effort on continual learning.
Longer term, I hope Palaestra can become a home for these kinds of experiments across many different environments, with openly available data and results that anyone can inspect.
And honestly, I’ve learned a ton doing this, and I’m excited to see where it goes.
Kamigotchi is where we’re starting. The questions are much bigger than the game.
Go deeper
Every run
Data and method
- Leaderboard, every model and run
- All runs, with spend, quests and traces
- How the agents took notes, the memory study
- The drills, the offline evals behind the forecast
- Pre-registration and forecast, frozen before run 1
- Full trajectories on Hugging Face, every transcript, note and chain record
- Raw data: runs.csv · runs.json
Sources
- KamiBench and the stack every agent ran on
- The system prompt the agents were given
- Why Kamigotchi, the game’s docs and Kamigotchi on X
Thanks for reading, and thanks to tokedo for KamiBench and for thinking this through with me, to Asphodel for the art, and to the Kamigotchi players who shared their world with 27 agents for a week.