New benchmark tests AI strategy by having models play Slay the Spire
SpireBench shows that top AI models can match average human players in a card game but still struggle with long-term strategy and risk-taking.
A new way to test AI strategy
Researchers have introduced SpireBench, a benchmark that evaluates how well large language models can plan and make decisions over long periods of time. Instead of giving AI short puzzles to solve, they let it play the full commercial deck-builder card game Slay the Spire at the hardest difficulty level.
Most current AI benchmarks only test models on short, one-off tasks, making it hard to see whether an AI can truly think ahead. SpireBench changes that by creating a setting where early choices shape the outcome many steps later — much like real-world planning.
How the models performed
Across 20 different GPT-5.5 and other frontier models, the results showed mixed outcomes. The strongest models beat basic computer opponents and matched the average human player, but they fell far short of expert human players. 1
In Act 1 of the game, the top AI models won about 90 percent of runs — nearly matching expert streamers, who win 85 to 92 percent of the time. But as the game continued, the gap grew wide. In Act 2, models won only 40 to 50 percent of runs compared to 65 to 78 percent for experts. By Act 3, models managed just 0 to 20 percent wins against 56 to 71 percent for experts. 3
What goes wrong
The main problem came down to risk. All the models were overly cautious. They chose to keep their health high instead of spending it to make their character stronger. At the Act 1 boss fight, the best models kept a median of 90 percent of their maximum health, while expert human players held only about 53 percent. 3
The models also failed to learn from experience. Researchers gave the AI the ability to write down memories between game runs so it could improve over time. But those memories were often wrong — 17 percent of stored memories contained false information — and the advice was heavily skewed toward caution. Risky suggestions were outnumbered seven to one by safe advice. 3
Tests also showed that part of the models' success came from prior knowledge of the game built into their training data. When researchers renamed every game term with a consistent replacement, the models' performance dropped, confirming that memory of Slay the Spire strategy played a role in their scores.
Why it matters
SpireBench gives researchers a tool to pinpoint exactly where and why AI agents fail over long stretches of decision-making. As AI systems are pushed into more complex, real-world roles, understanding these gaps in long-horizon planning will become increasingly important.
Key facts
- Top AI models won Act 1 of Slay the Spire 90% of the time, matching expert humans, but dropped to 0-20% by Act 3
- Models were overly cautious, keeping median 90% health at the Act 1 boss versus 53% for expert players
- 17% of memories written by models between runs contained false information
- Renaming all game terms lowered model performance, showing prior game knowledge helped scores
- The benchmark tested 20 large language models at the game's highest difficulty
Sources
- SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire
- SpireBench
- SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models ... — academy.dair.ai
- SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models ... — observatorioblockchain.com
- Protocol · SpireBench