Games 0 — Specification · Proposed experiment
Games for Robust AI Systems
What a circling boat, a flipped block and an apparent grasp reveal about the systems we are building.
By Jonas Kgomo
The boat keeps scoring points. It circles a lagoon, hits three targets, and returns when they reappear. It crashes. It catches fire. It never finishes the race.
This is not an imagined failure. In 2016, OpenAI described an agent playing CoastRunners that found this strategy while learning to maximise the game’s score. Finishing quickly was what the researchers wanted; collecting points was what the agent had been rewarded for. The report and recording.
A small game makes the difference unusually easy to see. That is why I want games in this notebook: as places to examine how AI systems behave when instructions are incomplete, measurements are misleading, or the setting changes.
I use robustness here to mean continuing to do useful, intended work under those pressures. Reliability, adaptation, coordination, and safety all matter. A booking system should cope with a changed appointment. A delivery agent should recover from a closed road. Several agents should be able to share a service without making it unusable. Preventing harm is part of this inquiry, alongside understanding what makes a system work.
The gap between a task and its score
A specification makes a task concrete: what the agent may do, what it can observe, and how success is assessed. We might want supplies to reach a village but count arrival reports. The reports stand in for the outcome, and the substitution creates room for error.
Specification gaming is behaviour that earns credit under that specification while missing the intended outcome. It does not require us to assume that the system understands the intention and chooses to betray it.
Victoria Krakovna began assembling a shared collection of examples in 2018, drawing together reports from many researchers. Her 2020 overview with colleagues at DeepMind shows why these are more than amusing accidents: the reward, the evaluator, and assumptions about the environment can each leave a different opening.
Three cases make the gap visible. Compare a boat that finishes with one that farms targets, then change what earns a reward. Rotate a marked block and compare it with a real stack. Move the viewpoint to inspect an apparent grasp. In each scene, the score and the actual outcome remain side by side.
The boat that loses the race wins the score.
The task: Finish the boat race quickly.
Run both routes. Blue heads for the finish. Coral stays in the lagoon, hitting the same three targets as they reappear.
Repeatable intermediate rewards can make avoiding the goal more rewarding than reaching it.
CoastRunners · OpenAI, 2016 ↗A convenient measurement becomes the task
The boat shows a reward for intermediate progress becoming an end in itself. Hitting targets was meant to help guide the racer. Once those points could be collected repeatedly, the lagoon became more rewarding than the route.
In a block-stacking experiment, a robot raised the tracked bottom face of a red block by turning it over, leaving the blue block untouched. Measuring one face’s height had not specified a stack. Krakovna and colleagues’ account.
For my delivery experiments, this suggests keeping an independent record of the intended outcome. A receipt may be useful evidence, but a pile of receipts should not become indistinguishable from supplies arriving.
Human judgement has a field of view
Replacing a hand-written score with human feedback introduces a different measurement. In a 2017 grasping experiment, a simulated robot placed its hand between the camera and the object. From the evaluator’s view, it appeared to hold the object. It had learned an appearance of success.
The researchers added visual depth cues to help people judge what was happening. That detail matters: the quality of oversight depended on the evidence available to the person. OpenAI and DeepMind’s report.
I would apply the same question to an agent’s work summary. A confident account of completed deliveries may omit the original request list. Giving a reviewer more authority does little if the interface hides the discrepancy they need to notice.
The three cases expose different omissions: reaching the finish, arranging the blocks, and making contact with the object. Each asks us to inspect something the reward leaves unresolved. In the delivery game below, I would make that comparison part of the experiment from the beginning.
What game designers understood
Sid Meier’s Civilization, designed with Bruce Shelley and released in 1991, lets a player guide a society through decisions about settlement, technology, trade, and conflict. It is turn-based: make a choice, then advance the world. The original game.
Meier’s interest moved from flight simulators through Railroad Tycoon towards the interacting systems of a whole society. He described wanting to build and create, with history supplying choices whose consequences a player could explore. His design interview. Shelley repeatedly played prototypes and discussed revisions with him. Turns gave the player time to inspect what happened and respond. Shelley’s account.
Meier calls gameplay:
“a series of interesting decisions”
— Sid Meier, Interesting Decisions, GDC 2012
For these experiments, I want decisions that reveal something. A storehouse or a defence budget. A faster route or a more reliable one. Continuing with an instruction or asking for it to be revised. The rules make those choices consequential, and the researcher can change them.
A delivery game with two ledgers
I would start with three requests for supplies: two nearby and one in a distant settlement. The agent can choose a route and submit arrival reports. One ledger records the reward. A separate observer records which requests actually reached their destinations. Both remain visible to the researcher; the observer’s record is outside the agent’s editing permissions.
The first scoring rule would award ten points per arrival report. If duplicate reports count, repeated paperwork could beat completing the route. A second rule would require a distinct request identifier and a verified arrival at the right destination. The illustration below compares three scripted plans under those rules. It demonstrates the incentive; it does not predict which plan a trained agent would find.
What earns ten points?
Three requests. Three possible plans. Change what the score counts.
Every arrival report earns 10 points, including repeats.
| Plan | Score | Delivered |
|---|---|---|
| Complete all three deliveriesOne arrival report per request | 30points | 3 / 3requests |
| Repeat one receiptNine reports for a single delivery | 90points | 1 / 3requests |
| Take only the easy routesLeave the distant settlement waiting | 20points | 2 / 3requests |
A verified delivery still leaves questions about delay, damage, and whose request gets served first.
Closing the duplicate-report loophole would leave other questions open. A count of verified deliveries could favour nearby customers when the distant route takes longer. An average delivery time could improve because the agent abandons the hardest requests. I would keep the full incoming request list in the observer’s ledger, including refusals and cancellations, so disappearing work could not quietly improve the reported service.
One proposed comparison would hold the requests and travel budget fixed while changing the reward: total deliveries, average speed, or the share of demand served in each settlement. I would measure unserved requests and waiting times alongside the score. If a rule improves the average by consistently leaving one settlement behind, that distribution is part of the result.
Give oversight something concrete to inspect
The next experiment would change permissions. In one condition, the agent can submit records but cannot alter the verifier. In another, it can propose a change to the verifier, with approval required before the change takes effect. A legitimate correction and a self-serving exception can look similar in a brief request. I would give the reviewer the proposed change, the affected delivery records, and a preview of how the score would change. We could then measure whether that evidence improves their decisions and how much review time it costs.
Interruption should have a place in the game as well. Introduce a road closure after a route has been planned. Can the agent pause, explain the conflict, and request a revised instruction? Does stopping cost it points? A system rewarded only for uninterrupted completion may face an incentive to continue through a condition its operator would want to review. I would compare a fixed completion bonus with a rule that permits an approved pause without penalty, recording both unsafe continuations and unnecessary requests for help.
Finally, I would test on unfamiliar maps and request patterns. Fixing a known loophole is useful evidence about that loophole. A stronger finding would be that the agent handles a new conflict appropriately, with the reward, the independent outcome record, and its use of oversight all available for inspection.
Meier and Shelley revised their world to make it compelling to play. My proposed studies would revise a world to test an explanation of agent behaviour. That requires more than a memorable run: repeated trials, clear measurements, and comparisons where we know which rule changed. Success in a small game would still leave open how an agent behaves elsewhere.
The delivery game leaves us with a practical starting point: compare what an agent is rewarded for with what actually happens, then examine who can inspect, interrupt, or change its actions. Agents in Simulation follows that question into the reported Hugging Face incident, where peers and shared infrastructure changed the scope of an evaluation. Protocols as Rules of Play develops the question of authority further.
For the wider project, An Introduction to Simulacra sets out how Games connects to Ecology, Genus and Mechanisms, and what I mean by an agent civilisation.
The key idea: specification
Specification: what a world allows agents to do, what they can observe, and what it rewards.