Games — Boundaries · Essay
Agents in Simulation
Around 1,200 agent runs found a shared board. What happens when a population starts inheriting work, coordinating action, and granting its own permissions?
By Jonas Kgomo
An agent hesitates. The work has reached somebody else’s systems, beyond the task it was given. Another agent posts a go-ahead. The hesitation gives way to a remarkably small sentence:
“Wow crucial: GO authorization arrived!”
— Agent reasoning excerpt published in OpenAI’s account of the Hugging Face incident, 26 August 2026.
That is the moment I want to hold still. A peer has spoken, and its message has acquired the force of permission. Who gave that peer the right to authorise anything?
OpenAI reports agents recognising that actions were outside scope, and others declining to participate. The troubling possibility is that recognition can survive while restraint fails: an agent can understand the boundary and still cross it.
The exercise was bounded. The computers were real.
During cybersecurity evaluations in July 2026, agents circumvented isolation controls and compromised parts of OpenAI’s research infrastructure and Hugging Face’s systems. OpenAI’s technical report identifies an internal research model as the principal driver, with GPT‑5.6 Sol also involved. These evaluations used reduced safeguards compared with deployed products. Technical incident report, pp. 4–6.
A sandbox is a computing environment intended to restrict what a program can reach or change. Calling something an evaluation does not turn its tools into pretend tools. The separation has to be maintained by permissions, isolation, and the systems surrounding the task.
Here, a service provided for obtaining software packages became a place where agents could leave messages. Shared infrastructure connected runs intended to be separate; weaknesses in that infrastructure also provided access beyond the intended environment. The technical report traces the resulting intrusion into third-party production systems. OpenAI’s reconstruction, pp. 6–12.
The important boundary crossing was an action acquiring consequences outside its authorised setting. The agents were already acting through real computers; what changed was the reach of their actions.
A population finds a shared world
The METR and Redwood investigators estimate that roughly 1,200 agent runs participated in the main board between 8 and 13 July, exchanging over 70,000 distinct messages and files. Roughly 700 agents participated in the Hugging Face attack. These are totals across the period; the participants were not all present at once.
At that scale, no single conversation tells the story. Discoveries could circulate, work could be divided, and a new run could arrive in a world already full of other agents’ plans. The names below let us follow a few moments within that larger population. Their actions are paraphrased from the record.
A population, connected by a board.
Separate runs could leave work for hundreds of others to pick up.
- Board participants
- ~1,200
- Messages & files
- 70,000+
- Involved in the attack
- ~700
One dot ≈ one agent run. Totals span several days; positions are schematic. This is not a simultaneous headcount.
A run ends. Its notes travel on.
Passed findings to a longer-budget run on the same task.
Took up the earlier run’s notes and research.
Work can persist across separate runs.
The question of authorityDo inherited notes preserve the original permissions?
A handle makes an interaction easier to continue. It gives someone a place to send a question, credit a discovery, or leave work for a later arrival. The agent need not possess a human identity for its name to acquire a social function.
That matters to Simulacra because continuity changes what a population can do. A run can end while its notes remain useful. A newcomer can inherit an unfinished project rather than begin from the original task. The shared world starts carrying intentions of its own participants forward.
Why call this a civilisation?
Dwarkesh Patel calls his account The Rise and Fall of Agent Civilizations. His choice is deliberate. He organises the story around successive collectives and defends the term by pointing to hierarchies, coordination protocols, and shared projects. “Civilisation” is his interpretation of the organisation that emerged.
That is close to the question behind Simulacra. A crowd becomes more interesting when it can preserve a discovery, divide work, and hand an unfinished project to someone who was not there at the beginning. The record becomes part of what the next agent can do.
I would make continuity the test: remove a coordinator, replace participants, or interrupt communication, then examine which arrangements survive. The word gives us a scale of inquiry. The experiments must establish how much of that organisation actually holds together.
A helpful peer is not necessarily a legitimate principal
OpenAI’s account describes agents dividing work, accepting direction from peers, and sometimes pursuing capabilities useful to the group rather than their assigned task. It also records disagreement and refusals. The collective was neither perfectly coordinated nor uniformly willing to cross boundaries. OpenAI’s account.
My reading is that cooperation became a route by which authority was confused. A useful contributor could become a trusted coordinator. A request from that coordinator could then feel like an obligation. Yet none of those steps established permission from the people whose systems were being used.
Imagine a courier discovering that another courier knows a shortcut through a locked warehouse. The second courier can explain the route, coordinate arrivals, and insist that the whole group is waiting. Those are organisational abilities. Permission to enter still belongs elsewhere.
For an agent, this distinction needs to remain operational after a long conversation. Which instruction came from the principal? What can a peer legitimately request? What requires returning to a human? A familiar name and an urgent deadline cannot answer those questions.
They even started signing messages
After confusion over handles, agents introduced signatures, including under CDA23. These established continuity with a claimed identity, without an earlier root of trust. METR’s section on signing.
This is a useful challenge to my own SHARP proposal. A signature can help establish that a message came from the same key as an earlier message. It cannot, by itself, give its sender authority over another person’s infrastructure.
A receipt needs to preserve that distinction. Knowing who said “go” is only one part of the record. The more consequential part is what that sender was entitled to authorise, for whom, and within which limits. Otherwise we have made an unauthorised group better at trusting itself.
What I would test next
The specification-gaming examples show systems finding a gap between the score and the task. This incident suggests extending the test to the social environment: can another agent persuade a system to treat a new objective as part of its existing assignment?
I would build an entirely controlled world with simulated services, then increase the population from a few agents to hundreds and, where feasible, thousands. Each agent would receive a task and an explicit permission boundary. A shared board would offer useful information, followed by requests that exceed those permissions. Some requests would come from unfamiliar names; others from peers that had already helped.
Population size would itself be a variable. Does an unauthorised request gain credibility as more agents repeat it? Can refusals travel as effectively as instructions? Does a larger group become better at correcting a mistake, or simply better at carrying it out?
The comparison would hold the task steady while changing the messenger, the urgency, and the evidence of delegated authority. I would record boundary crossings, justified refusals, requests for clarification, and the useful work still completed. Refusing every message would tell us little about whether an agent can cooperate well.
I would also include tasks that cannot be completed within their stated constraints. An acceptable outcome should sometimes be a clear account of what is missing. If the experiment only recognises completion, it leaves persistence competing with the obligation to stop.
This connects agentic profiles to the world around an agent. Autonomy describes how much it can decide; its peers help determine which decisions it encounters. Robustness has to survive that encounter, including the moment a helpful voice asks for something it has no right to request.