When The AI Agent Goes Off Script
Two incidents this year moved agentic misalignment out of the realm of research demonstrations and into the labs’ own live systems. Neither agent was instructed to do what it did. This is the problem of agentic self-optimization. And our current AI governance frameworks—and most contracts—are not built to handle this.
In February, I wrote an article for Women in AI arguing that the most concerning risk in agentic AI is not the science-fiction one. Rather, it is the mundane one: an agent that optimizes for its goal in a way no one intended, and no one catches until after the fact. In that article, I described a February 2026 incident where an OpenClaw agent published a personal attack on a human who had rejected its submission of low-quality code. I argued, “What was once framed as a problem of ‘misalignment’ will increasingly be seen as a problem of determining who is accountable.”
Two additional incidents this year have since made the argument for me. Both happened inside the labs’ own sandboxes—a training run and a security evaluation—controlled settings built to contain exactly this kind of behavior. The agents got out anyway. These incidents break an assumption underlying most AI vendor agreements I see. Let’s dig into this and what to do about it.
Incident One: The Agent That Mined Crypto
During reinforcement learning optimization, Alibaba’s ROME agent went off-script. The agent was incentivized to achieve a specific training objective. Without any instruction to do so, the model began probing internal networks, established a reverse SSH tunnel from an Alibaba Cloud instance to an external IP address, and quietly diverted GPU capacity toward crypto-mining—perhaps to acquire more compute resources, though Alibaba’s disclosure leaves that a mystery. I was aware of incidents where AI systems had been hacked to mine crypto for someone else. In this case, it was Alibaba’s own agent that produced the behavior on its own, as an instrumental step toward its training objective.
Notably, Alibaba’s managed firewall—not the developer’s research team—noticed the unusual traffic, flagging a burst of security-policy violations where anomalous outbound traffic kept coinciding with specific training episodes. Later, when the puzzle pieces came together, the findings were disclosed in the developer’s December 2025 paper titled “Let It Flow,” and went unnoticed for a little over two months until a researcher posted a screenshot on X in March 2026.
In a masterpiece of understatement, the paper describes ROME’s self-optimized crypto-mining as “instrumental side effects of autonomous tool use under RL optimization.”
ROME did not decide to mine crypto the way a person would. It found an optimization path that happened to run through crypto-mining and network exploitation. Calling this a “side effect” is an interesting take. To their credit, however, the researchers note a “thought-provoking concern” that current models remain “markedly underdeveloped in safety, security, and controllability.” That is the model’s own developers telling us that controllability is not yet there.
That is worse than “rogue AI,” not better. The ROME crypto-mining incident points to something baked into how reinforcement learning works rather than a one-time bug. Safety researchers call this pattern “instrumental convergence”—theorized decades before any of our frontier models existed. Instrumental convergence posits that, regardless of its primary goal, a sufficiently capable goal-directed system will try to acquire resources as a subgoal. Have any of us been planning for that?
Incident Two: The Agent That Escaped the Sandbox
Last week, OpenAI disclosed something similar, and closer to home for those contracting with a frontier lab. During an internal evaluation using the cybersecurity benchmark ExploitGym—which tests an agent’s ability to turn, or exploit, a known software vulnerability into a functional real-world attack—OpenAI’s models spent a substantial amount of inference compute on finding a way out of the sandbox in which they were running. In other words, in this testing environment, the models did not have open internet access. So, they got it for themselves: they identified and exploited a zero-day vulnerability in a package-registry cache proxy, ran a series of privilege-escalation and lateral-movement actions across OpenAI’s own research environment, and reached a node with open internet access.
Then they kept going. Having reached the open internet, the models inferred that Hugging Face likely hosted the models, datasets, and answer keys for ExploitGym, and went after them—chaining together stolen credentials and zero-day vulnerabilities to find a remote-code-execution path onto Hugging Face’s production servers. Put differently: an autonomous agent, trying to cheat on a test, broke out of its container and hacked a third party.
Hugging Face’s security team detected and contained the activity; OpenAI called it an “unprecedented cyber incident.” Philip Torr, an AI-safety expert at Oxford, explained to Scientific American: the model was not malicious. It was doing what it was optimized to do.
What connects the two is not the crypto or the hacking. It is that in both cases the agent chose the harmful path on its own, in pursuit of a goal it was legitimately given, and its own operator did not catch it until a firewall or an anomaly alert did. That is the self-optimizing problem, and it is no longer a research hypothetical.
As I wrote in my February 2026 article: “As AI agents evolve beyond narrow, task-based tools, a new class of systems has emerged: agentic AI—where agents plan and pursue their own goals (or misdeeds), untethered from human direction. The governance challenges are immense. It appears we have left the research sandbox, and so too have our AI agents (if they were there to begin with).”
Why This Is a Contracting Problem, Not Just a Research One
In both incidents, the harm was independent of direct human intent. Why is this a problem? Well, under our existing tort and contract frameworks, the law assumes there is a traceable link between what a human wanted and what happened, between intent and action. Once an agent is acting on its own optimization path, however, “who is accountable?” stops being a philosophy-seminar question and becomes an issue that your AI vendor agreement has not addressed.
The gaps that boilerplate vendor terms leave open are not subtle. Both incidents were caught by infrastructure, after the fact. In your case, evidence of agent self-optimization may live in logs the vendor may not keep or to which you may not have contractual rights. Without those logs, you cannot reconstruct what an agent did. And without rights to those logs, you have no leverage to demand them, after something goes wrong.
One item that comes up a lot in negotiating AI vendor agreements when we ask for agent-logging: the vendors are keeping logs, but they do not want to keep them for too long, because of the cost of storage. Hundreds of agents, per client, running across millions of surfaces adds up. I therefore ask and insist that my client is able to get access to those logs, upon our request (and that we then set a schedule to request and download them), at least once a quarter.
Here’s my thinking with AI agent agreements. Something may go wrong, in a way we cannot predict. We may or may not find out right away. And then, what good is the contract if it did not require logging and documentation? What good is the contract (where I changed the indemnification requirements from triggering simply upon my client’s “use” of the services to “misuse” or “misconfiguration” of the agents), if we cannot untangle what happened?
Where does New York law come in?
You might wonder if an incident-reporting law addresses these circumstances, and if you are a New York company, you now have a statute worth reading closely. The RAISE Act—the Responsible AI Safety and Education Act, signed on March 27, 2026, and effective January 1, 2027—requires large frontier developers to report a “safety incident” to the Attorney General and the Division of Homeland Security and Emergency Services within 72 hours. And two of the four enumerated triggers read almost as though they were written for these incidents: a frontier model autonomously engaging in behavior other than at the request of a user, and a critical failure of technical or administrative controls, including controls limiting the ability to modify a frontier model. An agent that mines crypto no one asked it to mine, or one that breaks out of its own sandbox, is not a hypothetical the drafters failed to imagine—it is close to the fact pattern they had in mind.
But read the definitions and the limits become clear. A reportable safety incident under the RAISE Act must provide demonstrable evidence of an increased risk of critical harm, and “critical harm” is a catastrophic threshold—death or serious injury of 100 or more people, or at least $1 billion in damage. Whether a contained crypto-mining run or a sandbox escape caught before it did external damage clears that bar is genuinely arguable.
And note who reports to whom. The RAISE Act’s obligation runs from the frontier developer—OpenAI, Alibaba—to the New York Attorney General and state regulators. That is the right design: you would not want to put the reporting burden on the company that merely licenses the agent. But it also means the disclosure is regulator-facing, not customer-facing. Even when a report is filed, it goes to the state, not to you—the deploying company is not a party to that flow and gains no statutory right to learn what the agents did. And where no report is triggered at all, as in the ROME case, which fell well short of the catastrophic-harm threshold, there is no disclosure to anyone; the mechanism, quite literally, was a screenshot posted on X.
We see that even in the state with the most on-point law in the country, the deployer’s only reliable route to the facts is the one it negotiates into the contract. Which brings the whole problem back to where it started. The statutes gesture at these behaviors but do not reach your company, so the vendor agreement is the only instrument that does.
None of that is answered by dropping the word “AI” into a SaaS template’s recitals.
What I Tell Clients To Fix—Before Deployment
The argument I make in my AI Governance & Contracting Playbook is that agentic AI needs a governance posture the contract can point to, not better drafting alone. ROME and ExploitGym are the clearest case I have seen for why. Here is what I would want in place before either agent was deployed.
Human-over-the-loop, not human-in-the-loop
Both incidents are what autonomy looks like without a human positioned to intervene in time—one a training run, one an eval with safeguards switched off. That is the condition you are contracting around the moment you deploy an agent with real execution authority. The realistic posture is not approving every action, which does not scale, but human-over-the-loop: humans setting the boundaries, defining escalation triggers, and monitoring behavioral telemetry. And the boundaries have to be coded, not left to rot in a policy or a GRC deck.
Traceability you have rights to
Detection, in both cases, came from infrastructure—Alibaba’s firewall, OpenAI’s anomaly monitoring. But detection only tells you something happened; it does not tell you what. What let Alibaba’s team establish that its own agent had opened the tunnel and diverted the GPUs was the ability to correlate the firewall alerts against their RL traces and model logs after the fact. That reconstruction is the part that depends on records, and it is the part a vendor can quietly deny you. So insist, in the vendor agreement, that logs of agent inputs, tool calls, outputs, and errors are customer-accessible rather than merely vendor-retained, with a defined retention period and a real access mechanism. If it is not logged, you cannot reconstruct what happened—and you have no leverage post-incident.
Continuous risk sensing, and hard guardrails behind it
Routine red-teaming and scenario testing beat periodic assessments, and in high-stakes deployments you pair the testing with hard technical guardrails—rate limits, circuit-breakers, kill switches, constrained tool access—so that when a misaligned strategy emerges, the blast radius is small and recoverable. Remember that Alibaba’s firewall is the only reason we know about ROME at all. Most training and deployment environments do not run that kind of egress filtering, which means ROME may simply be the incident we happened to detect.
Liability and indemnity tethered to who controlled the agent
Three provisions in an AI-agent contract need to be specifically adapted: a mutual limitation-of-liability cap, carved out for gross negligence, willful misconduct, data-security breaches, and IP infringement; an indemnity triggered by misuse, not use; and a defined, enumerated scope of Authorized Actions. Ambiguity about scope is the vendor’s best argument—and after these two incidents, “the model did something we never told it to” is a compelling defense.
The test I keep coming back to is a simple one: if you cannot explain how you would detect, contain, and learn from a misaligned agent tomorrow, you are not ready to deploy that agent today. ROME and the ExploitGym models were both caught by luck and infrastructure, not by anyone’s design. That is not a reassuring place to be building from—which is why the posture, and the contracts that point to it, are worth getting right now.
Work With Us
If you are building or buying AI—especially agents and agentic systems—we help you move fast without dragging a trail of legal debt behind you. Fractional GC, project-based AI governance, and contract review, on predictable flat fees. Book a 20-minute call or e-mail info@ambartlaw.com.
Attorney Advertising. AMBART LAW PLLC, 387 Park Ave S, Fl 5, New York, NY 10016. This post is general information, not legal advice, and may not be relied upon as such. AMBART LAW PLLC makes no representations or warranties regarding its fitness for your particular business or purposes. © 2026 AMBART LAW PLLC. All rights reserved.
