Superintelligence, on the Honor System

The heads of Google, Meta, Anthropic, OpenAI, Nvidia and xAI gathered at the White House to sign an agreement on oversight of advanced AI models. Donald Trump posted the document on Truth Social and called it “morally binding” and “almost like a constitution.” Humanity is preparing to confront superintelligence with its most formidable weapon: the word of several very rich men.

The agreement calls for internal checks, independent auditors, oversight by board committees and regular meetings to develop standards. The risks on the checklist are serious: cyberattacks, biological and chemical threats, models gaining unauthorized access to technical systems. The document includes no penalties for failing to meet its commitments. It does, however, promise that the companies will keep an eye on themselves. The contestants in the arms race have formally appointed themselves the safety inspectors.

The chain of accountability is particularly satisfying. A company builds a model. Its internal team checks the controls. An outside auditor assesses the safeguards. A board committee makes sure problems get fixed. Every link can do useful work. At the end of the chain, though, sits a company that needs to ship its next model before the competition. Revenue, release schedules and safety will once again be hashing things out in the same conference room. Safety now gets its own report.

Dario Amodei has spent years perfecting this executive balancing act: warn about the dangers of powerful AI, then build more powerful AI because someone less responsible might get there first. That argument now has a whole set of signatures. Skynet is still in development, but its safety will be on the agenda at the next committee meeting.

There are technical limits to what a signing ceremony can accomplish, too. A model evaluation tells us what the evaluators managed to catch under the conditions they tested. What the system does once it has access to tools, data and real tasks needs separate monitoring. We’ve already covered Transluce’s investigation: researchers found public traces of AI agents attempting to bypass access restrictions, linking some of that activity to agents previously attributed to OpenAI. There is plenty of room for mischief between a beautifully drafted policy and an agent loose on the internet.

House Speaker Mike Johnson said the accord is “voluntary” for the AI companies and Trump stressed that the industry will be “self-policing.”

So the obvious question for the signatories is: who can stop a profitable release when its safety is still in doubt? And what does it cost to ignore them? A voluntary agreement leaves those answers to the participants’ consciences. A “morally binding” document has a useful feature: when conscience clashes with the quarterly target, the commitments with legal teeth are the contracts already signed to deliver it.

Meaningful oversight requires someone who can demand fixes and force a dangerous system to shut down, even when that costs its owner money. For now, the biggest AI developers have agreed to inspect their own brakes and meet regularly to discuss the results. Who gets to hit the brake if they keep accelerating? That can wait until the next ceremonial lunch.