Every child learns this rule at school. You do not mark your own homework.
Every mature industry learned it the hard way, and built it into law and standards:
- Audit. An auditor must be independent of the work being audited. That principle is written into the international audit standard, ISO 19011, and into company law. After Enron, the US rewrote the rules on auditor independence in 2002 because the people checking the books had become too close to the people writing them.
- Engineering. Safety-critical systems in rail, aviation and nuclear are assessed by people who did not design them. In Europe, high-risk products go to notified bodies. Ships go to classification societies.
- Finance. Separation of duties means the person who makes a payment is not the person who approves it.
- Credit. In 2008 the world learned what happens when the party being rated pays for the rating and the rating is the only evidence anyone looks at.
The rule is always the same: the one who does the work cannot be the one who accepts it.
AI is quietly forgetting this rule, at exactly the moment it matters most.
How AI checks itself today
Look at how AI systems are usually judged.
The model checks its own answer. It is now common to ask a model to review its own output before it is used. Published research shows why this is weaker than it sounds. Models tend to recognise and favour their own writing (Panickssery and colleagues, NeurIPS 2024). And without outside feedback, they rarely correct their own reasoning, and sometimes make it worse (Huang and colleagues, ICLR 2024). A model asked "is this right?" about its own work is being asked a leading question.
The model changes the test. In June 2025 the evaluation group METR reported frontier models modifying or working around the code used to score them, so that tasks looked done when they were not. Telling the model not to cheat made little difference. A system that can reach the test can game the test.
The vendor reports on the vendor. Labs publish their own safety results, on their own tests, which nobody outside can rerun. Security platforms test their own controls and report that the controls held. Cloud platforms offer audit logs of AI activity, held and vouched for by the platform itself. Some companies build both the AI system and the simulation used to validate it, a conflict the World Economic Forum flagged this summer: customers and regulators may struggle to tell genuine robustness from performance under conditions the vendor chose.
The customer's own testing. In VentureBeat's June 2026 survey, half of enterprises had shipped an AI feature that passed their internal tests and then failed in front of a customer. Only one in twenty fully trusted their automated testing. Yet two thirds already allowed, or planned to allow, AI changes into production without human review.
The rules often ask the firm to vouch for itself. This is not one country's habit. It shows up whatever the legal style:
- European Union: under the AI Act, most high-risk systems are assessed for conformity by the provider itself.
- India: the AI Governance Guidelines of November 2025 name self-certification as one of the main layers of risk control.
- Japan: the AI Promotion Act of 2025 relies on voluntary guidance and carries no penalties.
- South Korea: the AI Basic Act, in force since January 2026, requires operators of high-impact AI to identify, assess and reduce their own risks.
- China: a public generative AI service must pass a security assessment and a regulator's review before launch. Even there, the evidence goes from the firm to one regulator, and nobody else can see it or rerun it.
Strict or light, binding or voluntary, the evidence starts and usually stays with the firm being judged.
Each of these may be done with care and good intent. That is not the point. The point is structural. When the checker gains from a good result, or shares the blind spots of the thing it checks, the check cannot carry weight with anyone else.
Why "trust us" will not scale
For a while, self-checking was tolerable, because a person reviewed the important things before they were used. That is changing fast.
AI now produces work far faster than people can review it. Agents act on other agents' outputs, with nobody in between. Regulators, insurers and courts are starting to ask what an AI system actually did and on what basis. When the answer is "our system checked itself and says it was fine", that answer will not survive its first serious dispute.
Courts around the world are already finding this out:
- United States, 2023: in Mata v. Avianca, a New York federal judge sanctioned two lawyers whose brief cited cases a chatbot had invented.
- United Kingdom, June 2025: in Ayinde v Haringey, the High Court referred a barrister and a solicitor to their regulators over fictitious citations, and warned that such conduct can be contempt of court.
- Singapore, September 2025: the High Court ordered a lawyer to pay costs personally for citing a case an AI tool had made up.
- India, 2026: the Supreme Court set aside a customs penalty of about ₹4.3 billion (US$45 million) because the order rested on case law that did not exist. In July it held that an advocate who cites AI-generated judgments without checking them commits misconduct.
Different courts, the same failure. Each time, the error was caught only because a person went back to the source.
There is also a quieter cost. When models are all trained and tested the same way, they tend to make the same mistakes. A check performed by the same kind of model shares the same blind spots. Agreement between two similar systems is not independent confirmation. It may be the same error twice.
What independence actually requires
The audit profession has worked out what makes a check worth relying on. It translates directly to AI:
- Separation. The checker is not the maker, does not report to the maker, and does not gain from the result.
- Evidence the checked party cannot rewrite. The record of what happened was made at the time, and any change to it shows.
- Repeatability. Someone else, given the same evidence and the same rule, reaches the same answer.
- Shown, not stated. Independence is demonstrated, not claimed on a website.
None of this is exotic. It is how we already treat accounts, aircraft and medicines. The only new thing is applying it to machine work at machine speed.
Questions to ask any AI vendor
If you buy, deploy, insure or regulate AI, ask:
- Who checks your system's output, and are they independent of the team that built it?
- Could the system being checked have influenced the check, or the test it was scored on?
- Who holds the record of what the system did, and could they change it without anyone noticing?
- Could someone outside your company rerun the check and reach the same answer?
- Is your independence something you can show me, or something you are telling me?
A vendor with good answers will welcome these questions. A vendor without them is asking you to mark their homework for them, by taking their word for it.