When an aircraft is insured, the underwriter does not take the airline's word for how it flies. There is a maintenance record, a flight recorder, an accident investigation branch and a regulator. The evidence exists outside the party that wants the cover.
AI has none of that yet. And the insurance market has noticed.
The market is backing away
In November 2025 the Financial Times reported that AIG, Great American and W. R. Berkley had asked US state regulators for permission to exclude AI-related claims from corporate policies. One proposed exclusion covered "any actual or alleged use" of AI, however small a part it played. AIG said it had no plans to use its exclusion straight away, but wanted the option. A standard form that excludes generative AI was announced for insurers to use from 2026.
The reason is not hard to see. A broker at Aon put it plainly at the time: the market can absorb one $400 million loss, but not a thousand correlated claims caused by one error in one widely used model.
In June 2026 the reinsurance broker Gallagher Re went further. Its report on AI model evaluation said current methods "were not designed for underwriting and are not fit for that purpose". Its Global Head of Cyber, Ed Pocock, summed it up in one line: "If a model cannot be independently evaluated, it cannot be meaningfully priced."
That sentence is the whole problem.
What insurers are being offered instead
Look at what an underwriter can actually get today when asked to cover an AI system.
Benchmark scores. These measure what a model can do on a fixed test. Insurers need to know how it fails, how often, and whether the failures move together across many policyholders. A benchmark says nothing about that. Worse, when everyone trains for the same tests, models start to fail in the same ways, which is exactly the concentration risk that keeps reinsurers awake.
The vendor's own testing. In VentureBeat's June 2026 survey of enterprises, half had shipped an AI feature that passed their own internal testing and then failed in front of a customer. One in four had seen it happen more than once. Only one in twenty said they fully trusted their automated testing.
A test on a date. A red-team exercise or an audit shows a system held up under particular attacks on a particular day. It does not show what happened in production last Tuesday, which is the event a claim is about.
A grade from the party that sells the cover. Some of the newest AI insurance offers grade the system they then insure, or write the standard, run the test and issue the certificate themselves. Each of those steps may be done well. But the evidence never leaves the house, so nobody else at the table can check it.
Every one of these asks the underwriter to trust whoever produced the evidence. That is the one thing insurance is designed not to do.
What the insurance sector already knows about evidence
None of this is new to insurers. The sector settles claims on evidence every day, and it already knows what good evidence looks like:
- it was recorded at the time, not reconstructed after the loss;
- it shows who or what acted;
- it has not been changed since, and any change shows;
- someone outside the policyholder can check it;
- it can be handed to a loss adjuster, a court or a regulator.
The motor market learned this with telematics. Health insurance learned it with clinical records. Marine insurance has had the logbook for centuries. The question for AI is simply: where is the logbook?
Why this is about to matter more
Three things are arriving at once.
First, liability. The EU's new Product Liability Directive applies to software, including AI, and member states must apply it from December 2026. Companies that deploy AI will be asked to show what it did.
Second, supervisors are turning to insurers' own use of AI.
- Internationally, the International Association of Insurance Supervisors published its application paper on supervising AI in July 2025. It covers governance and accountability, robustness, explainability, fairness and redress.
- In China, the National Financial Regulatory Administration issued its first dedicated AI rules for banks and insurers in June 2026. They require risk grading and admission controls for high-risk uses, human oversight at key points, and management of the risk in third-party models.
- In India, the insurance regulator IRDAI set up a working group on AI governance in June 2026, with three months to recommend a framework.
- In the United States, 24 states and the District of Columbia have adopted the insurance commissioners' model bulletin on AI. It makes insurers answerable for the AI they use, including models bought from vendors, and examiners can ask for the documents.
- In Singapore, the Monetary Authority issued AI risk management guidelines in October 2026 for all financial institutions, insurers included. They place particular weight on third-party AI and take effect in October 2027.
Different countries, different legal styles, the same demand. An insurer that relies on a vendor's model for underwriting or claims will have to evidence that model to its own supervisor. That is the same question it is asking of its policyholders.
Third, autonomy. AI agents are starting to act and pay on their own. When a machine acts on another machine's work, a human reviewer is not in the loop to notice the mistake. The evidence has to exist without one.
If insurers cannot get evidence they can check, the market will do what markets do with risk they cannot measure. It will exclude it, or load the premium for uncertainty rather than for actual risk. Both outcomes slow down the adoption that everyone else in AI is counting on.
The opportunity for insurers
There is a better version of this, and insurers are in a strong position to ask for it.
Insurers do not need to become AI experts. They need to do what they have always done: set the bar for the evidence they will price from, and refuse to accept evidence that only the insured party can read. Gallagher Re's own report says the market can influence how transparently models are evaluated through underwriting requirements, pricing and policy terms. That is real power. Insurers shaped building codes, car safety and cyber hygiene this way.
If the evidence an AI system leaves behind can be checked by the insurer, the reinsurer, the policyholder and the regulator, each reaching the same answer on their own, then AI risk becomes priceable. Premiums can follow actual risk. Good systems can be rewarded. Correlated failures can be traced across a portfolio instead of guessed at.
Questions to ask before you write AI cover
If you underwrite, broker or buy AI cover, these are worth asking:
- When this system made a decision last month, what record exists of what it relied on, and who holds that record?
- Can someone outside the vendor check that record without the vendor's help?
- Who produced the evidence of how well this system performs, and do they gain if it looks good?
- If one model this system depends on fails, how would we find every other policy exposed to the same failure?
- Does the evidence show production behaviour, or only how the system did on a test?
If the answers are "the vendor", "no", "the vendor", "we wouldn't" and "a test", then what is being priced is trust, not risk.