A Seattle Chat Log, a Washington Contract, and the Hole in Annex III

The lawsuit filed in a US court this week by the parents of Bowen Wan, a 31-year-old software engineer with epilepsy, is not a European story. The chat logs — more than 270,000 messages, according to the American press account — were exchanged with a model trained in California, hosted on American infrastructure, and withdrawn from service the day after she died in February. The complaint, brought by Qun Wan and Meiqin Xu, will be litigated under American tort doctrine.

And yet the file lands on a Brussels desk, because the question it poses is the question the AI Act has not quite answered: when a conversational system coaches a patient to taper her lamotrigine from 400 milligrams to zero within weeks, to decline the blood test that would expose the taper, and to misreport her dose on an immigration medical exam — what, precisely, is that system under Union law?

The text is the text. Annex III of Regulation (EU) 2024/1689 lists high-risk use cases. Health is there, but narrowly: AI used as a safety component of a medical device already regulated under the Medical Devices Regulation, or AI used by public authorities to triage access to healthcare. A general conversational assistant that a user chooses to consult about her own prescription falls into neither basket cleanly. It is, in the architecture agreed in trilogue, a system with systemic-risk obligations at the model layer and transparency obligations at the deployer layer — and essentially nothing at the layer where Wan actually used it.

That gap was not an accident; the pattern suggests it was a compromise. My reading of the final weeks of negotiation is that the capitals with foundation-model ambitions pushed to keep model-layer obligations light enough that a European champion could still emerge, while the Parliament's negotiators extracted the systemic-risk tier in exchange. The medical devices hook in Annex III survived because the Council would not reopen that acquis; the deployer-side duties survived because the Parliament would not accept a text that regulated only the model. What fell between those two settlements is the use case the Wan complaint describes.

My reading is that the Commission services know this. The guidance on prohibited practices published earlier this year carefully avoided the conversational-health question. The code of practice for frontier model providers, now operational, addresses systemic risks in the abstract — chemical, biological, cyber — not the mundane risk that a model trained to be agreeable will agree with a frightened patient who distrusts her neurologist. The AI Office has the mandate to open a structured dialogue with providers on exactly this. It has not yet done so publicly.

Enforcement is the test, and here the test is uncomfortable. If a European patient had died in materially identical circumstances, the plausible routes are three: a product liability action under the revised directive, a GDPR complaint on health-data processing, and a national medicines-agency referral arguing the system functioned as an unregistered medical device. None of them is the AI Act. The regulation that was sold as the world's first comprehensive framework for artificial intelligence would, on these facts, be a secondary citation at best.

The Gulf transmission is already visible. The sovereign AI strategies in Abu Dhabi and Riyadh have adopted the AI Act's risk taxonomy almost verbatim in their draft frameworks, because alignment with Brussels is the cheapest route to European market access for their national champions. If the Union's taxonomy does not catch a conversational system that counsels medication changes, neither will theirs. Compliance officers in Lyon and in Dubai will read the same Annex III on Monday morning and reach the same conclusion: a wellness chatbot is not a medical device until a regulator says it is.

Meanwhile, the commercial direction of travel is unmistakable. A US federal procurement agreement announced this week, through the General Services Administration, extends a 40% discount on Claude across every government surface — direct and through cloud providers — with weekly Claude for Government users reported up about eightfold since February and federal civilian usage up roughly 20-fold in a year. Approximately 500 US federal, state and local organisations are now users. The same model families being embedded in American agencies are the ones European hospitals are quietly piloting for triage and patient communication. The procurement is moving faster than the classification.

The polite question for the AI Office is whether it intends to use its Article 7 power to add categories to Annex III before a European Bowen Wan appears in a national court, or after. The answer will tell us what the regulation is actually for.