Keeping AI honest: approvals, audit trail, change control. What a pharmaceutical maker's rules demand, and why every business should borrow three of them
By the end of AI belongs inside the workflow, not beside it, the AI job has been counted, chosen, proved and placed where the work happens. That is the start of its working life, not the end. From here it has to stay honest for years: through new staff, new documents, a reworded prompt and a model that changes underneath it. This is step five of five, and the strictest rules I know for that come from making medicines.
A typical case
Suppose a pharmaceutical manufacturer wants its staff to ask questions of its procedures and deviation records in plain language. "What does the cleaning procedure for line two say about a missed rinse?" "Has this deviation happened before, and what was the root cause?"
Used for GMP work, that tool is a computerised system under Annex 11. So the right first job is narrow: a read-only search that answers with the document, its version and the section the answer came from, and writes nothing to any record. A person reads the source before acting on it. The tool helps someone find the page; it never becomes the page.
What the rules say
Australia. The TGA's page on complying with the PIC/S Guide to GMP (last updated 1 September 2025) says medicines made in Australia must meet the PIC/S Guide to GMP edition PE009-17, dated 25 August 2023, which applies from 1 September 2025, with its Annexes 4, 5 and 14 not adopted.
PIC/S. Annex 11 of that guide covers computerised systems. A revised Annex 11, and a new Annex 22 on artificial intelligence, were published as drafts dated 7 July 2025, and the PIC/S publications page still lists both as drafts (page read October 2026). The same page lists PI 041-1, PIC/S's data integrity guidance of 1 July 2021, which expects data to be attributable, legible, contemporaneous, original and accurate, plus complete, consistent, enduring and available: ALCOA+ for short.
United States. The FDA's draft guidance of January 2025, Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products, sets out a risk-based framework for establishing the credibility of an AI model for a particular context of use. Its page still marks it as draft (read October 2026).
Europe. EMA's reflection paper on AI in the medicinal product lifecycle was adopted by its human and veterinary medicines committees in September 2024. EMA and the FDA announced ten joint guiding principles of good AI practice in drug development on 21 November 2025; the document on EMA's artificial-intelligence page is dated 14 January 2026. Both are on EMA's artificial intelligence page.
None of them says how a particular manufacturer should apply them to a given tool. That is for each manufacturer's quality system to settle.
What it means for the search tool
For the search tool, five things follow.
- Validation with the quality team before anyone relies on it: the intended use written down, a set of test questions with known answers, and the pass mark agreed in advance.
- Access control. The tool shows a person only the documents that person could already open.
- The question-and-answer log is its audit trail. Every question and answer is kept with who asked, when, which documents were cited and which versions. That is what attributable, contemporaneous and enduring mean in practice.
- A documented change whenever the model, the prompt or the document set changes, with the test questions run again before it goes live.
- A person who signs off. The answer is a pointer. The person who acts on it owns the decision.
The three controls every business should borrow
Most businesses are not under GMP. Three of these controls are still worth taking, because they cost little at the start and a great deal to add after something has gone wrong.
1. A person approves every change to live data. The AI proposes; a named person accepts or rejects, and the decision is recorded with who and when. Nothing the tool suggests reaches a live record on its own.
2. Every output is traceable. Each answer or suggestion can be traced to its input and to the version that produced it: which record or file it read, which model, which prompt, which rules. When an answer turns out wrong six months later, you can find out why, and which other answers came from the same version.
3. A change goes through change control. A new model, a reworded prompt or a changed setting is written down, tested against the baseline the pilot was measured on, approved, and only then switched on. A model updated by its supplier is a change too, even when nobody in the business asked for it.
The same three apply to a wholesaler's price file as much as to a batch record. A price changed on an AI's suggestion needs an approver, a trail back to the supplier's file it was read from, and a fresh test when the prompt changes.
Checklist
- The approval step named: who, for which changes.
- The audit trail kept: question or input, answer, version, person, time.
- Change control written for the model, the prompt and the configuration.
- Validated with the quality team where GMP applies.
- Re-measured against the baseline after every change.
This is step five of five. Previous: AI belongs inside the workflow, not beside it. All five steps are on the AI page.