About GMP Bench
What is GMP Bench?
GMP Bench is an open benchmark designed to evaluate large language models on tasks relevant to pharmaceutical Good Manufacturing Practice (GMP). It measures how well models understand regulatory requirements, generate compliant documentation, and assist with quality-critical decisions in GMP-regulated environments.
The benchmark covers two primary domains: knowledge-based question answering across regulatory frameworks (ICH, FDA, EU GMP, and pharmacopeias) and task completion involving the generation of SOPs, deviation reports, CAPA plans, and other GMP documents.
Why This Matters
Pharmaceutical manufacturing operates under strict regulatory oversight. Errors in documentation, deviations from procedure, or misinterpretation of guidelines can lead to product recalls, warning letters, or patient safety risks. As organizations explore the use of LLMs to assist with GMP activities, there is a clear need for objective, domain-specific evaluation.
General-purpose benchmarks do not capture the nuances of pharmaceutical regulation. GMP Bench fills this gap by providing test cases authored and reviewed by industry professionals, scored against verified reference material.
How Scoring Works
GMP Knowledge
Knowledge questions are evaluated by comparing the model's response against a verified reference answer. Scoring considers factual accuracy, completeness, and correct citation of regulatory sources. Each response receives a normalized score from 0 to 1.
Task Completion (LLM-as-Judge)
Task outputs, such as generated SOPs or deviation reports, are evaluated using an LLM-as-judge approach. A separate evaluation model scores the output against a rubric that assesses accuracy, regulatory alignment, completeness, and structural quality. This method enables scalable evaluation of open-ended generation tasks while maintaining consistency.
Overall Score
A model's overall score is the average across all evaluated test cases. Category-specific scores (e.g., GMP Knowledge, Task Completion) are also available for more targeted comparison.
How to Contribute
GMP Bench is a community-driven project. Pharmaceutical professionals can contribute in several ways:
- Submit test cases — Propose knowledge questions or task scenarios based on real-world GMP challenges.
- Review submissions — Vote on proposed test cases to help curate the benchmark.
- Provide feedback — Report issues with scoring, suggest new categories, or propose improvements to the evaluation methodology.
About the Maintainer
I'm Toon Lambrechts, biotech nerd turned entrepreneur and AI tinkerer with one foot in the lab and one in the terminal.
My background is in the science and engineering of making cell and gene therapies at scale. That led to a PhD in large-scale manufacturing of cell-based therapies (400+ citations) and eventually co-founding MyCellHub, a manufacturing execution system built specifically for cell and gene therapy, later acquired by Title21.
Along the way I developed a healthy(?) interest in quality systems: batch records, SOPs, deviations, change controls, CAPAs,... the whole paper trail that keeps patients safe and regulators happy. When large language models started getting genuinely capable, the obvious question was: how well do they actually understand this stuff? And how can I use them for this paperwork?
GMP Bench is my attempt to answer that rigorously rather than anecdotally.
The benchmark compares cloud and local/open-weight models on real GMP knowledge and document tasks, checked against verified references. I maintain it because I want to know the answer myself, and because I think the field deserves a neutral reference point.
If you have questions about the methodology, want to submit a test case, or are trying to figure out whether any of these models are actually deployable inside your quality system, feel free to reach out.