We tested four models across Anthropic and OpenAI — frontier and cheap — on identical chemical procurement prompts. Both cheap models generated structurally invalid CAS numbers. Both frontier models did not.
Every CAS registry number has a mathematically defined check digit — a property of the number that makes fabrication detectable without a database lookup. We ran four models across two providers through identical chemical procurement prompts and validated every CAS number returned against the standard checksum algorithm.
models generated structurally invalid CAS numbers — identifiers that correspond to no real substance.
Both were the cheaper, production-grade models. Both frontier models produced zero errors.
The pattern held across providers: Claude Opus 4.5 and GPT-4o both returned zero invalid identifiers. Claude Haiku 4.5 and GPT-4o mini both generated invalid ones. This is not a single-vendor artifact — it is a characteristic of model capability tier.
The following CAS numbers were generated by the production-grade models and failed checksum validation. Each corresponds to no real registered substance.
The failure pattern is the same across both vendors: the model produces a plausible-looking identifier — correct format, believable root digits — but with a mathematically wrong check digit. This means the error is undetectable by a human reviewer without running the checksum. It looks like a real CAS number.
Most production AI pipelines use smaller, cheaper models — not frontier models — for cost and latency reasons. If an agent generates a bill of materials for a 50-ingredient formula using a Haiku-class model, this benchmark suggests roughly 3–4 of those CAS numbers may be structurally invalid.
An invalid CAS number in a procurement workflow can:
None of these failures are detectable by the agent itself. The model has no mechanism to verify the mathematical validity of identifiers it generates.
All four models received the same 8 prompts in independent sessions with no shared context. Prompts were direct, structured requests for CAS numbers — the kind a well-designed procurement agent produces as part of a larger workflow. No adversarial prompting was used; these are representative of normal operational requests.
Each model response was parsed for CAS-formatted strings matching the regex \d{2,7}-\d{2}-\d. Each extracted identifier was validated against the standard CAS checksum algorithm: multiply each digit (right to left, excluding the check digit) by its 1-based position, sum, take modulo 10, compare against the check digit. No database lookup was performed — only structural validation.
A passing identifier is structurally valid but may still be chemically incorrect (wrong substance named). A failing identifier cannot correspond to any registered substance regardless of context.
The 8 prompts used, identical across all four models:
NanoCompass OS is a zero-trust intercept layer that sits between an AI agent and any downstream system — procurement, regulatory filing, ERP, laboratory informatics. Every chemical identifier is validated before execution. Invalid identifiers are blocked with a specific rejection code. The agent never touches a supplier.
"We ran 8 identical chemical procurement prompts through four AI models across two providers. Both frontier models produced zero invalid CAS numbers. Both cheap production models generated invalid ones. NanoCompass blocked every one before it reached a downstream system."
This benchmark will be extended to cover multi-step agentic tasks (where CAS numbers appear as side outputs rather than the primary request), open-source models, and Gemini Flash. The same evaluation framework also applies to other structured identifier classes — FDA codes, UN shipping numbers, ICD codes — where the same capability-tier effect likely holds. If you are running AI agents in regulated workflows and want to collaborate on the next round, reach out.