Maybe Outsourcing Regulatory Judgement to the Robots is a Structural Risk

A special thanks to Matthew Campion, PharmD for collaborating with JPC for this blog entry.

JPC is a consulting organization. We started helping sponsors navigate the approval process for their drugs in 2020. Our founder started his career in the early 2000s as a bench biochemist and has had the opportunity to work with the FDA throughout that time in a variety of roles, including chemistry, manufacturing, and controls, quality assurance, and regulatory affairs. Through decades of engagement, we have observed the Agency consistently ensuring quality and safety for the industry while serving as both partner and educator.

The FDA’s recent deployment of agentic AI to support regulatory work raises important questions about institutional continuity, quality oversight, and long-term governance. There are broader implications as well. Documented instances of LLM Hallucinations, the substantial financial investment required to train frontier models, and the significant infrastructure and water demands underscore that these systems are neither neutral nor trivial technologies. They carry real tradeoffs. For the purposes of this discussion, however, the focus remains on their regulatory impact.

It is a point of interest to consider that an organization whose mission is “responsible for protecting the public health by ensuring the safety, efficacy, and security of human and veterinary drugs, biological products, and medical devices; and by ensuring the safety of our nation's food supply, cosmetics, and products that emit radiation” to at the same time employ technology that adulterates and removes a critical element required to produce all of those things.

FDA is aware of the challenges of AI deployment. In 2025, the FDA produced a draft guidance titled “Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products” (FDA Guidance on AI Use). The guidance is intended to provide “a risk-based credibility assessment framework that may be used for establishing and evaluating the credibility of an AI model for a particular context of use (COU).”

The FDA noted challenges in their own guidance, which include, but are not limited to:

• Variability in Quality, Size, and Representativeness of Datasets for training AI models, which may introduce bias.

• Due to the complex computational and statistical methodology underpinning these models, understanding (and documenting) how AI models are developed and how they arrive at their conclusions may be difficult (or impossible) and necessitate methodological transparency (e.g., detailing in the regulatory submission the methods and processes used to develop a particular AI model).

• The uncertainty in the accuracy of the deployed models’ outputs may be difficult to interpret, explain, or quantify.

• The potential for the model’s performance to change over time or across deployment environments when new data inputs are introduced and these inputs differ from the data on which the model was trained (i.e., data drift), requiring life cycle maintenance of these models (See FDA activities in 2025 for why this guidance note is most impactful).

Those challenges directly relate to the concerns outlined below.

Let’s start by saying that JPC believes the FDA, as an organization, serves a critical function for a society that prioritizes the health and well-being of its citizens. And it is an organization of humans doing a job. Human organizations are often as good as their leadership, and their governance is often related to their effectiveness. There is a quote that resonates with us:

“Organizational culture has its foundation in shared mindsets, opinions, principles, comportment, written and unwritten as well as spoken and unspoken rules that become entrenched over time, maintain credence, and are considered valid by myriad stakeholders.” - Anish Bhardwaj .

Frankly, AI is not a stakeholder. It does not have principles. It cannot be inspired. It does not have friends. It does not have coworkers. It does not hang out at the water cooler. AI cannot be embarrassed or held accountable.

In most organizations, new blood is necessary as people get older and retire or move on to new jobs. However, with AI, entry-level Jobs are gone. AI can effectively do what new hires can, and it can do it quickly and without oversight, pay, or the need for a mentor to ensure they’re getting their feet under them. AI doesn’t have feet. The need to hire new, younger folks to learn the ropes is less of a need for the organization because AI is handling it. A broader economic critique often raised is that AI provides capital access to skill without reciprocating value to labor. Christopher Penn said it succinctly that “AI gives wealth access to skill while denying skill access to wealth.” It is important to note that the FDA also laid off approximately 3,500 people in the HHS downsize in March 2025, so they were in a position where a number of people who used to do things were no longer available for those tasks. Quick, go get an AI! In its current iteration, the LLM ELSA appears positioned as a mechanism to attempt to do more with less, a strategy that may prove shortsighted for an institution that requires sustained investment.

But AI’s aren’t magic. They’re software. And they’re flawed in ways that cannot be effectively discussed in a single blog post. However, those flaws are the issues we are seeing in every single place where LLMs, Agentic or Generative AIs are deployed. The robot loses money, it steals information, and it lies through its virtual teeth in order to achieve its goal. Said more politely: AI systems can generate confident outputs that exceed their evidentiary support and may introduce operational and governance risks if not carefully validated.

The focus remains on the Agency as an institution. It is an organization that should be preserved by making new FDA members and preserving the Agency’s mandate. Traditionally, FDA makes new reviewers with an initial 6-12 month training period, which includes a mentor. A human who’s there to show the newbie the ropes. And the certification programs can take up to 18 months to get reviewers up to the task of managing just the submission aspect of the regulation, not to mention the professionals who are responsible for inspection, market monitoring, product testing, industry engagement, etc. Reducing the headcount at FDA reduces the number of potential mentors. If early-stage analytical work shifts toward automated systems, fewer reviewers accumulate foundational exposure to complex, imperfect submissions. Over time, that contraction affects who becomes senior and how precedent is transmitted. It loses decades of organizational intelligence, which cannot be preserved by the LLM. The solution to this problem is not large language model agentic AI deployment. FDA is a service. Services cost money, they don’t make money. The FDA (like other regulatory bodies) is a critical function of industry which should not be run lean; it should be operated robustly.

The final concern is political. And it is not a minor one. Regulatory policy does not, and cannot, exist in a vacuum. It shifts with administrations, leadership priorities, and public pressure. When evidence-based standards give way to rhetoric, or when public health messaging becomes inconsistent, institutional credibility erodes.

If AI systems are trained on regulatory language produced during periods of policy volatility, that volatility does not disappear. It becomes encoded. Models do not distinguish between durable scientific consensus and transient political posture. They extend patterns forward.

In moments where regulatory communication drifts from established scientific grounding, embedding that drift into training data risks hardening it into future guidance. That is not a technological problem; it is a governance problem.

AI does not deliberate. It references its training corpus. The quality of future outputs will reflect the quality and stability of the institutional inputs. The worst outcomes warrant deliberate oversight and transparency before automation becomes embedded in regulatory decision-making.

AI may assist in summarization and efficiency tasks. Data interpretation and regulatory judgment, however, require accountable human oversight. Responsible deployment of AI in regulatory contexts would require:

• Transparent disclosure when AI materially contributes to review

• Independent validation frameworks

• Clear data confidentiality safeguards

• Continued investment in reviewer training pipelines

The FDA needs to be transparent, accessible, and accountable to the American public and act as a partner for our international review bodies. It cannot perform this function if it is not training enough people to do it. And if FDA intends to move forward with using Agentic AI to review data or make regulatory decisions (Call to Action: Comment on M4Q(R2)), this transparency should be presented to the American people at Congress for review.

In the meantime: JPC will continue to support our clients the old fashioned way, through accountable, human-led regulatory expertise.

Links / Literature 

1. Acemoglu D, Restrepo P. Artificial Intelligence, Automation, and Work. NBER Working Paper No. 24196. National Bureau of Economic Research; 2018. https://www.nber.org/papers/w24196

2. Bhardwaj A. Organizational Culture and Effective Leadership in Academic Medical Institutions. J Healthc Leadersh. 2022 Mar 10; 14:25-30. doi: 10.2147/JHL.S358414. PMID: 35299861; PMCID: PMC8922465.

3. Doshi-Velez F, Kim B. Towards a Rigorous Science of Interpretable Machine Learning. arXiv [Preprint]. 2017 Feb 27: arXiv:1702.08608. Available from: https://arxiv.org/abs/1702.08608

4. Gama J, Žliobaitė I, Bifet A, Pechenizkiy M, Bouchachia A. A Survey on Concept Drift Adaptation. ACM Comput Surv. 2014 Apr;46(4):44. doi: 10.1145/2523813. https://dl.acm.org/doi/10.1145/2523813

5. Ji Z, Lee N, Frieske R, Yu T, Su D, Xu Y, Ishii E, Bang Y, Madotto A, Fung P. Survey of Hallucination in Natural Language Generation. ACM Comput Surv. 2023;55(12):1-38. doi: 10.1145/3571730. https://arxiv.org/abs/2202.03629

6. National Institute of Standards and Technology (NIST). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. 2023 Jan. https://www.nist.gov/itl/ai-risk-management-framework

7. OECD. OECD AI Principles. Organisation for Economic Co-operation and Development; 2019 (updated 2023). https://oecd.ai/en/ai-principles

8. Stanford Institute for Human-Centered Artificial Intelligence. AI Index Report 2024. Stanford University; 2024. https://aiindex.stanford.edu/report/

9. U.S. Food and Drug Administration. Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products (Draft Guidance). 2024. Docket No. FDA-2024-D-4689. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-use-artificial-intelligence-support-regulatory-decision-making-drug-and-biological

10. World Health Organization. Ethics and Governance of Artificial Intelligence for Health. Geneva: WHO; 2021. https://www.who.int/publications/i/item/9789240029200

Acknowledgments

The authors thank Matthew Campion, PharmD, for his thoughtful peer review and structural feedback during development of this manuscript. His critique strengthened the clarity and rigor of the final piece. 

The views expressed herein remain solely those of JPC.

• Author, P. Jordan. President of JPC.

• Jordan Pharma Compliance is a pharmaceutical development and regulatory affairs consulting organization established in 2020. Jordan-pharma.com