Deficiencies in AIs; What We Can Learn
No not that AI. I’m referring to FDA’s Additional Information (AI) letters. Also known as “hold letters” or AINNs (don’t ask me what the NN stands for - if you know please let me tell me!). Nonetheless, if you clicked only for the AI, keep reading; this post is relevant to artificial intelligence in medical devices, too.
Note: For clarity, I’ll call them “hold letters” throughout the rest of this article.
Hold letters are a treasure trove of current regulatory thinking, but often they are left untapped. Large companies that have access to many of them would do well to mine hold letters and deficiencies as a source of critical information about the current path to market. I’ve personally audited a sample of deficiencies; trust me there’s a ton of value to be gleaned if you have a few.
What is a Hold Letter, and why do they matter?
A hold letter is an almost-inevitable communication from FDA that lists what additional information (thus “AI”) is needed for the review team to clear or approve a device submission. In other words, a hold letter describes where and how the sponsor’s submission is deficient. Hence “deficiencies.”
After reviewing a submission in its entirety (leaving aside the interactive review format, for the moment), the review team gives the sponsor a list of deficiencies. These gaps could be minor or major, spanning from the lack of a small piece of information that won’t materially change the outcome, to a major safety concern that could ultimately cause the submission to fail. Once a submission receives its hold letter, a clock starts. Sometimes companies scramble to respond, sometimes they’re ready, sometimes they could have predicted the deficiency list perfectly, and sometimes they’re caught completely by surprise. Recently (especially in cybersecurity) I have seen unplanned response efforts in the millions of dollars. A CEO of a midsized company, blindsided by a list of more than nineteen major cybersecurity deficiencies, told me “This is existential, and our market is competitive. I don’t care how much it costs. Every day matters. Save me days.” His experience is not unique.
What are deficiencies?
Deficiencies are the atomic element of a hold letter. They are the punch list, the final steps needed to get regulatory approval.
They’re written by the review team writes deficiencies to explain to the sponsor what is needed for clearance.
They can be major or minor.
They can be “stock” or bespoke. (For example, there are at least 17 pre-written or “stock” deficiencies for cybersecurity, based on my observations across multiple companies’ submissions.)
There are guidances about them, and the latest guidance must be updated as part of MDUFA V.
Deficiencies are so important to the industry, they’re referenced in MDUFA commitments, with progressively more specificity with every new MDUFC commitment letter:
MDUFA III includes a commitment that FDA will issue complete deficiency letters - that the deficiencies will be the total set, and FDA cannot add more later, based on further reading of the submission.
In MDUFA IV, FDA committed to updating guidance around deficiencies and explicitly committed to including a statement of the policy or scientific basis for the deficiency. Industry wants to compel FDA staffers to cite their sources, think hard, and explain their rationale before requiring additional information. And get a manager to review it. (Note: I intend to write about the mutual perceptions of industry and FDA review staff in a future post. It is quite interesting to have seen both sides’ beliefs about each other's motivations and day-to-day life. My summary takeaway is: have empathy, folks, for the FDA heroes who toil for the public good.)
In MDUFA V (2023 through 2027), FDA committed to annual audits of deficiencies’ adherence, updating guidance for even more clarity on the four-part harmony, additional training for staff, and progressively increasing performance against the four-part harmony standard (i.e., 75% compliance in FY 23 to 90% in FY26).
Why are they a treasure trove?
Deficiencies are the latest FDA thinking. Guidances are too, but they take a long time to publish. Guidances are a culmination of the FDA’s thinking. While the guidance is being developed, hold letters with deficiencies keep getting sent out. Deficiencies are the latest. They may not represent an agency-wide view, but each one is a data point.
Why should large companies mine and catalog deficiencies as a standard process?
Imagine being able to take in new deficiencies and distill the lessons learned, feed the lessons back to the R&D teams so that they may avoid the same problems with the next submission. Knowledge management nirvana!
Realistically:
Imagine a database with all of your company’s deficiencies pulled from AI PDFs, categorized, and searchable.
Imagine an updated SOP or checklist of very common mistakes and a quick job aid or training for staff.
Imagine finding deficiencies discordant with public FDA statements; imagine a real-time feedback loop for deficiencies that enables a product team to push back on FDA’s request.
Imagine realizing that a new deficiency is consistently being issued, and it takes a long time to remediate. Imagine it’s a harbinger of requirements for all submissions, and there’s value in systemic change. Imagine it offers an opportunity to save millions in remediation costs.
It happens. The next section gives an example.
Sometimes a deficiency is a canary, signaling trouble ahead for anyone not paying attention: An Encryption Case Study.
Here’s a cautionary tale. It’s a real story, as seen in multiple manufacturers.
Medical devices lag consumer and connected technology in encryption design. Cybersecurity leaders in medtech knew this in 2017 and earlier, and they also knew that some devices were getting through FDA even with outdated, breakable encryption designs. Despite this knowledge, the cyber leaders could not easily move their colleagues to make a systemic change fast enough, because it appeared that FDA wasn’t yet explicitly and concretely requiring it.
But FDA was requiring it. FDA was issuing deficiencies with encryption requirements, albeit not consistently across all devices. Despite receiving hold letters with encryption deficiencies that caused major disruption in one product, some manufacturers didn’t apply the learning to the other product areas. The companies failed to recognize and act on the canary for what it was. Now and in the very recent past, they are retrofitting encryption designs to the tune of tens of millions of unplanned dollars and market delays of months or years. If they had been mining deficiencies, they may have saved an order of magnitude of costs for implementation.
The lack of modern encryption isn’t the only cybersecurity gap that is costing way more than it should to fix during hold letter remediation. And, there is more to the story than what I know and what I can fit here. But I know this story isn’t rare. (I am aware of at least two leaders at different companies who tipped off FDA to cybersecurity design gaps so that the R&D team would receive the deficiency and finally close the security gap in the design. This is guerrilla regulatory innovation that I don’t condone, but can’t argue with its effectiveness.)
Sometimes a deficiency (or absence of one) is a harbinger of good things to come.
Deficiencies aren’t all bad news. Sometimes, they show where FDA is opening a new door. For example, the first time a manufacturer submits a predetermined change control plan to facilitate updates to a machine-learned or artificial intelligence medical device without additional regulatory review, it is a win of huge proportions that should be shouted from the rooftops so that other teams can take advantage of the development.
Deficiency and hold letter mining is fun.
I designed the audit of deficiencies while I was leading the team conducting the most recent independent assessment of FDA’s MDUFA IV performance. Together with my team, we created a repeatable process to mine deficiency text, tested the reliability of the mining, label it with findings and categories, and summarize the results. The audit was successful and is now required annually (per MDUFA V). And it was great fun. We read hundreds of deficiencies across a wide variety of 510ks, PMAs, de novos from all the OHTs. I wouldn’t mind doing it again.
Given a small or large corpus of deficiencies at a medium to large manufacturer, I 99% guarantee there’s untapped value in mining the dataset for insights. Possibly with LLMs, but more likely with humans who can read. (I’ll bet a combo of the two is best.) But I digress.
~Shannon, the Optimistic Optimizer
Ps. Hat tip to Matt Healy, Ph.D., my former boss and mentor, who taught me what I know about MDUFA, among other things. Matt has led or overseen every MDUFA Independent Evaluation so far.