The Food and Drug Administration is working on guidance for use of generative artificial intelligence in medical devices, following years of debate about the technology.
The agency is taking a closer look at generative AI as consumer tools proliferate, but no medical devices on the market today use it in a regulated way.
Medtech firms are watching for what direction the FDA plans to take as big questions loom on how to evaluate devices using the technology for safety and efficacy, and how to ensure they continue to work over time.
The agency’s device center shared a discussion paper in August calling for feedback on how to ensure medical devices that use AI are safe and effective before and after they enter the market. The FDA intends to “better understand the unique challenges presented by generative AI-enabled medical devices and inform the development of future guidance,” Grace Davis Jamison, spokesperson for the Department of Health and Human Services, wrote in an email to MedTech Dive.
Medical device experts shared their thoughts on the FDA’s proposals and what they would like to see in future guidance on generative AI. Here are four questions about the FDA’s approach to the regulation of generative AI:
1. What does generative AI look like in medical devices today?
Generative AI refers to a technology that can create a variety of outputs, such as text or images. Chatbots, such as OpenAI’s ChatGPT and Anthropic’s Claude, and image-generation tools like Midjourney are some widely known examples of consumer tools.
The challenge with generative AI is that it can provide many different outputs that may change. This makes it more difficult to evaluate for safety and efficacy, and to ensure that a model’s performance doesn’t worsen over time. The technology’s tendency to produce “hallucinations,” incorrect outputs that may appear authentic, and its environmental costs are also major concerns raised by potential users and communities.
While the FDA has authorized more than 1,500 devices with an AI component so far, it has not yet authorized a generative AI-enabled device, Davis Jamison wrote.
The agency is now taking a closer look at the technology.
“They’re really starting to go through the motions more than just talk,” said Suzanne Levy Friedman, a partner at Honigman.
The FDA has granted breakthrough designation to a handful of medtech tools using generative AI, such as as two separate features being developed by Aidoc and a Radiology Partners subsidiary to interpret chest X-rays and draft radiology reports, and another tool made by Modella AI that analyses pathology images and clinical data.
The FDA has also included generative AI-based devices in a new digital health pilot, called Technology-Enabled Meaningful Patient Outcomes, or TEMPO. It allows companies to collect real-world data while being exempted from premarket authorization requirements. For example, one of the participants, Limbic, provides cognitive behavioral therapy through phone calls with an AI voice agent.
Some types of AI features don’t currently fall under FDA review. Diabetes tech firm Dexcom added a generative AI feature to its over-the-counter glucose sensors in 2024 that analyzes users’ data to provide personalized wellness recommendations. Dexcom said the feature did not require a premarket submission.
PathChat, a tool that uses generative AI to help pathologists diagnose complex cases, received the Food and Drug Administration’s breakthrough device designation. The device is not yet FDA authorized.
Courtesy of Modella AI
2. What’s new in the FDA’s discussion paper?
The FDA paper builds on existing concepts while acknowledging that generative AI doesn’t fit neatly into the current device framework, attorneys told MedTech Dive.
Kayla Cristales is an attorney with Haynes Boone.
Permission granted by Hanyes Boone
Kayla Cristales, an attorney with Haynes Boone, sees the paper as a step in the right direction, adding that it was an “open acknowledgement that this is a totally different beast.”
The agency introduces the idea of a “competency-based assessment,” taking inspiration from how physicians are evaluated using exams, supervision and public reporting. Generative AI devices may have a range of possible inputs and outputs that are too large to test, the FDA said in the paper, so it is looking at benchmarking as part of premarket review.
The idea is that with generative AI models, “we’re moving past a place where you can really see completely under the hood,” Levy Friedman said.
Another challenge with evaluating this type of technology is that while some companies might develop their own models from scratch, others might build off of existing, third-party models. This could make it difficult for the device developers and the FDA to know how the underlying model works. As one solution, the FDA has proposed the concept of foundation model device master files, where developers of these underlying models or platforms could voluntarily submit information to the FDA that the agency would hold confidentially.
Brigid Bondoc, a partner with Morrison Foerster, said the concept is similar to one the FDA uses for drugs, where a manufacturer of a gel capsule might not want to share proprietary information about the ingredients or how it’s made, but the FDA would need to know this when assessing a drug that uses this capsule. In essence, this would be a way for the regulator to get information about the foundation models that might be going into a device, Bondoc said.
“I think what they may end up doing is heavily relying on the companies themselves to provide updates.”
Kayla Cristales
Attorney with Haynes Boone
Finally, the FDA emphasized the importance of monitoring generative AI devices after they go to market, adding that the agency is considering whether it may be appropriate to accept greater premarket uncertainty with more reliance on postmarket monitoring.
“I think they’re going to have to come up with some type of a way to stay on top of and to monitor the evolution of these devices,” Cristales said. “I think what they may end up doing is heavily relying on the companies themselves to provide updates.”
3. Do medtech firms want guidance on generative AI?
Some device developers are forging ahead with generative AI, while others are waiting for more guidance, said Ketryx CEO Erez Kaminski, who works with medical device firms using AI for compliance. The approach depends on the size of the company and its appetite for risk.
Suzanne Levy Friedman is a partner at Honigman.
Permission granted by Honigman
Honigman’s Levy Friedman said most generative AI tools on the market today are “dancing around the FDA-regulated space.” Companies are focusing on areas that don’t trigger device regulations, such as administrative tools to streamline workflows or follow up on established guidelines.
“A lot of people are just focusing on that space until [the] FDA clarifies what is actually going to be needed,” Levy Friedman said. “No one really wants to be the guinea pig.”
In contrast, companies that have already accepted that they are going to be regulated by the FDA and want to use generative AI are looking for more clarity, she added.
“A lot of people are just focusing on that space until [the] FDA clarifies what is actually going to be needed. No one really wants to be the guinea pig.”
Suzanne Levy Friedman
Partner at Honigman
Aidoc CEO Elad Walach said the company views generative AI-specific guidance as helpful, but not critical. Sharif Vakili, CEO of UpDoc, a company using agentic AI for patient follow up between visits, praised the discussion paper and said regulatory leadership from the FDA is the most important. Without it, companies face either a patchwork of different regulations or bad actors and reactionary responses, Vakili said.
One question looming over the medtech sector is whether the FDA will take a deregulatory approach, as the Trump administration has called for faster AI adoption and removing regulations that slow the development and deployment of AI. While the FDA has loosened some restrictions on what digital tools can fall under its wellness exemption, the way the FDA regulates AI and software tools that are considered medical devices has not changed, Levy Friedman said.
Aidoc’s aiOS platform is used by Asklepios Group, a private hospital operator in Germany, to analyze CT and x-ray images. Aidoc has received the FDA’s breakthrough designation for a separate tool to interpret chest X-rays and draft radiology reports, but the tool has not been authorized by regulators and is not yet available on the market.
Courtesy of Aidoc
4. What do patient and clinician stakeholders want to see?
Stakeholders outside of the device industry said they support more postmarket monitoring of generative AI-enabled devices, and they have called for broader FDA oversight of chatbots and other commercial tools.
“I’m happy to see FDA taking a stab at regulation of generative AI, even if this is still a very early stage,” said Kellie Owens, an assistant professor of medical ethics at NYU Grossman School of Medicine. “I’m particularly happy to see a focus on post-market monitoring, since I think monitoring standards are still looser than they should be.”
Owens said academic medical centers such as NYU and their peers do a lot of internal scrutiny of AI tools to ensure they’re safe and effective. But there are still many questions about whose responsibility it is to ensure the systems’ performance doesn’t degrade over time. Owens sees it as the vendors’ responsibility to put out products that continue to work well, but from her experience, “that’s not always happening.” Healthcare institutions, meanwhile, should vet these AI tools and use them responsibly, Owens said.
Another area of concern for stakeholders is commercial chatbots, which don’t currently fall under the FDA’s purview. Owens said she sees this as one of the more dangerous applications of generative AI for health purposes.
Luis Gil Abinader is the policy director for Generation Patient.
Permission granted by Generation Patient
Luis Gil Abinader, policy director for Generation Patient, a nonprofit focused on young adults with chronic medical conditions, said he would like to see greater oversight of AI companions and chatbots, regardless of whether they’re marketed for health purposes.
In an FDA digital health advisory committee meeting last year, experts raised concerns about people turning to unregulated chatbots for mental health support. Nearly a quarter of large language model users reported using a LLM for mental health, according to a study published in JMIR Mental Health, and some commercial chatbots falsely claim to be therapists, William Agnew, a postdoctoral fellow studying AI ethics at Carnegie Mellon University, said in the meeting.
While the HHS did not explicitly answer whether the FDA is working on specific policy related to generative AI in mental health, the device center has listed guidance on clinical evidence considerations for digital mental health devices on its “under construction” list for the current fiscal year.
