Dive Brief:
- The Food and Drug Administration is seeking public feedback on how to measure and evaluate the real-world performance of artificial intelligence-enabled medical devices.
- In the consultation, which began Tuesday, the FDA has posed six sets of questions about ways to perform ongoing, systematic performance monitoring to see how AI behaves in clinical settings.
- With the rate of AI submissions accelerating, the FDA is wary of the potential for “data drift” to cause devices to perform worse in the real world than in tests run to support authorization.
Dive Insight:
The FDA’s call for feedback is informed by evidence that the performance of AI-enabled medical devices can change over time. As the agency explained in its request for comment, factors such as changes in clinical practice, patient demographics, data inputs and healthcare infrastructure can affect devices. User behavior, workflow integration and changes to clinical guidelines may also impact performance.
Commonly called data drift, the tendency for AI-enabled devices to change over time can degrade performance, cause bias or reduce reliability. The phenomenon sets AI-enabled devices apart from traditional products, the performance of which should remain constant.
In a January draft guidance, the FDA recommended that developers have a post-market monitoring plan in place for AI-enabled devices because their performance can degrade over time.
Regulatory processes designed around traditional devices can help establish a baseline for performance but fail to predict how AI-enabled products will behave in dynamic, real-world environments. The FDA is seeking feedback on how to ensure AI-enabled devices stay safe and effective throughout their lifecycles, including by adopting approaches to detect, assess and mitigate performance changes over time.
FDA officials are yet to propose or implement changes to the evaluation of AI-enabled devices. Rather, the FDA is framing the consultation as a chance for people outside the agency to share insights and a way to spur a broader discussion about the ongoing evaluation of AI in the real world.
The agency has asked questions about performance metrics, real-world evaluation methods, postmarket data, triggers for additional assessments, interactions between humans and AI and best practices. Across the questions, the FDA is requesting information on current, practical ways to measure and evaluate the performance of AI-enabled medical devices in the field.
Through the consultation, the FDA is aiming to find strategies for identifying and managing performance drift, such as detecting changes in input and output. The agency is particularly interested in methods that are currently deployed at scale in clinical environments and supported by real-world evidence.
The consultation closes on Dec. 1.
