A year ago, integrating AI-powered clinical workflow tools could take months of review and planning across disparate teams within a pharmaceutical company’s IT, security, and risk teams. Today, with the proliferation of tools that have less than a month of onboarding and can provide immediate value to clinical teams, the way risk is approached has changed. For better or worse, AI capabilities are outpacing the security programs meant to govern them, and many companies are simply assuming risk that they have not quantified or do not fully understand.
I have built security programs across half a dozen regulated companies, and the bottom line is always the same: you can’t eliminate all risk, but you will be in a much better place if you understand the risk and manage it.
Most conversations about AI in pharma focus on what it can do: automate tasks, save time, accelerate development. Far fewer discussions focus on how such systems can be deployed securely. In regulated environments, leveraging AI tools represents a point of great risk for organizations. If pharma organizations understand the risk, they can take action to reduce it while also realizing the immense value AI will bring to the industry. Simply stated, ignoring risk should not be an option.
AI-powered tools can directly affect patient safety and trial outcomes in clinical use cases; use cases such as protocol design, endpoint selection, and data analysis all have a regulatory impact. Unfortunately, vendors are rushing to ship products built over a weekend, creating immense security and compliance risk that will become costly to remediate downstream. By the time this risk is realized and has a negative impact on an organization, the tools are already embedded and an investment has already been made.
In April, the FDA gave out its first warning letter for the improper usage of AI. The result? The FDA ordered drug production halted and put a recall on the table. The question is not whether a submission for a new drug will be denied but when. What looks like speed early on often turns into downstream liability when that risk becomes a reality.
Security questions pharma should be asking
Pharma organizations need to perform rigorous security reviews, streamline the risk evaluation process, and define their organizational risk tolerance before deploying AI solutions. That due diligence comes down to a pair of questions that many procurement and evaluation processes are not asking today.
Has the system been assessed by an independent party for its security and compliance posture?
A vendor pointing towards internal benchmarks does not adequately address security in practice and risks the organization’s posture being nothing more than security theater. A credible evaluation requires an independent third party who evaluates the system against a documented standard, defines expected secure and compliant operation for a given use case, tests against those expectations, and identifies conditions that fall outside of them.
An attestation of a trusted AI governance standard such as ISO 42001 can help confirm that a vendor is managing risk appropriately, but the story does not end there. Pharma organizations should, at a minimum, spot-check vendors to ensure they have the correct mechanisms in place to control the risks related to AI.
What level of risk has been defined for each deployment?
Different workflows carry different levels of exposure. A tool with access to patient health information, addresses, and contact details operates at a fundamentally different risk level than one for ideating on the design of a clinical trial. The problem lies when a vendor’s risk appetite doesn’t match your organization’s internal culture or standards.
If you have clinical trial design software that doesn’t have the proper controls in place or isn’t designed to keep risk within acceptable bounds, there’s a chance that your organization will be the next one with a warning letter from the FDA or in the news as the latest breach. Before any deployment, teams should understand their baseline level of risk. Companies that haven’t defined their risk threshold often minimize the threat or actualization of incidents after the fact, but once an incident happens, there’s no putting the genie back in the bottle.
Security defines what comes next
It’s tempting to respond to these risks by turning away from AI entirely, a path that most pharma companies initially took. But it’s become increasingly clear that AI has real potential to bring incredible speed and efficiency to drug development and, ultimately, to patients who will benefit once the drugs go to market. For that potential to become progress, pharma needs the proper risk management and security evaluations in place.
A prototype built in a few days is not a secure system and a compelling demo doesn’t replace the necessary rigor of managing AI risk.Even established tools and market leaders deserve scrutiny, so imagine the scrutiny needed for a vendor with no security or governance posture.
The gap between a product built with a solid foundation of security and governance and one without is where many AI implementations in regulated industries will fail. Unfortunately, failure tends to only surface when the cost is at its greatest.
About Alexander Neff
Alex Neff leads security, IT, and compliance at Faro, an AI-native life sciences SaaS company. He has spent 15+ years building security programs at the cutting edge of regulated tech.
