In today’s pharmaceutical industry, real-world evidence (RWE) offers significant potential across all phases of the product life cycle, from trial design to product launch, from pricing and competitive reviews to evaluating the effects of switching medications.
As RWE becomes a key tool for life sciences organizations to shape clinical trial strategies, advance treatment development, and meet regulatory requirements, the quality of the underlying data and trust in how it was generated is paramount.
While just about everyone claims to use artificial intelligence (AI) to extract and curate real-world data (RWD) from healthcare systems, how do you know what to look for, who to trust, or what questions to ask when seeking to generate RWE?
In other words, what’s actually happening amidst the curation workflow to ensure that the data is ultimately usable for your purpose?
Powerful but not infalliable
Real-world data comprises a wealth of vital information. Yet an estimated 80% of the healthcare data in electronic healthcare records (EHRs) is unstructured, meaning it isn’t in a format that can be readily analyzed or queried.
The details found in clinical notes, such as patient symptoms and experience, physical exam findings, diagnostic test results, and clinical decision-making on assessment and plan, require AI-driven techniques to unlock at scale.
Similarly, important patient care information found in unstructured text, such as operative notes, radiology reports, pathology reports, and send-out test results such as genetics, must be transformed or curated into a structured format in order to be accessed for analysis and true insights.
AI-tools become essential in the effort to create structured variables from unstructured data. Yet the use of AI is not a panacea. The successful application of AI-driven techniques to analyze RWD is dependent upon a broad-based collaboration among researchers, medical experts, data scientists, and other stakeholders.
To enable meaningful insight through AI advances, it is critical to combine well-defined structured data and high-quality unstructured data in a consistent and reliable way. A key issue becomes how to build a platform to reliably transform unstructured data into a research-ready resource at scale.
Here are five topics to consider before embracing AI-curated RWD.
Are you starting with high-quality data?
Even the most sophisticated model or AI-driven analysis applied to low-quality data will yield unusable results.
But what makes for quality data?
“You start with the sourcing,” says Tim Hoctor, formerly vice president of Global Life Science Services at Elsevier, and member of the Board of Advisors, Pistoia Alliance. “Is the sourcing accurate? Is it consistent? Is it timely?”
Quality data is fit for its intended purpose and meets specified criteria. Data quality also hinges on its uniqueness, validity, and how well it conforms to established standards and requirements.
“High-quality data is really the foundation of your approach,” says Scott Schliebner, vice president and global head, Drug Development Consulting, Novotech. “You have to start with decent data or you’ll really never end up with anything helpful or productive.”
“There are a lot of data sources that we can leverage to guide and de-risk a clinical trial upfront,” says Schliebner. “Quality data can offer insights into the natural history of the disease, standard of care, variations among regions, and patient populations as well.”
Which AI techniques are you using?
Organizing unstructured data and curating fit for purpose, real-world data requires the appropriate methodology, grounded in scientific principles to ensure data quality and the credibility of the evidence that is generated.
Common AI-techniques applied in healthcare include machine learning (ML), natural language processing (NLP), large language models (LLM), and generative AI.
“They all sort of build on one another,” says Mariah Baltezegar, a vice president and general manager for Thermo Fisher Scientific. “Machine learning being the foundation, used to train all of the tools. NLP, being a specific area of machine learning focused on language. LLMs being a type of NLP that use ML, and are trained on huge amounts of text.”
Machine learning is a way for computers to learn from the data instead of being programmed with specific rules, she says. “It’s like giving a superpower to researchers that helps them quickly find patterns or trends from large amounts of health data, things that would take forever to sort through manually. For example, machine learning can be effectively applied to find patients who are good fit for certain studies.”
One important thing to consider, Baltezegar points out, is that none of these techniques replace people. A scalable and clinically based data quality assessment process that includes clinician input and artificial intelligence is essential to bring both structure and meaning to RWD at scale.
For example, “Generative AI could be used to draft a protocol based on inputs and a human in the loop, as we say in AI-speak, would review and refine it.”
How is your AI process developed and tested?
Extensive measures must be taken to ensure that AI-processes deliver high-quality, clinically validated data. This starts with a clear and explicit understanding of what you are trying to solve for and the success criteria.
Supervised machine-learning models and AI processes should begin with a very clear, high-quality labeled dataset that has been reviewed by humans; validation by subject matter experts should continue throughout the life cycle.
After models are developed, the generated output must be appropriately tested relative to distinct data validation datasets that have been reviewed by subject matter experts and clinicians. This ensures that the machine output meets or exceeds expectations. Additionally, organizations should engage in continuous model refinement to prevent bias and maintain accuracy.
An example of how AI processes can be applied to advance our understanding of a disease can be found with geographic atrophy (GA), an advanced form of dry age-related macular degeneration (AMD).
Ophthalmology uses heavily standardized use of images to track key variables such as lesion size, location and growth rate, total number of lesions, and other criteria. After a robust quality assessment, control and curation process, Verana Health has validated tens of thousands of high-quality images for more than 2,000 GA patients which can be utilized to train AI models to identify GA disease progression at scale.
Do you have deep, subject matter expertise?
Clinical involvement is critical at every step of AI-driven research processes.
It takes a team of subject matter expertise (SME), including clinicians, nurses, clinical informaticians, data scientists, epidemiologists, biostatisticians, and engineers working together to effectively curate and standardize data while retaining its original clinical context.
“I would really push that you need that SME-understanding, to analyze and gauge the output of the model, to define the objective of the model, and to make sure that the data you’re using to build your models is appropriate to the objective you want on the other end,” says Hoctor.
When evaluating an AI-driven RWD effort, be sure to inquire about the expertise involved:
- Can you share the credentials and involvement of your medical team?
- What role does your medical team play in your curation process?
- Who is doing the in-depth chart reviews?
- What experience does your staff have in implementing and deploying AI/ML?
- Who oversees maintenance and monitoring of your deployed models?
“You have to have the medical and scientific knowledge from the very beginning, to understand and define the models that you’re building and the objectives that you have,” says Hoctor.
How will you gauge impact?
“One of the ways I think we can gauge impact is by looking at how AI and RWE is improving data completeness and accuracy,” says Baltezegar. “How does that compare to previous outcomes?
“Do we have answers for why there may be such variation five years, or 10 years later, from what we saw before?” she says. “Is it true or is it an artifact based on which methodology we’re using? Engaging that impact is going to be key moving forward.”
Baltezegar also raises a cautionary note on growing use and adoption of AI related to the Hawthorne effect, which describes the phenomenon where individuals modify or improve an aspect of their behavior in response to their awareness of being observed: Will patients be more reserved in the future talking to their physicians if they know this data is being utilized, albeit in an anonymized way, to drive potential treatment insights?
Capitalizing on AI
AI has the potential to revolutionize clinical research by enabling analysis of real-world data, accelerating trial timelines, and improving patient outcomes. But it must be applied with thoughtfulness, as part of a comprehensive strategy.
Knowing what to look for and how to evaluate AI-driven processing of RWD is a critical first-step to effectively employing the technology.
In the end, volumes of RWD and powerful computing models are not enough to save lives. It’s the insight generated from the careful, deliberate application that is of value. To capitalize upon this potential, we must implement AI-driven analysis in the correct way.
About Aracelis Torres, PhD, MPH
Aracelis Torres, PhD, MPH, is Senior Vice President of Data & Science at Verana Health, where she oversees the Quantitative Sciences team working on rigorous methodology to generate sound scientific evidence from real-world data, as well as the commercial services and operations team that manage client deliverables. Dr. Torres is an epidemiologist with over ten years of academic and industry experience. Prior to joining Verana Health, she was a Director of Quantitative Sciences at Flatiron Health, where her work focused on translation of real-world oncology data to generate evidence.
