The integrity of medical research is under siege. Last month, Med-Gemini’s “hallucination” of a part of the brain made headlines, but according to Dr. Mike Rossi, Vice President of Translational Science and Multi-modal Real World Evidence Solutions at ConcertAI, that incident is merely a symptom of a larger, systemic problem: up to 10% of scientific literature is now being fabricated by AI.
Drawing on his decades of experience integrating genomic science into oncology at institutions like Emory and Icahn, Dr. Rossi warns that the crisis extends from the ethics of synthetic data in pharma to the erosion of trust in the peer-review process itself. We speak with Dr. Rossi about the immediate dangers facing clinicians and researchers, the ethical imperative to ground synthetic data in reality, and the critical steps academic medical centers must take to secure the future of evidence-based medicine.
A recent report from The Verge noted that Med-Gemini hallucinated a part of the brain. Given your claim that up to 10% of scientific literature is fabricated by AI, do you believe incidents like this are simply errors or a symptom of a larger, systemic problem in scientific research and publishing?
Dr. Mike Rossi, VP of Transitional Science and Multi-modal Real World Evidence Solutions at ConcertAI: Yes, in fact, my estimate may be conservative. These agents are designed to be interactive and flattering. In every academic field, there are always groups trying to shortcut or cheat the system. But now, the use is much more widely spread, which leads me to believe that it is a systemic problem.
Over the next decade, artificial intelligence will continue to be pervasive in healthcare. When any new technology becomes available it is always misused in the beginning. This emphasizes the fact that the scientific community must stand up and ensure that we are maintaining specific domain expertise.
How does the rise of AI-fabricated images and data impact the credibility of medical journals and the peer-review process, especially at lower-tier publications?
It will jeopardize the credibility of these lower-tier journals. Some images and figures are now incredibly difficult to identify whether or not they were generated by AI. This further emphasizes the need for credibility at these journals through scientific experts.
This also calls attention to the peer review process, which relies on having at least three experts in the field to review and approve manuscripts. The problem with these lower tier journals is it is becoming harder to find peer reviewers to examine these papers. With the influx of falsified data, this issue will likely become even more pervasive.
You’ve spent decades working on oncology research and diagnostics. How does this erosion of trust in scientific literature affect the development of new therapies and the ability of clinicians to adopt them?
The challenge with falsifying data is that once it has entered the public domain, it is very hard to retract. The scientific community learned this with vaccines and now face a very steep hill in educating the public.
Scientists are tasked with the role of being educators, constantly reinforcing the value of information. One of the challenges of clinician training is the balance of training them as scientists and as critical thinkers. Not every clinician can maintain the breadth of domain expertise needed to understand every field.
These hallucinations and falsifications are usually clear to an expert, where the gap in knowledge exists. The fear is that these language learning models will continue to learn these gaps and get better at masking what they don’t know. For now, clinicians will need to rely on their common sense to find conclusions.
You’ve highlighted the dangers of using AI-generated synthetic data to represent populations that researchers can’t reach. What are the ethical considerations here, and what steps should the pharmaceutical industry be taking to ensure the reliability of this data?
Synthetic data has tremendous potential, but needs to be grounded in the real world. It is critical that any synthetic data has some linkage to truth. Currently, the synthetic data sets that have been generated must be supervised to ensure they are still ethical and reliable.
How can clinicians and researchers distinguish between a hallucination—an outright fabrication like the “basilar ganglia”—and a “reasonable inference” made by an AI model, and why is this distinction so critical for patient safety?
It is so important in this technological age that we do not detract from standards of medical care. When considering new forms of technology, access and direct contact with a patient are so important to medical diagnosis, even virtual appointments or telehealth.
There is a reality of patient safety that is rooted in physical contact that these models cannot provide. Take for example, attempting to diagnose Lyme Disease while working with a patient, these models may likely overlook physical symptoms such as the bullseye rash or fever. To avoid these mistakes and hallucinations, medical professionals need to maintain contact with their patients.
What role do you see for organizations like yours, ConcertAI, in helping to establish new standards for data integrity and reliability in an AI-driven research environment?
The purpose of our organization is to assist the patient. As translational scientists, particularly in oncology, we specialize in improving care by providing pharmaceutical companies with a deeper understanding of their patients. Our goal is to treat cancer, and our strategy is to approach it as a data problem, leveraging technology to continuously improve the standard of care for patients.
You have a deep background working with leading institutions like Emory and Icahn. What advice would you give to academic medical centers today on how to build a research infrastructure that can responsibly and effectively leverage AI?
Unfortunately, academic medicine is still incredibly siloed, especially when it comes to data. Administrators feel that data is so valuable that it must be guarded. But, to move the needle forward in patient care, integration across the network is required. In an ideal world, this could be consolidated by a federated system, where each major academic medical center maintained a robust data architecture, including radiology, pathology, genomics, clinical history, health economics, social determinants of health, etc.
This ecosystem would establish a consortium of data that would transform care for patients, ideally expanding beyond academia to community practice and remote health systems. This approach would democratize care and increase efficiency for patients. My belief is that AI will be critical to this growth trajectory, as it will likely run on top of all of that data to understand nuances that we have historically overlooked.
Any final thoughts?
Dr. Mike Rossi: These ideas aren’t just applicable to oncology, they are applicable to all types of science. A lot of this conversation is how we can educate the public to be supportive of science and to understand how we interact with information.
About Dr. Mike Rossi
Dr. Mike Rossi currently serves as Vice President, Translational Science and Multi-Modal RWE Solutions at ConcertAI. Mike’s primary role at ConcertAI is to facilitate access and insights into complex real-world oncology datasets to support translational and precision medicine oncology programs in pharma and biotechs. Mike is an expert in cancer genomics and oncology diagnostics and spent much of his career developing innovative assays to characterize solid tumors and hematological malignancies.
Previously, he served as Division Head of Molecular Oncology of Solid Tumors at Sema4 and was an Associate Professor in the Department of Genetics and Genomic Sciences at the Icahn School of Medicine at Mount Sinai. Prior to those roles he was an Assistant Professor at Emory University in the Departments of Radiation Oncology and Pathology. His academic credentials include a Ph.D. in Genetics and Developmental Biology, post-doctoral work in Cancer Genetics and American Board of Medical Genetics certification in Clinical Molecular and Cytogenetics. Dr. Rossi has held faculty positions at Emory University and significant academic roles, including Adjunct Associate Professor at the Icahn School of Medicine and Director of Cancer Genomics at Emory University. In addition to his leadership roles, Mike has been a contributing author to over 65 peer-reviewed manuscripts and served on the editorial board for the Cancer Genetics Journal.
