The Rise of Multimodal AI in Medical Research

Key takeaways

• Combining imaging, text, and genomic data in one model can lift diagnostic accuracy by several percentage points over single-modality approaches, according to a large 2024 review of published studies.
• Foundation models, pretrained on broad datasets, are cutting the amount of new data researchers need to collect before testing a multimodal approach.
• The biggest current bottleneck is data: well-labeled multimodal datasets remain scarce, and privacy rules limit how much institutions can share.
• Pairing pathology slides with genomic and other omics data is opening biomarker discovery paths that a single dataset alone couldn't reveal.
• Cureus's Collections feature is one place where guest editors are already curating this kind of cross-disciplinary work for readers.

Three. That's how many studies in one major review combined more than one type of medical data, like imaging with genomics or tissue slides with patient records, in 2018. By 2024, that same scoping review counted 150 in a single year, a fiftyfold jump in six years.

The field has a name for this growth: multimodal AI.

Instead of training a separate model for each data type, researchers are increasingly building systems that read imaging, text, lab values, and genomic data together, closer to how a physician draws on several sources of evidence at once. This piece looks at where that shift is coming from, what it's making possible across imaging, pathology, genomics, and clinical data integration, and where the research still runs into trouble.

What makes a model “multimodal”?

A multimodal model is trained on two or more kinds of data, like an imaging study alongside a pathology slide and a block of clinical text. The multi-source input lets it pick up on relationships a single-input model would never see.

Many of today's systems start from a foundation model (a large network pretrained on broad data) and then get fine-tuned for a narrower task in medical imaging or genomics. The pretraining head start is a big part of why multimodal medical AI has been able to move faster than earlier generations of purpose-built algorithms.

Where the research is happening: imaging, pathology, genomics, and clinical data

Imaging

Imaging is where multimodal approaches are furthest along. Pairing a radiology scan with its report, prior studies, or lab values gives a model context that pixels alone can't provide.

The same 2024 review found multimodal imaging models beat single-modality counterparts by roughly six percentage points in area under the curve (AUC), a meaningful gain for anyone benchmarking diagnostic tools. Vision-language systems that pair scans with free-text reports also draft radiology reports for review; earlier models handled that task poorly.

Pathology

Digital pathology has followed a similar arc. For example, whole-slide images paired with molecular or genomic data let researchers connect what a tumor looks like under the microscope with what's happening at the cellular level.

Foundation models trained on large slide libraries are now routinely fine-tuned on smaller, specialty-specific datasets, a pattern that keeps showing up across biomedical AI research. Whether a model trained on one hospital's slides holds up on another's is now a bigger open question than how accurate it looks on its own training data.

Genomics

Genomic data adds a different axis. Systems that combine variant calls, gene expression, and clinical history help researchers spot biomarkers that stay hidden in any single dataset, particularly in oncology, where treatment response can hinge on a mutation profile that only becomes clear once genomic and clinical data sit side by side.

Much of today's multimodal AI in healthcare research is concentrated right here, at the intersection of a patient's biology and their chart.

Clinical data integration

Structured and unstructured data from electronic health records (EHRs) get less attention than imaging or genomics, though they may carry just as much signal. Large language models (LLMs) can now pull usable information out of clinical notes and lab reports at a scale that manual chart review never allowed.

Fuse that with imaging or genomic data, and researchers get a fuller picture of a patient's course, which is what clinical data integration research is built to test.

Why researchers are drawn to it

Radiology and pathology teams that once built entirely separate models increasingly share architectures, a direct effect of multimodal approaches maturing across specialties. A single model that reads imaging, text, and lab data together can support hypothesis generation in fields that used to work in isolation from each other. Readers can already browse this kind of cross-disciplinary work on Cureus's own specialty homepages.

Foundation models also lower the barrier to entry. A research group without the resources to train a model from scratch can fine-tune an existing one and still get competitive results, which is opening the door for smaller teams in fields like oncology and neurology to run studies that once needed far bigger budgets. Some healthcare AI research groups now share pretrained checkpoints outright, a sign that the field has matured from scattered pilot projects into something closer to a shared discipline.

Where the research still runs into trouble

A 2024 systematic review of foundation models for medical imaging documented several of the obstacles researchers keep running into:

• Data scarcity. Large, well-labeled multimodal datasets are far rarer than single-modality ones, and privacy rules limit how much clinical data institutions can share.

• Modal misalignment. Imaging, text, and genomic data operate on different scales and timelines, and lining them up accurately takes infrastructure most research groups don't have.

• Missing benchmarks. Without standardized evaluation sets, comparing one multimodal model against another (or reproducing published results) is hard.

• The black box problem. Foundation models are notoriously difficult to interpret, which complicates explaining why a model reached a given output, a problem that matters for research validity as much as clinical trust.

• Cost. Training or fine-tuning large multimodal models is expensive, which can put this kind of work out of reach for smaller, under-resourced labs.

What's next

Smaller, more efficient models built for a specific multimodal task chip away at the assumption that bigger models are always better, which is welcome news for labs without huge budgets. Self-supervised pretraining also cuts down on how much labeled data a study needs to get off the ground.

As more institutions publish checkpoints and shared benchmarks, replicating and building on existing work should get easier, and that alone could widen who gets to do this kind of AI medical research.

Publish your research with Cureus

If your work touches imaging, pathology, genomics, or clinical data integration, Cureus wants to see it. As an open access journal, Cureus makes published research available to readers everywhere at no cost. Manuscripts typically get an initial decision within one to two days and go through single-blind peer review, with a median time to publish of 29 days.

Review the publishing process and peer review guidelines whenever you're ready to submit.

Frequently asked questions

Q: What makes an AI model “multimodal” in medical research?

A: It's trained on two or more distinct data types, like imaging, text, and genomic data, rather than just one. Combining those inputs lets the model learn relationships a single-modality system wouldn't pick up on its own.

Q: How is multimodal AI different from a foundation model?

A: A foundation model is a large model pretrained on broad data and later fine-tuned for a specific task. Plenty of multimodal medical AI systems are built on top of a foundation model, but the two aren't the same thing: a foundation model doesn't have to be multimodal, and a multimodal model doesn't have to start from a foundation model.

Q: What's the biggest hurdle facing this research right now?

A: Data. Large, well-labeled multimodal datasets are scarce, and privacy rules restrict how much clinical data institutions can share for research.

Q: Is multimodal AI ready for clinical use in healthcare?

A: Not broadly. Most published research in this space is still in the research and validation stage rather than routine clinical deployment, and reviewers keep flagging generalizability and interpretability as open problems.