Verily, NVIDIA and NIH Build First Multimodal Foundation Model Integrating EHR and Genomic Data to Improve Type 2 Diabetes Risk Prediction
核心洞察
Verily Health (搜索), collaborating with NVIDIA (搜索) and the NIH All of Us Research Program, developed the first multimodal foundation model integrating electronic health record and genomic data.
Adding genetic risk scores to EHR data substantially improved the model's ability to identify people at high risk for Type 2 diabetes (搜索), boosting accuracy while reducing false alerts.
The model projected disease risk over clinically meaningful five-to-ten-year timeframes and achieved a threefold increase in pre-training efficiency using NVIDIA (搜索) NeMo AutoModel.
Verily Health (搜索), in collaboration with NVIDIA (搜索) and the National Institutes of Health (NIH) All of Us Research Program, has developed the first multimodal foundation model integrating electronic health record (EHR) and genomic data. The work offers an early example of how multimodal AI can bring together diverse sources of health data to advance precision medicine, and represents a significant step beyond models trained on clinical data alone.
Connecting Multiple Dimensions of Human Health
While models trained on clinical data alone remain extremely valuable, integrating additional modalities such as imaging and genomics marks an important advance. "Electronic health records capture an important piece of a patient's story, but they're only one chapter," explained Jonathan Amar, senior manager of Verily Data Science. "To truly understand disease risk, AI needs to learn from multiple dimensions of human health, including both our biology at birth and our lived clinical experience."
According to Amar, combining EHR data with genomic information is particularly promising because it can reveal inherited disease risk long before symptoms appear. Connecting the "nature" and "nurture" dimensions of health data, and contextualizing how they influence one another, has long been one of the biggest challenges for healthcare AI.
Putting the Multimodal Foundation Model into Action
Foundation models are particularly well-suited for integrating diverse health data modalities to identify complex disease risk patterns, because they can learn broad patterns from large, diverse datasets before being adapted for specific prediction tasks. As larger multimodal datasets become available, foundation models create new opportunities for AI to uncover relationships that would be difficult to detect from a single source of health data alone.
The approach was not without its challenges. "There were several factors in play — starting with the complexity of integrating static genetic data with dynamic clinical histories. They're fundamentally different types of data with very different structures. You also have the computational demands of processing both together that most CPU-based workflows can't handle," Amar noted.
Access to genomic data is typically unavailable for many patients, adding another layer of complexity. Nation-level initiatives like the All of Us Research Program, which collects multimodal data including genomic data, open new doors for developing models that capture a more nuanced, precise understanding of health.
"We were able to integrate genetic risk scores (measurements that predict an individual's likelihood of developing a disease based on their DNA) with EHR data in a single foundation model, powered by NVIDIA (搜索) GPUs — with exciting results," Amar explained.
Improved Type 2 Diabetes Risk Prediction
When genetic risk information was added to the model, its ability to identify people at high risk for Type 2 diabetes (搜索) improved substantially. By accurately spotting more true cases while reducing incorrect alerts, the model demonstrated a significant boost in performance. The model also showed it could project risk over clinically meaningful timeframes of five to ten years, rather than focusing only on near-term predictions.
On the computational side, the model achieved a threefold increase in pre-training efficiency by using the NVIDIA (搜索) NeMo AutoModel compared to Hugging Face Accelerate on identical H100 GPUs. This translated to faster time to insight and faster iteration in a complex research environment.
Forecast™ 1.0 and Future Directions
This pioneering model is now available to all researchers via GitHub as Forecast™ 1.0. Forecast 1.0 gives global research teams hands-on access to accelerate their own disease risk discovery pipelines within a secure Trusted Research Environment (TRE).
In this case, combining EHR and genomic data improved the precision of Type 2 diabetes (搜索) prediction, but the implications extend far beyond a single disease or even genomics. This work provides an early blueprint for how multimodal foundation models could integrate many different dimensions of human health — including wearables, clinical notes, medical imaging and patient-reported outcomes — to advance precision medicine.
To realize this future, Verily continues to research and develop improvements in its predictive and generative models, driven by the continuous expansion of relevant, multidimensional datasets — including rich, unstructured clinician notes — with a vision to evolve Forecast™ models over time. By continuously enriching the datasets the model learns from, Forecast will capture an increasingly high-resolution, dynamic picture of patient health.
"By bringing together two types of typically mismatched data, we demonstrated how multimodal AI can improve disease prediction while laying the groundwork for a future that incorporates many more health data modalities," emphasized Amar. "This provides a preview of what's possible in areas such as pharmaceutical biomarker discovery, diagnostic risk stratification and health system precision medicine programs."