An artificial intelligence (AI) model has been developed to predict the disease course of sepsis patients, potentially improving the selection of participants for clinical trials. The study, published in the Journal of Medical Internet Research, used data from the Medical Information Mart for Intensive Care Database-IV (MIMIC-IV) and the eICU Collaborative Research Database (eICU-CRD) to train and validate the model. The AI aims to identify patients likely to have a prolonged illness, ensuring they remain in the trial long enough to benefit from the therapy being tested.
AI Model Development and Validation
The AI model was trained to predict three sepsis trajectories: rapid death (death within 48 hours), recovery (liberation from vasopressor support within 48 hours), and persistent illness (ongoing need for vasopressor support after 48 hours). The model incorporated both static and time-varying variables extracted within the first 24 hours of sepsis identification. A gradient boosting machine (GBM) architecture was found to have the highest consistent performance, achieving a mean area under the receiver operating characteristic curve (AUROC) of 0.807 (SD 0.010) for predicting persistent illness in internal validation. In external validation using the eICU-CRD, the model achieved AUROCs of 0.878 for rapid death, 0.764 for recovery, and 0.696 for persistent illness.
Addressing Model Uncertainty with Conformal Prediction
To address the risk of automation bias and evaluate model uncertainty, the researchers implemented a conformal prediction (CP) framework. This framework allowed the model to generate confidence measures, enabling the monitoring of predictions and identification of uncertain outcomes based on adjustable confidence levels for human review. When the model was evaluated using a mixed confidence approach (85% for recovery, 75% for rapid death and persistent ill), the model with CP reduced overall prediction errors by 27.6% in the internal validation cohort and 30.7% in the external validation cohort.
Clinical Application and Web-Based Calculator
A web-based calculator was developed to assist clinicians in understanding the AI model's outputs. This calculator, currently deployed at Zhongda Hospital, allows clinicians to input patient data and view the risk score predicted by the model. Clinicians can also adjust the confidence level to obtain a reliable prediction, with uncertain predictions flagged for human review. According to the study, this approach could streamline patient screening for clinical trials, reducing the need for manual screening of every patient.
Implications for Sepsis Clinical Trials
"Our interpretable AI-based model using clinical data accurately identifies patients with different disease courses, which can reduce the heterogeneity of patients with sepsis in future clinical trials," the researchers stated. By identifying patients with a high likelihood of remaining persistently ill, the model can help ensure that clinical trials enroll a more homogeneous population, increasing the chances of observing a treatment effect. The use of conformal prediction further enhances the model's utility by providing a measure of confidence in its predictions, allowing clinicians to make more informed decisions about patient enrollment.
Future Directions
The researchers acknowledge several limitations to their study, including the retrospective nature of the analysis and differences in sepsis onset definition between the MIMIC-IV and eICU-CRD databases. They plan to conduct further prospective external validation using the web-based calculator to assess the model's performance in real-world clinical settings.
