Impact of Automated Sepsis Metric Evaluation on Provider Performance

Sepsis is a leading cause of morbidity and mortality worldwide and remains a major focus of hospital quality improvement efforts. In the United States, the Centers for Medicare & Medicaid Services (CMS) Severe Sepsis and Septic Shock Management Bundle (SEP-1) is a publicly reported quality measure that evaluates adherence to evidence-based processes of care, including timely antibiotic administration, fluid resuscitation, and laboratory testing. Despite its importance, SEP-1 is widely recognized as a complex measure, consisting of dozens of individual elements that must be completed within specific timeframes. Assessment of compliance typically relies on manual chart abstraction, which is resource-intensive, subject to variability, and performed on a small subset of cases with delays of several months. These limitations reduce the ability of SEP-1 reporting to drive timely improvements in clinical care.

Advances in artificial intelligence (AI), particularly large language models (LLMs), offer an opportunity to automate the extraction and interpretation of clinical information from unstructured medical records. Prior work has demonstrated that LLMs can achieve high agreement with expert reviewers when applied to complex clinical abstraction tasks. Building on this foundation, this study evaluates whether integrating AI-enabled chart abstraction into a real-world clinical workflow can improve performance on a complex quality measure by providing near real-time feedback to clinicians.

This study was conducted as a prospective, cluster randomized quality improvement initiative across two academic emergency departments within a single health system. Attending emergency physicians were randomized at the provider level to either an intervention group receiving AI-generated feedback or a control group receiving standard quality reporting feedback. Randomization at the physician level was selected to minimize contamination while preserving real-world clinical workflows.

All adult patients presenting to the emergency department who met CMS criteria for severe sepsis or septic shock were eligible for inclusion. Cases were identified using clinical encounter diagnoses and evaluated according to SEP-1 specifications. Each case was classified as meeting or failing the measure based on completion of required elements within defined time windows.

In the intervention arm, an LLM-based system automatically reviewed each eligible patient encounter at the time of emergency department discharge. The system analyzed structured and unstructured clinical data to determine SEP-1 compliance and identify specific elements of care that were incomplete or not documented. Physicians received individualized feedback shortly after patient care was completed. Feedback included a summary of the case, the determination of compliance, and targeted recommendations for improvement when deficiencies were identified. In selected cases, additional follow-up communication was provided to reinforce learning and clarify opportunities for improvement.