Epic Generative Artificial Intelligence Chart Summarization Tool to Reduce Ambulatory Provider Cognitive Task Load: A Randomized Controlled Trial
Overview
- Phase
- Not Applicable
- Status
- Active, not recruiting
- Enrollment
- 284
- Locations
- 1
- Primary Endpoint
- Change from Baseline Physician Task Load
Overview
Brief Summary
This is a RCT of 284 outpatient physicians at a large academic health system, randomized 1:1 to an electronic health record (EHR) produced generative AI outpatient chart summarization tool or a usual-care control group. The 90 day study will observe the effects of the tool prior to system-wide roll out of the tool.
Detailed Description
The primary aim of this study is to evaluate the impact of an EHR developed generative AI outpatient chart summarization tool on self-reported physician-task load score (PTL), comparing the tool to a control group. Exploratory outcomes include EHR-derived time metrics (Caboodle and Signal), Professional fulfillment Index (PFI), usability (SUS), provider satisfaction and productivity, and patient experience item results from CG-CAHPS. We will also evaluate whether AI literacy modifies adoption and effect of the tool using the short-form Meta AI Literacy Scale (MAILS). On an exploratory basis, we will also perform adjustments based on provider specialty, access to an ambient-listening AI scribe, panel complexity, provider age group, provider sex, and time-varying effects by month over the study period.
Enrolled participants are randomized to one of two groups. Randomization will be stratified by whether the participant has an active AI scribe license, and covariate-constrained randomization will be performed within strata to improve balance on baseline PTL (NASA-TLX-adapted score) and a modified baseline chart review time (Caboodle-derived). Due to the nature of the intervention, participants cannot be blinded to group assignment.
The primary purpose of the initiative is to improve quality, efficiency, and business operations at University of California, Los Angeles (UCLA) Health and will inform the operational implementation of the tool across all providers within the UCLA Health System. Nevertheless, the UCLA study team plans to rigorously examine and publish the impact of this intervention across the health system, which is why the study team pre-registered the initiative.
Study Design
- Study Type
- Interventional
- Allocation
- Randomized
- Intervention Model
- Parallel
- Primary Purpose
- Health Services Research
- Masking
- Single (Investigator)
Eligibility Criteria
- Sex
- All
- Accepts Healthy Volunteers
- No
Inclusion Criteria
- •Ambulatory care providers within the UCLA Health system including physicians and advanced practice providers (APPs), such as nurse practitioners and physician assistants with at least one half-day clinic session per week.
- •Providers complete baseline pre-survey
Exclusion Criteria
- •Trainee providers (e.g., residents, medical students), and psychologists
Arms & Interventions
Intervention Arm
Participants in this arm will have access to Epic's outpatient chart summarization tool and will continue their usual clinical practice, supported by the generative AI tool, which is integrated within the EHR. The tool provides a summary for providers and does not provide clinical decision support. They have access to an educational module and tipsheet, and weekly town halls to help with any questions for the first three weeks of the trial.
Intervention: GenAI Chart Summarization (Other)
Care As Usual
Participants in this arm will not have access to chart summarization tool and will continue their usual clinical practice.
Outcomes
Primary Outcomes
Change from Baseline Physician Task Load
Time Frame: Baseline and 90 days after initial exposure to the intervention
Physician task load adapted from the NASA Task Load Index (TLX), a validated tool for assessing EHR-related cognitive task load in four sub-scales (mental demand, temporal demand, physical demand, and effort). This outcome is adapted to capture the task of pre-charting, defined for this study as the practice of reviewing patient information in the EHR before a patient visit to prepare for the encounter. Each sub-scale is rated from 0 (low) to 100 (high) and is aggregated to a 0-400 point scale. No patient level information will be collected for this outcome measure.
Secondary Outcomes
- Change in Modified Total Chart Time Per Encounter(Baseline, after 60 days of exposure to the intervention, and after 90 days of exposure to the intervention.)
- Change from Baseline Professional Fulfillment Index Score(Baseline and 90 days after initial exposure to the intervention)
- Change from Baseline Self-Reported Pre-Charting Effectiveness(Baseline and 90 days from initial exposure to the intervention)
- Provider Satisfaction Scores(90 days after initial exposure to the intervention)
- System Usability Scale(90 days from initial exposure to the intervention)
- Safety Events(Over the 90 day period following after initial exposure to the intervention)
- Qualitative Tool-Specific EHR Feedback(Over the 90 days following initial exposure to the intervention)
- Change from Baseline Consumer Assessment of Healthcare Providers and Systems Clinician & Group Survey (CG-CAHPS) Metric(Baseline and 90 days after initial exposure to the intervention)
- Change from Baseline Epic Signal (Activity) Metric: Time Outside Scheduled Hours(Baseline and 90 days after the initial exposure to the intervention)
- Clinician Relative Value Units (RVUs) per Week(60 days after exposure to the intervention, and 90 days after exposure to the intervention)
Investigators
John N. Mafi, MD, MPH
Associate Professor of Medicine
University of California, Los Angeles