Using Explainable AI Risk Predictions to Nudge Influenza Vaccine Uptake

Not Applicable

Completed

Conditions: Health Promotion
Vaccination
Influenza
Health Behavior
Risk Reduction

Interventions: Behavioral: Reminder
Behavioral: Risk reduction
Behavioral: Medical records-based recommendation
Behavioral: Algorithm-based recommendation

Registration Number: NCT05009251

Lead Sponsor: National Bureau of Economic Research, Inc.

Brief Summary: The study team previously demonstrated that patients are more likely to receive flu vaccine after learning that they are at high risk for flu complications. Building on this past work, the present study will explore whether providing reasons that patients are considered high risk for flu complications (a) further increases the likelihood they will receive flu vaccine and (b) decreases the likelihood that they receive diagnoses of flu and/or flu-like symptoms in the ensuing flu season. It will also examine whether informing patients that their high-risk status was determined by analyzing their medical records or by an artificial intelligence (AI) / machine-learning (ML) algorithm analyzing their medical records will affect the likelihood of receiving the flu vaccine or diagnoses of flu and/or flu-like symptoms.

Detailed Description: Geisinger has partnered with Medial EarlySign and developed an ML algorithm to identify patients at risk for serious (moderate to severe) flu-associated complications on the basis of their existing electronic health record (EHR) data. Geisinger will apply this algorithm to current patients during the 2021-22 flu season.

This study will evaluate the effect of contacting patients identified as high risk with special messages to encourage vaccination. These communications will inform patients they are at high risk with either (a) no additional explanation, (b) an explanation that this determination comes from an analysis of their medical records, along with a short list of the top factors from their medical record that explain their risk, and (c) the additional explanation that an AI or ML algorithm made this determination, along with a short list of the top factors from their medical record that explain their risk.

Included in the study will be current Geisinger patients 18+ years of age with no contraindications for flu vaccine and who have been assessed by the Medial algorithm and assigned a risk score. The primary study outcomes will be the rates of flu vaccination and flu diagnosis during the 2020-21 season by targeted patients.

Recruitment & Eligibility

Status: COMPLETED

Sex: All

Target Recruitment: 45061

Inclusion Criteria

Aged 18 or older
Current Geisinger patient at the time of study
Falls in the top 10% of patients at highest risk, as identified by the flu-complication risk scores of machine learning algorithm (which operates on coded EHR data)

Exclusion Criteria

Has contraindications for flu vaccination
Has opted out of receiving communications from Geisinger via all of the modalities being tested

Study & Design

Study Type: INTERVENTIONAL

Study Design: PARALLEL

Arm && Interventions

Group	Intervention	Description
High Risk Only	Reminder	Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.
Reminder Control	Reminder	Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status.
High Risk with Explanation Based on Algorithm	Risk reduction	Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.
High Risk with Explanation Based on Algorithm	Medical records-based recommendation	Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.
High Risk Only	Risk reduction	Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.
High Risk with Explanation Based on Algorithm	Reminder	Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.
High Risk with Explanation Based on Medical Records	Reminder	Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.
High Risk with Explanation Based on Medical Records	Risk reduction	Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.
High Risk with Explanation Based on Medical Records	Medical records-based recommendation	Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.
High Risk with Explanation Based on Algorithm	Algorithm-based recommendation	Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.

Primary Outcome Measures

Name	Time	Method
Flu Vaccination at 2 Weeks After Final Outreach Date	Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start	Received flu vaccination

Secondary Outcome Measures

Name	Time	Method
Healthcare Utilization	11 months (between September 9, 2021 and July 31, 2022)	Visited ER or was hospitalized NOTE: Our prespecified outcome was description was "Visited ER, was hospitalized, or had flu-related insurance claims." We did not receive claims data, so this outcomes includes only ER visits and hospitalizations.
Flu Diagnosis	8 months (between September 9, 2021 and April 30, 2022)	Received a "high confidence flu" diagnosis (with positive polymerase chain reaction \[PCR\]/antigen/molecular test) and/or "likely flu" diagnosis (as assessed via International Classification of Disease \[ICD\] codes or Tamiflu administration or positive PCR/antigen/molecular test) Note that "likely flu" is a superset of the "high confidence flu" diagnoses.
Flu Complications	11 months (between September 9, 2021 and July 31, 2022)	Diagnosed with flu-related complications
Flu Vaccination at 9 Weeks After Final Outreach Date	Within 9 weeks of the final outreach date, 106 days (15.14 weeks) after the study start	Received a flu vaccination