Enhancing Interdisciplinary Understanding of Ophthalmology Notes Through a Local Large Language Model

Not Applicable

Completed

• Conditions: Communication; Artificial Intelligence (AI); Artificial Intelligence Technology; Interdisciplinary Communication
• Interventions: Other: Large Language Model-generated Plain Language Summary of Ophthalmology notes

• Registration Number: NCT06624605
• Lead Sponsor: John J Chen

Brief Summary

This prospective, randomized controlled trial evaluated the efficacy of adding Large Language model (LLM)-generated Plain Language Summaries (PLSs) to Standard Ophthalmology Notes (SONs) in enhancing comprehension among non-ophthalmology providers. The study utilized surveys to assess non-ophthalmology providers\' comprehension and satisfaction with the notes and ophthalmologists\' evaluation of PLS accuracy, safety, and time burden. An objective semantic and linguistic analysis of the PLSs was also conducted.

Detailed Description

Not available

Study Timeline

• Study Start: 2024-02-01
• Primary Completion: 2024-05-31
• Study Completion: 2024-05-31
• Study First Submitted: 2024-09-30
• Status Verified: 2024-10
• Study First Submitted QC: 2024-10-01
• Last Update Submitted: 2024-10-01
• Study First Posted: 2024-10-03
• Last Update Posted: 2024-10-03

Recruitment & Eligibility

• Status: COMPLETED
• Sex: All
• Target Recruitment: 851

Inclusion Criteria

• Evaluation by an Ophthalmologist in the Mayo Clinic Department of Ophthalmology in Rochester, MN in either the inpatient or outpatient setting between Feb 1, 2024 - May 31, 2024.

Exclusion Criteria

• No in-network non-Ophthalmologist care providers in the medical record
• Procedure-only visits containing only a procedure note

Study & Design

• Study Type: INTERVENTIONAL
• Study Design: PARALLEL

Arm && Interventions

Group	Intervention	Description
Intervention	Large Language Model-generated Plain Language Summary of Ophthalmology notes	Participants receive standard ophthalmology notes with an appended LLM-generated plain language summary, reviewed and edited by the ophthalmologist. The plain language summary is clearly marked as generated as part of a Quality Improvement initiative.

Primary Outcome Measures

Name	Time	Method
Non-Ophthalmologist Note Comprehension and Satisfaction	From enrollment to 8 weeks after enrollment	Assessed via survey responses evaluating understanding of the patient\'s ophthalmology diagnosis, satisfaction with the note, and overall preference between Standard Ophthalmology Note and Standard Ophthalmology Note + Plain Language Summary.; Degree of understanding was assessed on the following Likert scale: Not at all Neutral Moderately A great deal; Satisfaction was graded on the following Likert scale: Not satisfied at all Somewhat unsatisfied Neutral Somewhat satisfied Satisfied; Preference between note types was assessed on the following Likert scale: Standard Note, a great deal Standard Note, somewhat Neutral Plain Language Summary, somewhat Plain Language Summary, a great deal

Secondary Outcome Measures

Name	Time	Method
Ophthalmologist Evaluation of Summary Accuracy and Time burden	Assessed at a single time point <24 hr after enrollment	Assessed via survey responses regarding Plain Language Summary accuracy in reflecting the Standard Ophthalmology Note findings, time spent reviewing/editing, and perceived burden.; Accuracy in reflecting findings was graded on the following Likert scale: Not at all Neutral A little A great deal; Time spent reviewing/editing was reported using the following options: \<1 minute; 1. minute; 2. minutes; 3. minutes; 4. minutes; 5. or more minutes; Perceived burden was graded on the following Likert scale: Not at all Neutral A little A great deal
Semantic and Linguistic Quality of Plain Language Summaries	Assessed at a single time point <24 hr following enrollment	SBERT Cosine Similarity Measure of Linguistic Similarity; Scale: -1 (least similar) to +1 (most similar)

Trial Locations

Locations (1)

Mayo Clinic; 🇺🇸 Rochester, Minnesota, United States