Better Real-time Information on Documentation of Goals of Care for Engagement in Serious Illness Communication
- Conditions
- Advanced CancerCancerEnd of Life
- Registration Number
- NCT07147023
- Lead Sponsor
- Dana-Farber Cancer Institute
- Brief Summary
The goal of this study is to test the accuracy of Large Language Model-generated serious illness communication (SIC) summaries, the feasibility of delivering the SIC summaries, and to collect perspectives on the SIC summaries from clinicians and participants with cancer. Large Language Models (LLMs) are artificial intelligence programs that can perform various natural language processing tasks.
- Detailed Description
The study will enroll 60 Dana-Farber patients admitted to Brigham and Women's hospital who have an elevated mortality prediction at admission. The research study procedures include randomization to intervention or control arms on admission (3:1 intervention to control). For patients in the intervention arm, a large language model query will summarize serious illness communication documentation in their medical record screening from the prior 6 months. A summary of the documentation will be sent to the inpatient and outpatient oncology clinicians within 24 hours of admission; these clinicians will be asked to review the summaries, discuss with the patient and incorporate into care plans as appropriate. For patients with no SIC documented on day 4 of the admission, a second email will be sent. Control patients will receive usual care, which is no email sent to their oncology teams. All patients will be offered to participate in an interview on day 4 of their admission about their communication with their care teams. For patients in both arms, inpatient and outpatient oncology attendings will be sent an email survey regarding SIC for the patient and, if applicable, the utility of the SIC summaries.
Recruitment & Eligibility
- Status
- RECRUITING
- Sex
- All
- Target Recruitment
- 60
- Dana-Farber Cancer Institute participants
- Admission to an inpatient solid tumor medical oncology service at BWH (including beds that are considered to be DFCI beds within BWH)
- Age ≥ 18 years
- Mortality prediction ≥ 40%
- Admitted on a Sunday after 4pm, Monday or Tuesday before 4pm. These time restrictions are necessary for the workflow of this pragmatic pilot trial, as an individual will need to send out the intervention emails and interview patients approximately 48-72 hours after admission, both of which need to occur during the work week.
Participant
-Patients with elective inpatient admissions (typically for chemotherapy or other treatments)
Clinician Inclusion Criteria:
-The clinician fills one of the following roles for the enrolled patient:
- inpatient attending on the medical oncology team
- outpatient Dana-Farber medical oncologist
Study & Design
- Study Type
- INTERVENTIONAL
- Study Design
- PARALLEL
- Primary Outcome Measures
Name Time Method Serious Illness Conversation (SIC) summary accuracy 1 week We will begin by creating a gold standard dataset of manually curated SIC conversations through structured chart review, using established methods previously applied in our work. Documentation will be annotated using a standardized codebook that includes clear definitions and examples of what does and does not qualify as SIC. Annotations will be conducted using the Label Studio tool, which facilitates efficient comparison of annotations across multiple reviewers.
Agreement across human annotators for the gold standard dataset will be assessed using Cohen's kappa scores. The accuracy of LLM-generated SIC summaries will be determined by comparing them to the manually annotated gold standard using standard performance metrics including accuracy, recall, precision, F1 score, specificity, and a hallucination index. We will use an accuracy threshold of identifying SIC documentation for the goals of care domain in 70% of patients to define a positive trial outcome.
- Secondary Outcome Measures
Name Time Method Feasibility (successful delivery of the intervention) 1 week Feasibility is defined as the successful delivery of the summary email, and will be calculated as the number of participants with successful delivery of the intervention email divided by the total number of intervention participants, with an 80% feasibility threshold.
Clinician Perspectives of Clinical Utility 1 week Clinician perspectives of clinical utility are defined as clinicians' views on the usefulness and relevance of serious illness communication (SIC) and, for intervention-arm clinicians, the LLM-generated SIC summaries for informing participant care. These perspectives will be assessed via structured REDCap surveys sent to both inpatient and outpatient clinicians, with follow-up phone interviews conducted by the research assistants for non-respondents.
Clinician responses to surveys and interview questions will be summarized and stratified by intervention vs control groups. We will compare survey responses between the two study arms using Chisquared and Mann-Whitney U tests for binary and likert items, respectively. Responses to open-ended questions will be qualitatively summarized.Participant Perspectives on Inpatient and Outpatient Clinician Understanding of their Care Preferences and Acceptability of SIC Summaries 1 week Participant acceptability is defined as participant perceptions of the appropriateness and usefulness of LLM-generated summaries of serious illness conversations (SICs). It will be assessed through structured in-person interviews including the validated "Heard and Understood" questionnaire and open-ended questions regarding the acceptability of SIC summaries and participant-clinician communication.
Participant responses to interview questions will be summarized and stratified by intervention vs control groups. Responses to open-ended questions will be qualitatively summarized.New Documentation of SIC 1 week New documentation of serious illness communication (SIC) during the index hospitalization is defined as any new note or entry created in the electronic health record (EHR) during the current hospital admission that reflects a discussion of the participant's values, goals, preferences, or prognosis, and meets the standardized codebook criteria for SIC.
Trial Locations
- Locations (2)
Brigham and Women's Hospital
🇺🇸Boston, Massachusetts, United States
Dana-Farber Cancer Institute
🇺🇸Boston, Massachusetts, United States
Brigham and Women's Hospital🇺🇸Boston, Massachusetts, United StatesChristopher Manz, MD, MSHPContact617-632-3315Christopher_manz@dfci.harvard.eduChristopher Manz, MDPrincipal Investigator