Evaluating Conversational Artificial Intelligence for Depression Management
- Conditions
- Major Depressive Disorder (MDD)
- Registration Number
- NCT07105397
- Lead Sponsor
- George Mason University
- Brief Summary
The goal of this clinical trial is to evaluate how a conversational method of collecting medical history affects patients' perceptions and experiences compared to traditional online, closed-ended surveys. Both methods collect identical medical history information, can be completed by patients at home, and do not disrupt routine clinical care.
The primary questions this study aims to answer are:
1. Does conversational intake affect patients' perceptions of empathy during their clinical interactions?
2. Does conversational intake strengthen the therapeutic bond patients feel toward their clinicians compared to traditional surveys?
Participants will be randomly assigned to one of two intake methods:
1. Conversational intake: Participants answer questions about their medical history through a natural, dialogue-based interface.
2. Closed-ended survey intake: Participants complete a structured, multiple-choice questionnaire about their medical history.
After completing their assigned intake method, participants will rate their experience, particularly in terms of empathy and therapeutic bond, and compare it to their usual interactions with their own clinicians.
- Detailed Description
Conversational artificial intelligence (AI) systems, such as those based on Large Language Models (LLMs) like ChatGPT, offer innovative ways to engage patients in health-related conversations. Despite these advances, challenges remain regarding patient safety and system reliability. Specific concerns include biased recommendations against certain patient groups, inaccuracies or misleading responses, and mechanical, unempathic interactions, particularly during sensitive moments such as when patients express suicidal thoughts. Testing conversational AI in healthcare settings is complicated due to the diverse medical, linguistic, and behavioral characteristics exhibited by patients.
This study addresses these challenges by developing an advanced conversational AI system guided by a structured knowledge-based topic network to maintain conversation relevance and coherence. Additionally, the investigators introduce a novel patient simulator methodology that mimics diverse medical histories, linguistic styles, and behavioral interactions, enhancing pre-clinical testing rigor.
The research focuses specifically on the clinical context of depression management, aiming to optimize antidepressant selection. Currently, many patients undergo a frustrating and costly trial-and-error process to find effective antidepressants. The study compares two approaches designed to streamline and personalize this process:
1. Conversational AI Intake: Engages patients through flexible, open-ended dialogue to gather medical history and generate personalized antidepressant recommendations.
2. Structured Questionnaire Intake: Utilizes a closed-ended, multiple-choice format to systematically collect patient medical histories for antidepressant recommendation.
Both methods leverage a curated, evidence-based knowledgebase of 15 commonly used antidepressants, considering factors like patient age, gender, comorbidities, and previous antidepressant use. The accuracy and completeness of the AI-generated recommendations are rigorously verified in by clinicians prior to any medication changes, adhering to FDA safety requirements.
A primary goal of the project is to evaluate how conversational AI impacts patient-centered outcomes, specifically patient perceptions of empathy, therapeutic bond, and communication quality. Patients with major depressive disorder will be recruited online, enhancing participant diversity and representativeness. Participants will be randomly assigned to either the conversational AI or the structured questionnaire method. Outcomes will include differences in data completeness, patient perceptions of empathy, and strength of therapeutic alliance.
Beyond immediate clinical outcomes, the project's methodological advancements, particularly the development of robust, bias-mitigated conversational systems and comprehensive patient simulation for AI testing, will have broad applicability across healthcare domains. The conversational AI and patient simulator will be made publicly available at no cost, providing tools that other researchers, clinicians, and healthcare providers can utilize and adapt to various health contexts.
Patient and stakeholder engagement is integral to the study. A representative advisory board, including patients with lived experience of depression, clinicians, mental health advocates, and researchers, guides all phases of the project. This collaborative framework ensures that the research remains patient-centered and responsive to real-world clinical needs and experiences.
Recruitment & Eligibility
- Status
- NOT_YET_RECRUITING
- Sex
- All
- Target Recruitment
- 130
- 18 - 85 years old
- have a major depression diagnosis from a clinician and score 10 or higher on the Patient Health Questionnaire (PHQ-9)
- reside in a state where study clinicians are licensed
- have access to the Internet via phone or computer
- have no language, sensorial, or cognitive barriers to providing written informed consent
- must have a primary care provider, a mental health specialist, or agree to see a study clinician
- has clinically diagnosed bipolar disorder
Study & Design
- Study Type
- INTERVENTIONAL
- Study Design
- PARALLEL
- Primary Outcome Measures
Name Time Method Perceptions of empathy From enrollment up to 4 months after participation The primary outcome assesses the impact of intake methods (conversational AI vs. structured survey) on patients' perceptions of empathy. Patient empathy perceptions will be measured using the Jefferson Scale of Empathy (JSE), a validated instrument. A higher JSE score means higher perceptions of empathy. We hypothesize that patients interacting with the conversational AI will report higher perceived empathy scores compared to those using the structured survey. Data analysis will employ comparison of independent means through Analysis of Variance (ANOVA). If randomization does not yield comparable groups, we will apply covariate-balanced ANOVA. Power analysis (conducted in RStudio) indicated a sample size of 130 patients (power = 0.80, significance level = 0.05), based on a mean JSE score of 116.6, standard deviation of 10.56, and an assumed medium effect size (5% mean difference).
- Secondary Outcome Measures
Name Time Method Communication Accommodation From enrollment up to 4 months after participation We will assess communication accommodation to determine whether the conversational AI meets patient needs through a content analysis of conversation transcripts. This analysis will rate three key dimensions: empathy, communication quality, and effectiveness. Codes for each are drawn from validated, widely used scales. From this, we will generate composite scores to classify AI conversations as high or low in each dimension and overall. To validate our findings, we will compare content analysis scores with patient self-reports on perceived levels (high or low) of empathy, quality, and effectiveness. Together, these data will provide a robust evaluation of the AI's communication accommodation.
Therapeutic Alliance Between System and Patient From enrollment up to 4 months after participation We will use a multi-measure strategy to evaluate the therapeutic alliance between the AI system and patients. This includes four related measures. First, a content analysis of patient-AI conversation transcripts will rate the level of empathy using a validated scale from prior studies. These ratings will generate a score classifying each conversation as high or low in empathy, a key component of therapeutic alliance. To validate these findings, we will compare content analysis scores with three patient self-reports: perceived therapeutic alliance, perceived empathy in the AI conversation, and baseline empathy typically experienced with their clinicians. Together, these measures will offer a robust evaluation of the therapeutic alliance established by the AI system.
Adherence to recommendations From enrollment up to 4 months after participation Adherence to the recommendations provided by either the structured questionnaire or the conversational AI will be evaluated approximately one month after patient participation. Follow-up includes a structured questionnaire and brief interview, where patients report (1) if they discussed system-generated recommendations with their clinician, and (2) their current medications by direct reference to medication containers. Collected data will allow classification of clinician prescriptions as either concordant or discordant with system-generated recommendations, facilitating evaluation of adherence behaviors.
Trial Locations
- Locations (1)
George Mason University
🇺🇸Fairfax, Virginia, United States
George Mason University🇺🇸Fairfax, Virginia, United StatesFarrokh Alemi, PhDPrincipal InvestigatorKevin Lybarger, PhDPrincipal Investigator