Diagnostic Reasoning With Large Language Model Chat Bots
试验速览
- 阶段
- 不适用
- 状态
- 已完成
- 入组人数
- 50
- 试验地点
- 1
- 主要终点
- Diagnostic reasoning
研究概览
简要总结
This study will evaluate the effect of providing access to GPT-4, a large language model, compared to traditional diagnostic decision support tools on performance on case-based diagnostic reasoning tasks.
详细描述
Artificial intelligence (AI) technologies, specifically advanced large language models like OpenAI's ChatGPT, have the potential to improve medical decision-making. Although ChatGPT-4 was not developed for its use in medical-specific applications, it has demonstrated promise in various healthcare contexts, including medical note-writing, addressing patient inquiries, and facilitating medical consultation. However, little is known about how ChatGPT augments the clinical reasoning abilities of clinicians.
Clinical reasoning is a complex process involving pattern recognition, knowledge application, and probabilistic reasoning. Integrating AI tools like ChatGPT-4 into physician workflows could potentially help reduce clinician workload and decrease the likelihood of missed diagnoses. However, ChatGPT-4 was not developed for the purpose of clinical reasoning nor has it been validated for this purpose. Further, it may be subject to disinformation, including convincing confabulations that may mislead clinicians. If clinicians misuse this tool, it may not improve diagnostic reasoning and could even cause harm. Therefore, it is important to study how clinicians use large language models to augment clinical reasoning prior to routine incorporation into patient care.
In this study, we will randomize participants to answer diagnostic cases with or without access to ChatGPT-4. The participants will be asked to give three differential diagnoses for each case, with supporting and opposing findings for each diagnosis. Additionally they will be asked to provide their top diagnosis along with next diagnostic steps. Answers will be graded by independent reviewers blinded to treatment assignment.
研究设计
- 研究类型
- Interventional
- 分配方式
- Randomized
- 干预模型
- Parallel
- 主要目的
- Diagnostic
- 盲法
- Single (Outcomes Assessor)
盲法说明
The grading of responses will be performed by assessors blinded to participant identity and treatment assignment.
入排标准
- 性别
- All
- 接受健康志愿者
- 是
入选标准
- •Participants must be licensed physicians and have completed at least post-graduate year 2 (PGY2) of medical training.
- •Training in Internal medicine, family medicine, or emergency medicine.
排除标准
- •Not currently practicing clinically.
研究组 & 干预措施
GPT-4
Group will be given access to GPT-4.
干预措施: GPT-4 (Other)
Usual resources
Group will not be given access to GPT-4 but will be encouraged to use any resources they wish besides large language models (UpToDate, Dynamed, google, etc).
结局指标
主要结局
Diagnostic reasoning
时间窗: During evaluation
The primary outcome will be the percent correct (range: 0 to 100) for each case. For each case, participants will be asked for three top diagnoses and findings from the case that support that diagnosis and oppose that diagnosis. Participants will receive 1 point for each plausible diagnosis. Findings supporting the diagnosis and findings opposing the diagnosis will also be graded based on correctness, with 1 point for partially correct and 2 points for completely correct responses. Participants will then be asked to name their top diagnosis, earning one point for a reasonable response and two points for the most correct response. Finally participants will be asked to name up to 3 next steps to further evaluate the patient with one point awarded for a partially correct response and two points for a completely correct response. The primary outcome will be compared on the case-level by the randomized groups.
次要结局
- Time Spent on Diagnosis(During evaluation)
研究者
Jonathan Chen
Assistant Professor of Medicine
Stanford University