Physician Usability and Output Quality Evaluation of a Retrieval-Augmented Language Model System for Specialty Medical Consultation: A Human-Computer Interaction Study
试验速览
- 阶段
- 不适用
- 状态
- 尚未招募
- 入组人数
- 15
- 试验地点
- 1
- 主要终点
- Clinical actionability of the clinician's decision
研究概览
简要总结
The goal of this study is to test an artificial intelligence (AI) tool called SAGE. SAGE helps primary care doctors with questions that often need a specialist. Primary care doctors are the doctors people usually see first. SAGE reviews a case and suggests what a specialist might advise.
The main questions it aims to answer are:
- Do doctors make sound, timely care decisions when they use SAGE?
- Do those decisions match what a specialist would advise?
Researchers will compare the decisions doctors make with and without SAGE.
Doctors in the study will:
- Review made-up patient cases (these are not real patients)
- Make a decision for each case, their usual way and with SAGE
研究设计
- 研究类型
- Interventional
- 分配方式
- Na
- 干预模型
- Single Group
- 主要目的
- Health Services Research
- 盲法
- None
入排标准
- 年龄范围
- 18 Years 至 —（Adult, Older Adult）
- 性别
- All
- 接受健康志愿者
- 是
入选标准
- •Physician practicing in primary care internal medicine and/or family medicine
- •Attending physician, or resident physician at postgraduate year 2 (PGY-2) or higher
- •Stanford-affiliated (faculty, staff, or trainee)
- •Able to complete a single, approximately 60-minute remote (Zoom) session conducted in English
排除标准
- •Not currently practicing in primary care internal medicine or family medicine
- •Member of the study team or otherwise involved in the development of SAGE
- •Unable to complete the remote study session
结局指标
主要结局
Clinical actionability of the clinician's decision
时间窗: Post-session independent review, through study completion (up to approximately 2 months)
For each case, independent reviewers judge whether the physician's decision is clinically actionable, meaning the physician can act on it now rather than wait for a specialist. The outcome is the proportion of cases judged actionable, measured separately for decisions made with SAGE and for decisions made the standard way.
Concordance of the clinician's decision with the specialist reference answer
时间窗: Post-session independent review, through study completion (up to approximately 2 months)
Agreement between the physician's management decision and a specialist reference answer, rated by independent reviewers, with inter-rater agreement summarized. Assessed for decisions made with SAGE and for decisions made the standard way.
次要结局
- System usability (System Usability Scale, SUS)(End of study session (Day 1, approximately 60 minutes))
- Order acceptance in the SAGE condition(During study session (Day 1))
- Time to complete case review(During study session (Day 1))
研究者
Jonathan Chen
Associate Professor of Medicine
Stanford University