Artificial Intelligence Assessment of Trainee Communication Compared to Human Assessment: A Prospective Cohort Study
试验速览
- 阶段
- 不适用
- 状态
- 已完成
- 入组人数
- 30
- 试验地点
- 2
- 主要终点
- Reliability of AI Conversational Performance Assessment
研究概览
简要总结
This study will evaluate whether a web-based artificial intelligence platform (AI) (Clinical Mind AI [CMAI] Stanford, CA), can assess communication skills in anesthesiology trainees, including residents and fellows, in the setting of disclosing medical errors. All participants will participate in an AI-generated simulation by using the platform remotely, and CMAI will assess trainee performance immediately after the simulation.
详细描述
The goal of this study is to evaluate the accuracy of CMAI platform to assess participant performance following a voice-based AI simulation designed to help trainees practice disclosing medical errors. The platform will feature a custom clinical case, created using CMAI's patient creation tool, involving a discussion with the parent of a child who suffered a dental injury that occurred during intubation. The platform will provide an audio-based, simulated encounter with the parent enabling the participant to interact with, and then assess trainee communication performance, followed by the delivery of questionnaires to the trainee to determine usability and satisfaction. A human evaluator will also assess the trainees' performance using the same scales, and the investigators will compare the AI performance evaluation to the human evaluation. This study will allow the investigators to determine:
- Reliability of the performance assessments of the CMAI platform compared to human raters
- Usability for the CMAI audio voice model for simulated patient encounters
- Satisfaction related to an innovative, educational technique
By evaluating these domains, we aim to determine the educational value of using simulated voice communication for training in emotionally complex, clinical scenarios.
研究设计
- 研究类型
- Interventional
- 分配方式
- Na
- 干预模型
- Single Group
- 主要目的
- Supportive Care
- 盲法
- None
入排标准
- 年龄范围
- 18 Years 至 99 Years（Adult, Older Adult）
- 性别
- All
- 接受健康志愿者
- 是
入选标准
- •Participant's must be 18 years of age or older
- •Graduate medical trainess such as residents and fellows at all levels of training excluding PGY1 trainees at Department of Anesthesiology, Perioperative, and Pain Medicine at Stanford University
- •Able to speak and understand English
- •Willing and able to provide consent to participate in research
- •Able to participate in Artificial Intelligence communication simulation
排除标准
- •Non-English speaker
- •unable to provide consent to participate in research
- •individual with no access to the necessary technology (internet, computer/smart device) required for participating in the digital, voice-based interaction with the AI platform
研究组 & 干预措施
Voice Artificial Intelligence (AI) Group
Participants will remotely engage with an AI-simulated parent of a pediatric patient through a voice call, discussing a clinical scenario concerning their child. The participant's communication skills will be assessed using an artificial intelligence (AI) system and two human evaluators using the same rating scales.
干预措施: Communication Assessment through AI and Human (Other)
结局指标
主要结局
Reliability of AI Conversational Performance Assessment
时间窗: Immediately after the simulation
The primary outcome is to evaluate the reliability of the Clinical Mind AI (CMAI) platform in accurately assessing the communication skills of medical education trainees during a simulated interaction. To measure this outcome, the CMAI will use the Breaking Bad News Assessment Schedule (BBAS), a validated tool assessing participant's communication skills in five domains: 1) Setting the scene, 2) Breaking the news, 3) Eliciting concerns, 4) Informative giving, and 5) Empathy and Support. It includes 17 items, each with sub questions, rated on a 5-point Likert scale from (1-5), where score meanings may vary by each question. The scoring will be guided by Breaking Bad News Rubric, which outlines performance criteria for each item on the BBAS. In addition, two trained human evaluators will independently asses the participant's communication skills using the same BBAS tool and rubric. Lastly, both CMAI and human scores will be compared to determine the reliability of the CMAI platform.
次要结局
- Usability of CMAI Simulation Platform(immediatley after the simulation)
- Satisfaction of CMAI Simulation(immediately after the simulation)
研究者
Thomas Caruso
Clinical Professor
Stanford University