Testing the Efficacy of an Artificial Intelligence Real-Time Coaching System in Comparison to In Person Expert Instruction in Surgical Simulation Training of Medical Students - A Randomized Controlled Trial
试验速览
- 阶段
- 不适用
- 状态
- 已完成
- 入组人数
- 98
- 试验地点
- 1
- 主要终点
- Change in performance
研究概览
简要总结
Background:
Trainees learn surgical technical skills through apprenticeship model working closely with surgeons and given increased responsibility in patient cases under expert supervision. However, factors such as surgeons' busy schedule, number of available patient cases, patient safety and lack of objectivity and standardization in training pose strong limitations. Virtual reality surgical simulators integrated with artificial intelligence (AI) systems provide a standardized realistic simulation environment and detailed performance data that allows accurate quantitation of surgical skills and tailored feedback. These platforms make repetitive practice of surgical skills possible in a risk-free environment.
The Intelligent Continuous Monitoring System (ICEMS), a deep learning application integrated in NeuroVR simulation platform, was developed to assess surgical performance continuously in 0.2 second intervals and provide coaching and risk detection. Although a predictive validity for assessment module was provided previously, the effectiveness of real-time coaching and risk detection ability with this AI system remains to be explored.
The objective of this study is to compare the error-oriented intelligent feedback provided by the ICEMS to in-person expert instruction in surgical simulation training by monitoring the improvement of medical student technical skills on a series of virtual reality tumor resection tasks.
详细描述
Background Advancements in the fields of surgical simulation and artificial intelligence have resulted in the development of intelligent technical skill assessment and tutoring systems to meet the needs of the competency-based approach in medical education. These novel educational tools offer solutions for standardization and objectivity of the learner's technical competence assessment and training, in risk-free simulated operative environments. These systems can supplement the gold-standard apprenticeship model in training in surgery and other procedural based medicine by providing detailed information about technical skills, action-oriented procedural guidance, and risk assessment. Such applications can help educators in developing objective formative and summative student assessment and teaching methodologies.
Rationale The Intelligent Continuous Monitoring System (ICEMS), an artificial intelligence (AI) deep learning application was developed to assess surgical performance continuously in 0.2 second intervals with the ability of coaching trainees to expert level and risk detection. This system involved in a patent pending entitled 'Methods and systems for continuous monitoring of task performance on virtual simulators' (2020; patent No. 05001770-883USPR). Previously, a predictive validity for assessment module was provided however, the effectiveness of this system in real-time coaching trainees and risk detection needs to be explored.
Research Objectives To compare the effectiveness of an intelligent real-time coaching system to the gold standard expert mediated feedback. To test the transfer of learning during the practice tasks to the more complex realistic scenario.
Hypotheses
- The ICEMS feedback group will significantly improve in the composite-score between the first attempt to the last attempt in practice scenario.
- The control Group with no-expert mediated post hoc benchmark feedback will not statistically improve in the composite-score between first attempt to the last attempt in practice scenario.
- The ICEMS feedback group will have a statistically higher composite-score compared to the control group, in the fifth attempt in the practice scenario.
- The ICEMS feedback group will have a composite-score statistically non-inferior compared to the expert mediated instruction group in the fifth attempt on the practice scenario.
- The global OSATS score of the ICEMS feedback group will be non-inferior to that of the expert mediated instruction group in the complex realistic scenario.
- That ICEMS feedback will not elect a difference in emotion or cognitive load compared to the expert mediated instruction group.
研究设计
- 研究类型
- Interventional
- 分配方式
- Randomized
- 干预模型
- Parallel
- 主要目的
- Health Services Research
- 盲法
- Double (Participant, Outcomes Assessor)
盲法说明
Double (Participant and Expert Rater)
Participants do not know the performance metrics used in calculation of their final composite-score, only that they will be learning and practicing technical skills used in neurosurgery while receiving feedback from an instructor or an intelligent system, in subpial tumor resection procedures.
Experts do not know to which group the video performance they are rating belongs, for the OSATS rating.
入排标准
- 年龄范围
- 18 Years 至 —（Adult, Older Adult）
- 性别
- All
- 接受健康志愿者
- 是
入选标准
- •Medical students who are actively enrolled in medical school in any Canadian institution who do not meet the
排除标准
- •Premedical students who are actively enrolled in medical school in any Canadian institution who do not meet the exclusion criteria.
- •Exclusion Criteria:
- •Participation in previous trials involving the NeuroVR (CAE Healthcare) simulator.
研究组 & 干预措施
Control Group No-expert mediated post hoc benchmark group
30 participants. Individuals receive identical introductory information,same time, to perform, same scenarios as other groups.
Students receive their scores on 5 performance metrics compared to expert performance benchmarks. Scores are presented in the 5 minute breaks between tasks. Student goal is to be within the benchmark in all five metrics.
Experimental Group - Intelligent Continuous Expertise Monitoring System group
30 Participants. Introductory information provided on simulator and scenario. They perform 5 simple practice subpial tumor resections with 5 minutes per trial. On 6th attempt 13 minutes to perform a complex realistic scenario.
During first practice task, participants receive no feedback. For the subsequent 4 practice tasks participants will receive real-time auditory feedback instruction by the intelligent system. After each of the 5 attempts, a student takes a 5-minute break. During each of the 5 breaks the participants will be shown the errors they made during the task by the intelligent system regarding five performance metrics monitored. After seeing each error outline, the participant will be shown a video demonstration to learn how to expertly perform on each performance metric. On their 6th attempt they will perform on the realistic scenario without any feedback given.
干预措施: Experimental: Experimental Group - Intelligent Continuous Expertise Monitoring System group (Behavioral)
Experimental Group In-person expert-mediated instruction group
30 Participants. Introductory information provided on simulator and scenario. They perform 5 simple practice subpial tumor resections with 5 minutes per trial. On 6th attempt 13 minutes to perform a complex realistic scenario.
During first practice task participants receive no feedback. For the subsequent 4 practice tasks participants receive real-time auditory feedback instruction by in-person expert during the task. After each of the 5 tasks, students takes a 5-minute break. During each of the 5 breaks the in-person expert provides feedback to the participant based on their OSATS score assessment during the previous trial. If the expert feels it is appropriate the expert will demonstrate how to do the specific procedure which has been found to be a concern on the simulator themselves so the participant can understand how to improve their performance. On their 6th attempt they will perform on the realistic scenario without any feedback given.
干预措施: Experimental Group In-person expert-mediated instruction group (Behavioral)
结局指标
主要结局
Change in performance
时间窗: 1 Day of Study
Performance will be measured using the composite-score assessed by the ICEMS system
Transfer of learning
时间窗: 1 Day of Study
Performance on the complex realistic scenario will be evaluated by the composite-score assessed by the ICEMS system.
次要结局
- Objective Structured Assessment of Technical Skills (OSATS) global rating scale(1 Day of Study)
- Differences in strength of emotions elicited(1 Day of Study)
- Difference in Cognitive Load(1 Day of study)
研究者
Rolando Del Maestro
Director, Neurosurgical Simulation and Artificial Intelligence Learning Centre
McGill University