Effect of Artificial Intelligence-Augmented Human Instruction on Surgical Simulation Performance: A Randomized Controlled Trial
Trial Snapshot
- Phase
- Not Applicable
- Status
- Completed
- Sponsor
- McGill University
- Enrollment
- 88
- Locations
- 2
- Primary Endpoint
- Intelligent Continuous Expertise Monitoring System (ICEMS) expertise score - Technical skill acquisition across practice tasks on NeuroVR simulator
Study Overview
Brief Summary
At the Neurosurgical Simulation and Artificial Intelligence Learning Centre, we seek to provide surgical trainees with innovative technologies that allow them to improve their surgical technical skills in risk-free environments, potentially improving patient operative outcomes. The Intelligent Continuous Expertise Monitoring System (ICEMS), a deep learning application that assesses and trains neurosurgical technical skill and provides continuous intraoperative feedback, is one such technology that may improve surgical education.
In this randomized controlled trial, medical students from four Quebec universities will be blinded and randomized to one of three groups (one control and two experimental). Group 1 (control) will be provided with verbal AI tutor feedback based on the ICEMS error detection. Group 2 will be tutored by a human instructor who will receive ICEMS error data and deliver verbal instruction using the same words as the ICEMS. Group 3 will be tutored by a human instructor who will be provided with ICEMS data and will then deliver personalized feedback.
The aim of this study is to determine how the method of delivery of verbal surgical error instruction influences trainee technical skill acquisition and transfer. Evaluating trainee responses to AI instructor verbal feedback as compared to feedback from human instructors will allow for further development, testing, and optimization of the ICEMS and other AI tutoring systems.
Detailed Description
Background: Expert surgical technical skill is linked with improved patient outcomes; however, training novices to master these skills remains challenging. The Intelligent Continuous Expertise Monitoring System (ICEMS) is a deep learning application that was developed at the Neurosurgical Simulation and Artificial Intelligence Learning Centre to improve neurosurgical education. The ICEMS assesses and trains bimanual surgical performance by providing continuous feedback via verbal instructions in order to improve trainee performance and mitigate errors.
Rationale: A previous randomized controlled trial (RCT) performed at our centre demonstrated that intelligent tutoring is more effective than expert tutoring in a simulated neurosurgical procedure (NCT05168150). However, during this study, expert instructors were not provided with ICEMS error data. Conducting a new RCT in which expert instructors are provided with quantitative ICEMS error data will allow us to determine the most effective method for teaching surgical technical skills to trainees in virtual operative procedures.
This report follows the Consolidated Standards of Reporting Trials-Artificial Intelligence (CONSORT-AI) as well as the Machine Learning to Assess Surgical Expertise (MLASE) checklist.
Hypotheses:
- AI-augmented personalized expert instruction will be more effective at improving technical skill acquisition and skill transfer in trainees compared to AI tutor instruction.
- Instruction delivered by human instructors will result in lower levels of negative emotions and cognitive load compared with instruction delivered by the AI tutor.
Study Design
- Study Type
- Interventional
- Allocation
- Randomized
- Intervention Model
- Parallel
- Primary Purpose
- Health Services Research
- Masking
- Single (Participant)
Masking Description
Single (Participant)
Study participants are blinded to group assignments and study outcomes.
Eligibility Criteria
- Ages
- 18 Years to — (Adult, Older Adult)
- Sex
- All
- Accepts Healthy Volunteers
- Yes
Inclusion Criteria
- •Medical students who are actively enrolled in their preparatory, first, second year of medical school at any Quebec institution who do not fit the exclusion criteria.
Exclusion Criteria
- •Prior use of the NeuroVR (CAE Healthcare) simulator.
Arms & Interventions
AI tutor instruction group
31 participants allocated. During their second, third, fourth, and fifth repetition of the practice subpial brain tumor resection scenario, participants will receive verbal ICEMS feedback when the system detects an error on their performance.
Expert instruction group
29 participants allocated. During their second, third, fourth, and fifth repetition of the practice subpial brain tumor resection scenario, participants will receive verbal feedback from an expert instructor. The expert instructor will deliver this feedback using the same words as the ICEMS.
Intervention: Expert instruction using AI tutor script (Behavioral)
Personalized expert instruction group
28 participants allocated. During their second, third, fourth, and fifth repetition of the practice subpial brain tumor resection scenario, participants will receive verbal feedback from an expert instructor. The expert instructor will use their expertise to deliver personalized feedback to the participant.
Intervention: AI-augmented personalized expert instruction (Behavioral)
Outcomes
Primary Outcomes
Intelligent Continuous Expertise Monitoring System (ICEMS) expertise score - Technical skill acquisition across practice tasks on NeuroVR simulator
Time Frame: 1 day of study
The ICEMS will continuously evaluate the trainee's performance during each practice task and calculate average expertise scores on a scale of -1.00 (novice) to 1.00 (expert). This will allow us to assess learner technical skill acquisition from the first through sixth repetitions of the practice task.
Intelligent Continuous Expertise Monitoring System (ICEMS) expertise score - Technical skill transfer during complex realistic task on NeuroVR simulator
Time Frame: 1 day of study
The ICEMS will continuously evaluate the trainee's performance during the realistic task and calculate an average expertise score on a scale of -1.00 (novice) to 1.00 (expert). This will allow us to assess learner technical skill transfer from the practice tasks to a more complex realistic scenario.
Secondary Outcomes
- Strength of emotions elicited(1 day of study)
- Levels of cognitive load(1 day of study)
Investigators
Rolando Del Maestro
Director of the Neurosurgical Simulation and Artificial Intelligence Learning Centre
McGill University