Comparison of Artificial Intelligence-Generated and Academician-Developed Multiple True/False Questions in Anesthesiology Education: A Prospective Cohort Study
试验速览
- 阶段
- 不适用
- 状态
- 已完成
- 发起方
- 入组人数
- 26
- 试验地点
- 1
- 主要终点
- Item Difficulty Index of AI-generated and expert-authored questions
研究概览
简要总结
This prospective observational study aims to evaluate the effectiveness and educational value of artificial intelligence (AI)-generated multiple true/false questions compared to those developed by experienced academicians in anesthesiology training.
A total of 27 anesthesiology residents will be included in the study. Question sets consisting of 200 multiple true/false items will be created, with half generated by academicians and the other half generated using an artificial intelligence model (ChatGPT-based system). The questions will be based on standardized educational materials from the anesthesiology training curriculum.
Participants will complete the test in a single session. Each correct answer will be scored as one point, and total scores will be calculated. In addition to test performance, item difficulty, discrimination indices, and test reliability will be analyzed. Furthermore, participants' perceptions regarding question quality will be evaluated.
The study aims to determine whether AI-generated questions can provide a reliable and effective alternative to traditional question development methods in medical education and contribute to more objective and standardized assessment processes.
详细描述
This single-center, prospective observational cohort study is designed to evaluate the effectiveness, reliability, and educational value of artificial intelligence (AI)-generated multiple true/false (MTF) questions compared to those developed by experienced academicians in anesthesiology training.
The study will be conducted at the Department of Anesthesiology and Reanimation, Kütahya Health Sciences University. A total of 27 anesthesiology residents will be included.
A total of 200 MTF questions will be developed based on standardized anesthesiology educational materials. Half of the questions (n=100) will be prepared by experienced academicians, while the remaining half (n=100) will be generated using an artificial intelligence model (ChatGPT-based system). All questions will be structured according to predefined criteria, including difficulty level (easy, moderate, difficult), clinical relevance, and educational appropriateness.
Participants will complete the question sets in a single session under standardized conditions. Each correct answer will be scored as 1 point, and incorrect answers will be scored as 0. Total test scores will be calculated for each participant.
Item analysis will be performed to evaluate the psychometric properties of the questions. Item difficulty index, item discrimination index, and overall test reliability will be calculated. Additionally, perceived question quality will be assessed using participant feedback.
研究设计
- 研究类型
- Observational
- 观察模型
- Cohort
- 时间视角
- Prospective
入排标准
- 年龄范围
- 18 Years 至 —（Adult, Older Adult）
- 性别
- All
- 接受健康志愿者
- 是
入选标准
- •Being an anesthesiology resident
- •Voluntary participation in the study
- •Providing informed consent
排除标准
- •Refusal to participate
- •Incomplete test responses
研究组 & 干预措施
AI-Generated Questions
Multiple true/false questions generated using an artificial intelligence model
Academician-Developed Question
Multiple true/false questions prepared by experienced academicians in anesthesiology.
结局指标
主要结局
Item Difficulty Index of AI-generated and expert-authored questions
时间窗: Assessed once after completion of each participant's single 60-minute examination session; final item analysis performed after all participants complete the examination, up to 1 month.
For each question, the item difficulty index will be calculated as the proportion of participants who answer the item correctly. Item difficulty indices will be compared between AI-generated and expert-authored questions.
次要结局
未报告次要终点
研究者
Serkan TELLİ
Assistant Professor of Anesthesiology
Kutahya Health Sciences University