Al to Improve the Diagnosis of Rare Rheumatic Diseases

Advanced artificial intelligence (AI) technologies, particularly large language models such as OpenAI's ChatGPT, hold significant potential for enhancing medical decision-making. While ChatGPT was not specifically designed for medical applications, it has shown utility in various healthcare scenarios, including answering patient inquiries, drafting medical documentation, and aiding consultations. Despite these advancements, its role in supporting diagnostic reasoning-especially among less experienced medical students-and for complex rare diseases remains underexplored.

Diagnostic reasoning is a multifaceted process that combines pattern recognition, knowledge synthesis, and probabilistic thinking. Tools like ChatGPT could potentially alleviate cognitive burden, enhance diagnostic accuracy, and ultimately accelerate the diagnosis for rare diseases. However, ChatGPT is not tailored for diagnostic reasoning and lacks comprehensive validation in this domain. Additionally, it is susceptible to generating misinformation or plausible-sounding but inaccurate responses, which may hinder rather than support clinical decision-making. Therefore, understanding how medical students utilize such AI tools is essential before they are integrated into educational or clinical workflows. This study will also assess a standardized prompt to facilitate ChatGPT usage and will give students direct access to enable a realistic scenario.

This study will investigate the impact of ChatGPT on the diagnostic accuracy of medical students when tackling cases of rare rheumatic diseases. Participants will be randomized into two groups: one with access to ChatGPT and one using conventional diagnostic tools. Each participant will analyze diagnostic cases by providing up to 5 differential diagnoses and and rating the diagnostic confidence. Independent reviewers, blinded to group allocation, will evaluate the accuracy and quality of their responses. This study hence aims to provide insights into the potential benefits and limitations of integrating AI tools like ChatGPT.