AI Medical Interviewing and Diagnostic System Performance Evaluation: One-Shot Vision Differential Diagnosis (OSVDE) and Multi-Step Conversational Non-Inferiority (MSCNE) Evaluation.

Artificial intelligence (AI) systems have demonstrated promising capabilities in medical diagnosis; however, rigorous benchmark evaluation is necessary prior to clinical deployment. AIMD.1 is a multimodal AI diagnostic system designed to assist with clinical reasoning through analysis of medical images and conversational diagnostic interactions.

This study is a benchmark performance evaluation of AIMD.1, a multimodal AI medical diagnostic system developed by Nolla Health and designed to assist clinical reasoning through analysis of medical images and structured conversational diagnostic interactions. The evaluation is conducted entirely in an offline research environment; the AI system is not used to guide real-world clinical care or patient management during this study. The study is designed as a benchmark performance evaluation prior to any prospective validation involving real patients.

The global healthcare system faces a significant workforce shortage, with projections suggesting a deficit of up to 11 million practitioners by 2030. AI systems for medical diagnosis have shown promise in addressing this gap in controlled research settings, but rigorous benchmark validation against clinician-level performance is needed before clinical deployment. This study addresses that need by evaluating AIMD.1 against both established AI benchmark systems and directly against board-certified clinicians completing the same diagnostic tasks.

The study employs two complementary evaluation stages designed to assess distinct aspects of diagnostic capability:

Stage 1 - One-Shot Vision Differential Evaluation (OSVDE): The AI system and clinician participants independently review individual de-identified medical images and generate ranked top-5 differential diagnoses based solely on visual features. The image corpus comprises approximately 11,500-15,000 de-identified images spanning at least 12 medical specialties (Dermatology, Internal Medicine, Otolaryngology, Gynecology, Orthopedics, Pediatrics, Geriatrics, Emergency Medicine, Ophthalmology, Endocrinology, Family Medicine, and others) and 48 disease clusters. Image sources include approximately 14,000 images retrieved from standard search engines (Google and Bing) and open access repositories such as the PMC Open Access Dataset, filtered for Creative Commons and public domain licensing, as well as approximately 1,000 de-identified clinical images provided by Nolla Health under terms of service permitting de-identified use for research purposes, with all images de-identified per HIPAA Safe Harbor standards. All source images undergo random combinations of affine and non-affine transformations (blurring, sharpening, contrast adjustment, color adjustment, pixel shifting, rotation, stretching, Gaussian noise, among others) to produce fundamentally distinct images from the originals while preserving clinically relevant visual features on average. Each transformed image is verified by at least one board-certified dermatologist or primary care clinician and relabeled as necessary, with ambiguous or non-clinically relevant images removed from the corpus. This preprocessing pipeline also provides additional de-identification through cropping or masking of potentially identifiable regions, including facial features. Images are localized using disease-name keywords defined in the study's disease ontology and downloaded in standard formats (JPEG, PNG) with associated metadata including ground-truth diagnostic labels and disease categories. Where available, additional metadata such as Fitzpatrick skin type (I-VI) and patient age range (pediatric, adult, geriatric) are recorded to enable subgroup analyses of diagnostic performance across demographic categories.