Diagnostic Accuracy of Two Large Language Models Against a Blinded Specialist Consensus Standard in Turkish Emergency Department Notes: A Retrospective Study of 600 Cases

STUDY DESIGN: Retrospective diagnostic accuracy study, STARD-AI 2025 reporting, single centre, cohort design.

AI INDEX TESTS: (1) GPT-4.1 (model version gpt-4.1-2025-04-14; OpenAI API). (2) Claude Sonnet 4.6 (model version claude-sonnet-4-6; Anthropic API). Both accessed via the providers' developer application programming interfaces from Python. Temperature = 0. Zero-shot direct prompting with a single locked prompt version; stateless single-turn sessions with no cross-case context, no retrieval augmentation, no external tools and no extended-reasoning mode. No task-specific fine-tuning or additional training was applied; the models were used as released.

MODEL INTERPRETABILITY: Interpretability analyses such as SHAP, Grad-CAM or layer-attribution visualisation are not applicable to this study. Because GPT-4.1 and Claude Sonnet 4.6 are accessed as black-box models through proprietary, closed-source commercial interfaces, internal weights, gradients and attention structures are inaccessible for post-hoc interpretability computation.

REFERENCE STANDARD: Three board-certified emergency medicine specialists independently assess each anonymized note, blinded to one another, to the code entered by the treating physician, and to the subsequent clinical course. The primary diagnosis assigned by at least two of three assessors, reduced to ICD-10 chapter level, constitutes the reference standard. Cases in which all three assessors assign different chapters are excluded without replacement. No joint calibration session was held and no adjudication round was performed; each assessor coded once, according to their own clinical judgement.

DATA PRIVACY: All anamnesis notes are de-identified before processing; direct patient identifiers are removed and no patient name is present in any note at any stage. Each case carries a study-specific sequential number that is not a hospital record number, and no file linking study numbers to patient identities was created or retained. Note text is transmitted to commercial application programming interfaces operated by providers established outside Turkiye; all queries are issued through the providers' developer interfaces in stateless single-turn calls, and no patient identifier is present in any submitted text. De-identified notes are stored in an encrypted, access-restricted database. Conducted in accordance with Turkish Personal Data Protection Law no. 6698.