Diagnostic Accuracy of Large Language Models (GPT-4o and Claude) in HEART Score Calculation and 30-Day MACE Prediction in Emergency Department Chest Pain Patients: A Prospective Observational Validation Study Against Three-Expert Consensus

AI SYSTEM SPECIFICATIONS AND PROMPT PROTOCOL Two distinct large language models (LLMs) will be evaluated as index tests: OpenAI GPT-4o (model string: gpt-4o-2024-11-20) and Anthropic Claude (model string: claude-sonnet-4-6). To ensure reproducibility and eliminate stochastic variation, both models will be accessed via standardized API calls using deterministic parameters (temperature = 0, max_tokens = 500, and strict JSON response format). The exact system prompt layout will be locked prior to initialization, and its integrity will be verified using a SHA-256 cryptographic hash. The models will evaluate each patient record independently in zero-shot isolation, with no cross-contamination or conversational history retention between runs.

REFERENCE STANDARD CONSENSUS PROTOCOL The reference standard consists of a structured consensus HEART score established by three independent emergency medicine physicians (each possessing >=3 years of clinical experience and specific training on HEART score criteria). The physicians will review the anonymized clinical charts while remaining strictly blinded to the LLM outputs and the final 30-day MACE outcomes. For each of the 5 HEART components (scored 0, 1, or 2), a majority vote (2/3 agreement) will determine the final component score. In the event of complete disagreement across all three reviewers on a specific component, a fourth independent adjudicator will resolve the tie.

INDETERMINATE RESULTS MANAGEMENT

In strict compliance with STARD-AI 2025 guidelines, cases with missing or uninterpretable parameters within the free-text clinical notes will be classified into predefined indeterminate tiers:

1. Complete Cases: 0 indeterminate components (eligible for primary diagnostic accuracy analysis).
2. Partial Indeterminate: Exactly 1 missing component preventing definitive automatic calculation.
3. Full Indeterminate: >=2 missing components. The proportion of indeterminate classifications will be quantified for both LLMs and evaluated alongside the routine documentation quality of the charts.