Medical AI Models Expose Rare Disease and Minority Patients to Privacy Attacks, Nature Study Warns
Key Insights
A Nature study finds that membership inference attacks on medical AI models can achieve near-perfect success rates for certain individuals, revealing whether their data was used in training.
Patients with rare diseases, minority racial groups, and low-income populations face disproportionately higher privacy risks due to AI models memorizing their distinctive characteristics.
As model size increases, both diagnostic performance and privacy leakage risk rise together, with larger dermatology models increasing vulnerable patients from 1 in 10,000 to 1 in 10.
A new study published in Nature reveals that medical artificial intelligence systems harbor a troubling vulnerability: malicious actors can determine whether a specific patient's data was used to train a given model, potentially exposing sensitive health information without the patient ever knowing.
The research, led by Professor Daniel Rueckert at the Technical University of Munich (search) (TUM) in Germany, systematically analyzed so-called membership inference attacks (MIA) across seven clinical datasets, including chest X-rays, dermatology images, ophthalmology images, mammograms, electrocardiograms, and emergency department electronic medical records from institutions such as Stanford Hospital (search) and Harvard Medical School.
How Membership Inference Attacks Work
MIA exploits a fundamental property of AI models: they tend to make more confident predictions on data they have seen during training. Attackers can statistically analyze differences in a model's confidence scores to infer whether a particular patient's data was included in the training set. Because the attack requires only the model's prediction outputs, virtually anyone posing as a normal user can attempt it with relative ease.
The privacy risk stems from the nature of the training data itself. If an AI model is trained exclusively on cancer (search) patient data, then learning that a specific person's data was used for training effectively reveals that the person has cancer.
Disproportionate Risks for Vulnerable Populations
For each of the seven datasets, the researchers created 200 models, each trained on a random half of all patients. From an individual patient's perspective, half of the 200 models were trained on that person's data while the other half were not. By comparing confidence scores between the two groups, the team calculated privacy risk for each individual.
While the average attack success rate across entire datasets did not differ greatly from random guessing—suggesting low overall risk—the picture changed dramatically at the individual level. For some patients, attackers were able to perfectly infer whether their data had been used for training.
When the researchers extracted the top 1% of data points most vulnerable to attack, they found stark disparities. In the emergency department electronic medical records, data from Black patients appeared 31% more often in the high-risk group than in the overall dataset. Medicaid beneficiaries appeared 126% more often, and patients with cancer (search) diagnoses appeared 18% more often.
"The smaller a group's share in the overall data, the more distinctly the AI model memorizes its characteristics, thereby increasing the risk of privacy leakage," the team concluded.
Larger Models, Greater Risk
The study also found that privacy risk scales with model size. In the dermatology dataset, enlarging the model boosted the proportion of patients for whom attackers could be exactly correct from 1 in 10,000 to 1 in 10. This means diagnostic performance and privacy leakage risk rose in tandem, presenting a difficult trade-off for developers.
Proposed Countermeasures
As a defense, the researchers proposed differential privacy techniques—adding random noise during training so the model learns overall diagnostic patterns while blurring traces of individual records. They stressed that when a single patient contributes multiple records, privacy guarantees must be applied at the patient level rather than the record level to be effective.
"Given this vulnerability, medical AI models and their deployment contexts should be assessed for the sensitive information that attackers could obtain by successfully inferring training dataset membership," the researchers warned. "Models with higher risks should either incorporate proven privacy-preserving techniques or have their access strictly restricted."
The study, titled "Disparate privacy risks from medical AI," underscores that as medical AI systems become more widespread—particularly in regions where specialized medical expertise is scarce—the privacy implications demand urgent attention before widespread clinical deployment.