Medical Associations Draw Red Lines: AI Must Remain a Tool, Not a Replacement for Physician Judgment
核心洞察
The Israel Medical Association (搜索) published a position paper establishing principles for integrating AI into clinical settings, emphasizing that AI systems must support rather than replace physician decision-making.
The American Medical Association adopted new policies calling for mandatory physician oversight, transparency, and accountability whenever AI is used in patient care or health insurance decisions.
Both organizations warn that AI risks include biased training data, false sense of security, hallucinations, and the erosion of the doctor-patient relationship if human context is sidelined.
Two major medical associations have independently drawn firm boundaries around the use of artificial intelligence in medicine, issuing guidelines and policies that converge on a single principle: AI must serve as an assistive tool under physician oversight, never as an autonomous decision-maker.
The Israel Medical Association (搜索) (IMA), through its Institute for Quality in Medicine and the Israeli Society for Risk Management and Patient Safety in Medicine, published a comprehensive position paper this month defining how medical organizations should integrate AI-based systems into clinical environments. Simultaneously, at its annual meeting, the American Medical Association (AMA) adopted new policies addressing AI's expanding role in both clinical practice and health insurance decision-making.
"AI has enormous potential in health care, but it cannot replace physician judgment," said John Whyte, MD, MPH, CEO of the AMA. "Patients deserve care decisions that are informed by the latest medical evidence and guided by a physician who understands their individual needs. Whether AI is helping a physician make a clinical decision or assisting with an insurance review, there must always be transparency, accountability and meaningful physician oversight."
The Rapid Spread of AI in Clinical Settings
The proliferation of AI in medicine has accelerated dramatically. As of May 2024, the US Food and Drug Administration has approved 882 medical devices utilizing artificial intelligence. Radiology dominates the field with 671 devices, representing 76% of all approvals, followed by cardiovascular diseases at 10%, neurology at 3%, hematology at 1.9%, gastroenterology and urology at 1.5%, and anesthesia at 1%.
In Israel, this transformation is already visible in practice. At Sheba Medical Center, the Aidoc (搜索) system—developed at the hospital—assists radiologists by scanning CT and X-ray imaging tests for urgent findings such as stroke, cerebral hemorrhage, pulmonary embolism, aortic dissection, or pneumothorax. The system flags suspicious scans, moving them to the top of the queue and marking areas of concern for the radiologist. The IMA position paper notes this does not replace medical interpretation but adds an important safety layer in environments where radiologists must review thousands of images under time pressure.
Another implementation, the Rounds (搜索) system, addresses physician burnout by recording medical visits, transcribing conversations, and generating full visit documents or medical summaries. This allows physicians to maintain eye contact with patients rather than diverting attention to screens during encounters. However, the IMA paper cautions that recording medical conversations demands strict privacy protections, information security measures, patient notification, and medical control to ensure generated summaries accurately reflect what occurred.
Clalit Health Services has deployed an AI system called AI-PRO (搜索), built on the C-Pi platform, which assists family doctors in proactive and personalized medicine. The system scans computerized medical records nightly, cross-references findings with clinical guidelines and knowledge bases, and surfaces recommendations—such as diabetic patients who have missed tests, patients with uncontrolled hypertension, or individuals at risk for osteoporosis. The decision remains with the physician: the system raises a flag, and the doctor determines whether to contact the patient, modify treatment, order tests, or disregard the recommendation.
The Risks: Bias, Hallucinations, and False Security
Both medical associations identify significant risks accompanying AI's promise. The IMA position paper highlights that AI systems generate answers based on the models and training data on which they were built. If that data is partial, biased, or unrepresentative of the patient population, recommendations may also be biased. A system trained predominantly on one population could produce errors when applied to patients from different age groups, ethnic origins, social backgrounds, or those with complex medical conditions insufficiently represented in training data.
A second risk is a false sense of security. When a computerized system delivers a fast, well-phrased, and confident answer, the temptation to rely on it can be substantial—particularly in crowded medical environments where physicians work under pressure, time constraints, and burnout. The IMA guidelines explicitly warn against "addiction" to AI and against situations where the physician stops seeing the patient and sees only the recommendation on the screen.
The AMA similarly emphasized concerns about "bias, long-term impact on both physicians and patient outcomes, explainability and transparency" in its press release. The organization stressed that AI technologies should only be used as an assistive tool rather than "an autonomous decision-maker."
Hallucinations—answers that appear reliable but are incorrect—represent another critical risk. In medicine, the IMA paper notes, such errors could manifest as inappropriate medication recommendations, wrong interpretations of test results, or failure to identify dangerous diagnoses. Consequently, the position paper requires organizations to define in advance which situations permit AI use, which do not, who is authorized to operate the system, and what level of human control is required.
Legal and Ethical Dimensions
The legal framework surrounding AI in medicine remains incompletely regulated. The IMA position paper raises unresolved questions: Who bears responsibility when a physician acts on an AI recommendation that proves wrong? Is informed consent required from patients for each AI use? Must patients know that a physician's answer was based on a computerized tool?
The Chairman of the Ethics Bureau of the Medical Association, quoted in the IMA paper during a Knesset discussion, stated unequivocally: "The doctor must disclose that the answer to the patient is based on artificial intelligence." This connects AI not only to accuracy but to the fundamental trust between doctor and patient.
AI in Insurance and Coverage Decisions
The AMA's new policies extend beyond clinical settings into health insurance, where AI is increasingly used to make coverage determinations. The organization is calling for regulations to guarantee that health coverage decisions based on AI are reviewed by physicians in appropriate fields, grounded in evidence-based, up-to-date medical information.
"When health plans use AI-driven tools to deny or delay care without explaining how those decisions were reached, physicians and patients are left in the dark," Whyte said. "AI should never function as an unaccountable black box. Health plans must be transparent about how these tools work, what evidence and data sources they rely on, and whether a qualified physician reviewed the decision."
The AMA is advocating for regular audits of AI-driven clinical review tools, including audits triggered by significant changes to training data, clinical guidelines, or AI models themselves, as well as comprehensive annual reviews to ensure continued alignment with standards of care. The policies also call for disclosure of any guidelines, data sources, or clinical logic used in adverse prior authorization decisions.
Implementation Requirements and Continuous Monitoring
The IMA position paper details a series of prerequisites before introducing AI systems into clinical use. Organizations must first verify that the system is truly necessary and improves a medical or administrative process without compromising safety. They must confirm regulatory approval or medical professional endorsement, even for systems not classified as medical devices. A risk management process must identify possible failure scenarios, define solutions, build an implementation plan, establish mechanisms for reporting unusual events, and train end users.
Training, the paper emphasizes, cannot be a generic presentation about AI. It must be practical: how to operate the system, what each alert means, what data enters it, what its limitations are, when consultation with an expert is mandatory, and how to document decisions made contrary to the system's recommendation.
Continuous monitoring is essential because AI models can change over time. Good performance at launch does not guarantee good performance a year later, particularly as patient populations change, treatment protocols are updated, or system data expands.
The Risk of Non-Implementation
The IMA paper also addresses the flip side: the risk of failing to implement AI. Systems capable of identifying early medical deterioration, detecting missing treatments, or alerting to life-threatening findings may prevent harm. According to the document, real-time monitoring systems already integrated into computerized medical records identified more than 10 times more cases of unusual events in real time and were capable of predicting medical deterioration up to 72 hours before its occurrence.
The AMA has committed to working with key stakeholders—including regulators, medical specialty societies, and AI developers—to create standards for evidence transparency, evaluation, attribution, validation, and explainability in systems that support clinical decision-making. The goal, according to the AMA release, is ensuring AI tools "reflect the principles of evidence-based medicine and provide physicians with information they can understand, evaluate and trust."
Both organizations converge on a shared conclusion: AI can serve as a safety layer—shortening time to interpretation, returning conversation time with patients to physicians, reminding of forgotten tests, identifying risk, and alerting to deterioration—but all of this requires clear boundaries, transparency, appropriate consent, information security, staff training, documentation, and continuous control. The physician remains the one who listens, examines, decides, explains, and bears professional responsibility.