soft.report

Physicians struggled to detect AI malfunctions in a randomized study

In a randomized study of 48 physicians published in npj Digital Medicine, diagnostic accuracy fell along with the reliability of a cardiac auscultation AI, and only 12.5% spontaneously noticed the system was failing.

Published Source date: November 13, 2026 soft.report Team

Human oversight is often presented as the main safeguard against errors made by artificial intelligence (AI) systems in medicine. A study published on November 13, 2026, in npj Digital Medicine put that assumption to the test, with concerning results.

What was done

German and Danish researchers ran a randomized experiment with 48 physicians (38% in training) on an AI-assisted cardiac auscultation task. Participants were told the system had 96% accuracy in validation (97% sensitivity and 89% specificity). Each classified 40 recorded heart sounds as normal or abnormal across four blocks.

Without the physicians knowing, the AI's reliability declined stepwise: 90%, 60%, 50% and 0%. In the final block, audio and visual cues were deliberately mismatched as an additional signal of severe malfunction.

Physicians were randomized to two groups:

  • AI-first: they saw the AI recommendation before hearing the sound and deciding.
  • AI-second (cognitive forcing): they made an independent classification first and then saw the recommendation, with the option to revise.

What they found

  • AI-assisted diagnostic performance fell from 81.5% in the first block to 58.3% in the last.
  • In the high-reliability block, combined physician-plus-AI performance was below the AI's own reliability (90%).
  • As the AI failed, physicians caught some errors, but performance dropped below what they achieved on their own without AI.
  • Deciding before seeing the AI did not improve results. In the final block, the AI-first group scored 64.6% and the AI-second group 52.1%.
  • Physicians' confidence in their own judgment declined faster than their trust in the AI, even though their independent performance did not worsen.
  • Afterwards, only 12.5% spontaneously reported noticing the drop in reliability; even when asked directly, only 37.5% said they had noticed.

Limitations

The study took place in a lab, on an isolated diagnostic subtask, without patient contact or other clinical information. The simulated malfunction was rapid and continuous, down to 0% accuracy, and blocks were always presented in the same order, so order effects cannot be ruled out. Real-world failures may be subtler or intermittent, and fatigue and time pressure are likely to make oversight even harder. The sample is small.

Why it matters in practice

Regulations such as the EU AI Act require human oversight of high-risk AI systems. This study suggests that simply putting a physician "in the loop" does not guarantee that errors will be caught. The authors argue that health systems should define clear goals for oversight and build processes for continuous performance monitoring, rather than relying only on individual vigilance.

For clinicians already using AI to screen images such as fundus photographs or OCT, the lesson is direct: trusting the advertised accuracy is not enough. Periodic audits and real-world performance metrics are part of patient safety.

Read the article in npj Digital Medicine.

Sources

  1. Challenges for human oversight in detecting system malfunctions in AI-assisted medical diagnostics — npj Digital Medicine, November 13, 2026

Informational content, written with AI assistance from the cited sources. It does not replace reading the original study or a medical evaluation.

Ophthalmology reports in your own template

The AI reads the exam, drafts the report in your template, and you review and sign. 15-day free trial, no card.

Discover soft.report