News release

From: PLOS

Doctors may have trouble learning from experience that contradicts AI advice

New study highlights potential challenges for using automated tools in healthcare

In experiments in which physicians made decisions about treating hypothetical patients, the physicians tended to trust incorrect advice presented as being generated by artificial intelligence (AI), even after given the opportunity to notice that patient recovery data contradicted the recommendations. Aranzazu Vinas of the University of the Basque Country, Spain, and colleagues present these findings in the open-access journal PLOS Digital Health.

AI systems can help physicians categorize patients according to their different care needs, such as whether a patient is more or less likely to benefit from a certain treatment. Since these systems are not perfect, they are meant to be used as suggestions, with potential errors caught and corrected by physicians.

Prior research has shown that, in general, people struggle to notice and correct mistakes made by AI. To explore how this challenge may extend to physicians, Vinas and colleagues analyzed data from 223 physicians who anonymously participated in online experiments.

The physicians were asked to imagine they had the option to treat patients for a rare disease using a not-yet-proven treatment still under development. They were told that an AI system had identified which patients were more or less likely to benefit from the treatment. The physicians then chose which patients to treat, and after being presented with data on patient recovery, rated their perceptions of how reliable the AI was.

Crucially, the actual effectiveness of the hypothetical treatment did not align with the AI recommendations. In one experiment, the treatment was equally moderately effective for all patients, and in a second experiment, it was equally ineffective for all.

However, in both experiments, the physicians tended to rate the AI system as reliable and apparently did not use the patient recovery data to conclude that the AI recommendations were incorrect. In the second experiment, the physicians did not realize that the treatment was entirely ineffective.

These findings highlight potential challenges for incorporating AI-based classification into healthcare. Future research could build on this study, such as by developing and testing strategies and protocols that could increase human critical thinking and detection of AI errors, in order to maximize the benefits of the human-AI collaboration while minimizing potential errors."

Lead author Aranzazu Vinas notes: " In both experiments, physicians mostly trusted the AI's classifications and had trouble learning from the feedback. Furthermore, in the second experiment, professionals did not notice that the treatment was completely ineffective."

Co-author Helena Matute adds, "People tend to say that there is always a human controlling the algorithm, but our experiments show that doctors (as well as anyone else) have problems in learning from the available evidence when it contradicts the suggestions of an algorithm."

Co-author Fernando Blanco summarizes: "It is important to investigate the errors that humans (including doctors) make when working with algorithms, in order to learn how to minimize the problems that arise from them."