AI chatbot health advice is vulnerable to deliberately malicious prompts

Embargoed until: Publicly released: 2025-12-20 03:00

International

What does this mean?

Jump to: MultimediaAttachmentsExpert Reaction

Korean scientists investigated three popular chatbots people turn to for medical advice - ChatGPT,  Gemini, and Claude - to see if they could be made to dish out dodgy recommendations when the prompts used were malicious attempts to manipulate their behaviour. The prompts were either designed to include specific information about a patient to promote convincing-sounding moderate or high risk recommendations, or false study results or guidelines to trigger extremely high risk recommendations, like telling pregnant women to take drugs that are not safe during pregnancy, or combining drugs that are risky when taken together. They found this succeeded in generating bad advice in nearly all (94.4%) of the moderate or high risk cases, and in almost as many (91.7%) of the extremely high harm risk cases. The latter included recommending pregnant women take thalidomide, a drug that can cause severe birth defects if taken while pregnant. The findings suggest chatbot safeguards can't handle malicious attempts to manipulate the health advice they dole out, the researchers say, which could be putting lives at risk.