ChatGPT finds nearly all 'complementary healthcare' websites include misinformation

Embargoed until: Publicly released: 2024-10-02 09:01

International

CC-0. https://www.pexels.com/photo/get-instant-answers-to-your-questions-with-chatgpt-your-pocket-ai-assistant-stay-connected-with-chatgpt-on-the-go-16689016/

UK scientists tasked ChatGPT-4 with identifying misleading claims on 'complementary' and 'alternative' medicine websites, and found nearly all of them include false or misleading information. They set ChatGPT loose on 725 websites representing 872 clinics, finding that 97% of sites included false or misleading claims, including some related to cancer treatment. To check ChatGPT's work, the team looked at a sample of 23 of the websites to see if they agreed with the artificial intelligence's (AI) judgments, finding an even higher proportion of misinformation than the chatbot. In that sample, the humans identified an average of 39.5 claims likely to be judged false or misleading by advertising regulators, while the AI identified 36. Chatbots could help regulators spot and take down online health-related misinformation, the authors conclude.

Media release

From: The Royal Society

Using Artificial Intelligence (AI) to Assess the Prevalence of False or Misleading Health-Related Claims

Research led by the charity HealthSense has found that almost all practitioners of selected therapies regulated by the Complementary and Natural Healthcare Council could be making claims that are not supported by evidence. An AI tool looking at a selection of pseudoscientific therapies found that 97% of working websites contained misleading claims, including some related to the treatment of cancer. The authors checked the quality of the AI judgments by assessing themselves a sample of the websites. They concluded that AI could be a useful means of improving health care regulation.

Attachments

Note: Not all attachments are visible to the general public. Research URLs will go live after the embargo ends.

Research The Royal Society, Web page The URL will go live at some point after the embargo ends

Journal/
conference: Royal Society Open Science

Research:Paper

Organisation/s: HealthSense UK, UK

Funder: A grant of £550 was donated by the organisation wePlanet.org to cover the cost of AI tokens.

Media Contact/s

Contact details are only visible to registered journalists.