Move over Dr Google, chatbots like ChatGPT can answer people's cancer questions accurately but they still have some issues

Embargoed until: Publicly released: 2023-08-25 01:00

International

Photo by Jonathan Kemper on Unsplash

Researchers asked four AI chatbots, including ChatGPT and Microsoft's Bing AI, to answer the five most common Google search queries for skin, lung, breast, colorectal, and prostate cancer, and found the responses to be accurate with no misinformation identified. However, the responses were written at a university reading level and performed poorly when it came to helping people identify what actions they could take based on the information. The authors say these limitations suggest that AI chatbots should still not be used as a primary source of medical information. A second study looked at the ability of chatbots to provide breast, prostate, and lung cancer treatment recommendations that fit with guidelines and they found that 12.5% of responses were 'hallucinated' and not part of any recommended treatment. The authors say doctors should advise patients that chatbots are not a reliable source of treatment information.

Media release

From: JAMA

Assessment of AI Chatbot Responses to Top Searched Queries About Cancer
JAMA Oncology

Editorial

Artificial Intelligence—From Starting Pilots to Scalable Privilege

JAMA Oncology
Research Letter
Use of Artificial Intelligence Chatbots for Cancer Treatment Information

About The Study: The findings of this study suggest that artificial intelligence (AI) chatbots generally produce accurate information for the top cancer-related search queries, but the responses are not readily actionable and are written at a college reading level. These limitations suggest that AI chatbots should be used supplementarily and not as a primary source for medical information.
Authors: Abdo E. Kabarriti, M.D., of the State University of New York Downstate Health Sciences University in New York, is the corresponding author.

(doi:10.1001/jamaoncol.2023.2947)
Editor’s Note: Please see the article for additional information, including other authors, author contributions and affiliations, conflict of interest and financial disclosures, and funding and support.
# # #

Attachments

Note: Not all attachments are visible to the general public. Research URLs will go live after the embargo ends.

Research JAMA, Web page Paper - 1 Please link to the article in online versions of your report (the URL will go live after the embargo ends).

Research JAMA, Web page Paper 2 - Please link to the article in online versions of your report (the URL will go live after the embargo ends).

Journal/
conference: JAMA Oncology

Research:Paper

Organisation/s: State University of New York, USA, Harvard Medical School, USA

Funder: The first study was supported by the Woods Foundation. In the second study Dr Loeb was supported by a grant from the National Cancer Institute (R01CA278997) and reported having a family member with equity in Gilead Sciences outside the submitted work.

Media Contact/s

Contact details are only visible to registered journalists.