Media release

From: PLOS

AI tools fall short in predicting suicide, study finds

Analysis of 53 studies using machine learning to predict suicide and self-harm finds low accuracy

The accuracy of machine learning algorithms for predicting suicidal behavior is too low to be useful for screening or for prioritizing high-risk individuals for interventions, according to a new study published September 11^th in the open-access journal PLOS Medicine by Matthew Spittal of the University of Melbourne, Australia, and colleagues.

Numerous risk assessment scales have been developed over the past 50 years to identify patients at high risk of suicide or self-harm. In general, these scales have had poor predictive accuracy, but the availability of modern machine learning methods combined with electronic health record data has re-focused attention on developing new algorithms to predict suicide and self-harm.

In the new study, researchers undertook a systemic review and meta-analysis of 53 previous studies that used machine learning algorithms to predict suicide, self-harm and a combined suicide/self-harm outcome. In all, the studies involved more than 35 million medical records and nearly 250,000 cases of suicide or hospital-treated self-harm.

The team found that the algorithms had modest sensitivity and high specificity, or high percentages of people identified as low-risk who did not go on to self-harm or die by suicide. While the algorithms excel at identifying people who will not re-present for self-harm or die by suicide, they are generally poor at identifying those who will. Specifically, the researchers found that these algorithms wrongly classified as low risk more than half of those who subsequently presented to health services for self-harm or died by suicide. Among those classified as high-risk, only 6% subsequently died by suicide and less than 20% re-presented to health services for self-harm.

“We found that the predictive properties of these machine learning algorithms were poor and no better than traditional risk assessment scales,” the authors say. “The overall quality of the research in this area was poor, with most studies at either high or unclear risk of bias. There is insufficient evidence to warrant changing recommendations in current clinical practice guidelines.”

The authors add, “There is burgeoning interest in the ability of artificial intelligence and machine learning to accurately identify patients at high-risk of suicide and self-harm. Our research shows that the algorithms that have been developed poorly forecast who will die by suicide or re-present to health services for the treatment of self-harm and they have substantial false positive rates.”

The authors note, “Many clinical practice guidelines around the world strongly discourage the use of risk assessment for suicide and self-harm as the basis on which to allocate effective after-care interventions. Our study shows that machine learning algorithms do no better at predicting future suicidal behaviour than the traditional risk assessment tools that these guidelines were based on. We see no evidence to warrant changing these guidelines.”