EXPERT REACTION: Acupuncture could offer relief from menopause

As the authors state, the major limitation is that the study was not blinded. It's curious that they say the outcome assessors were blinded, when the outcomes were all patient-reported. The patients were not of course blinded.

I am not sure it's fair to say that there is no validated sham acupuncture technique - I found several such studies, (e.g. https://www.ncbi.nlm.nih.gov/pubmed/20351375). I suppose it depends on what you mean by `validated', and the authors take the view that there will always be some residual effect of sham acupuncture. But the alternative view is that both real and sham are placebos. I don't think this study dismisses the possibility that acupuncture is a theatrical placebo.

Hot flushes are hugely troublesome for women going through menopause and can affect their ability to sleep as well as general well-being. So, more treatments are certainly welcome to help treat these women. We know that psychological treatments like cognitive behavioural therapy (CBT) can be helpful, so the findings seems plausible, however it is difficult from the way the study was carried out to make any firm conclusions about how effective acupuncture is for the treatment of hot flushes. Whilst it is not a drug, acupuncture does still have a cost attached, and therefore it is important to be sure that this treatment actually works before we recommend it. Although the study was randomised, which is a strength, it was not ‘blinded’ meaning that patients knew whether they were receiving the treatment or not. This means that the benefit observed could have been due to the ‘placebo’ effect, whereby people may experience a benefit because they are expecting one as they know that they are receiving a treatment. In order to get around this, it would have been necessary to do a ‘sham' or ‘fake' acupuncture in the control group, so that patients in both groups of the study believed that they could be receiving the treatment and the placebo effect would have been equally present in both groups.

Certainly, as acupuncture is not particularly harmful, and if people want to try it and are willing to pay for it, then it is fine to do so, but this study is not able to provide strong enough evidence that it definitely works for treatment of hot flushes.

Menopausal flushes will affect most women at some point. Hormone replacement therapy (HRT) is an excellent treatment but is not tolerated by all women. Also, HRT cannot be given to women with certain type of cancer. This leave a huge ‘gap in the market’ for simple and safe therapies to reduce hot flush symptoms. The authors show that acupuncture can reduce flush symptoms. However, we also know that flush symptoms can more than halve with a placebo. I suspect that most (if not all) of the effect of acupuncture is a placebo effect; but is that a really bad thing? Anything that is safe, affordable and helps the well-being of patients while reducing symptoms is worth considering if HRT is not an option.

It's been established for at least 70 years that blinding is an absolutely essential part of clinical trial design, and it's deeply shocking that the BMJ should publish a test of acupuncture in which the patients were aware of which group they were in.  There are at least two ways in which sham acupuncture could have been done. The fact that it wasn't means that this study is as likely to be misleading as it is to be helpful.

Given that there have already been well over 3500 trials of acupuncture, the best of which show little or no difference between "real" acupuncture and sham, the prior probability of there being a real effect in this trial must be quite low, and the corresponding risk of the results being false positives is high.  The authors themselves cite a Cochrane review which concluded that there was no good evidence for acupuncture helping with menopausal problems. They also cite (ref 41) a far bigger trial with proper sham controls that showed no effect whatsoever of acupuncture on menopausal hot flushes. 

It beats me why, in the light of all this previous information, the authors should have spent time on yet another small and poorly-controlled trial. And it's even more puzzling why the BMJ should publish it.”

A technical note:

There are also some statistical problems. It's stated that corrections were made for multiple comparisons - "the false discovery rate [was controlled] at 5% with the method of Benjamini and Hochberg)." 

But in fact in Table 3, and in the abstract, uncorrected p values are reported.  The only mention of the correction is in a footnote to Table 3 - "‡Significant at a 0.01 level to control for the false discovery rate at 0.05."

No details are supplied but this comment suggests that p = 0.01 has been chosen as a threshold for declaring "statistical significance".  Since 12 comparisons are made Table 3, the Bonferroni correction would give p = 0.05/12 = 0.004. 

Furthermore all this gives you is a corrected p value.  It is now very well established that p values (as commonly misinterpreted) overstates the evidence against the null hypothesis.

‘Pragmatic’ trials study treatment effects as they would occur in real-life settings, without excluding things like placebo effects. We know that treatments that are elaborate, prolonged or invasive can have heightened placebo effects, and acupuncture is all of these. But placebo effects are real nonetheless, and can translate into measurable physiological outcomes—outcomes that were unfortunately ignored in this study. Patient-reported outcomes are important, but also much more susceptible to nonspecific effects such as placebo. Placebo effects are particularly valuable for conditions with few evidence-based treatments available, such as hot flushes, and for people who are unable to tolerate other treatments. It is therefore disappointing that this study excluded people who had received other treatments (even herbal remedies). In fact more people were excluded from the study than were included, and this not-so-pragmatic element of study design may limit the generalisability of findings, especially in light of how many people experience moderate-to-severe hot flushes during menopause.

Like much of acupuncture research, this is very poor science, in my view.  The study had no control group capable of accounting for the often powerful placebo effects of acupuncture.  Therefore, the observed outcomes could be (and most probably are) entirely unrelated to acupuncture per se.  The authors nevertheless conclude that ‘acupuncture treatment produced a fast and clinically relevant reduction in moderate-to-severe menopausal symptoms’ – this conclusion is not backed up by evidence and so researchers, policy makers and patients may be misled.  I am puzzled to see that the BMJ-Open publishes such a study.