Professor Patrick Holford of Teesside University and Head of Science and Education at Biocare has been out and about in the mainstream media, providing the industry response to the Cochrane Review of Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases (pdf). In this, he was joined by assorted other representatives of the supplement industry in an interesting example of manufactroversy or an artificially contrived controversy. Bertrand Russell wrote:
A stupid man’s report of what a clever man says can never be accurate, because he unconsciously translates what he hears into something he can understand.
A disappointing amount of the criticism reflected Russell. Various critics implied that the Cochrane reviewers had been wrong to exclude various studies although the reviewers had followed a transparent, well-specified process and explained their reasoning within the review. They seem to have been criticised for only selecting trials that contained mortality data although mortality was the topic of the review.
Oddly, the reviewers were also responsible for all of the calculations and findings that industry critics accused them of distorting or overlooking. When discussing these findings, the industry spokespeople were careful to ignore the findings that even when you gather together all of the trials, and include those with poor methodology, the results indicate that the vitamins do not show any positive benefit for reducing mortality; the best that can be said is that there is no evidence of harm (the status of selenium is equivocal as the studies that reported reduced mortality were at high risk of bias influencing the outcome).
However, not withstanding the Russell-like misinterpretations, there seemed to be some concerted and purposeful strategy of confusion. Using tactics that are reminiscient of those used by Big Tobacco, the critics were uncritically allowed to muddy the waters as to the quality of the review and were rarely challenged to specify a worked-out and substantiated specific criticism. Amidst the disinformation about the number of participants and the trials included or excluded in various analyses, the critics seemed to have overlooked the reviewers’ helpful elaborations.
More than two thirds of the included trials with more than 180,000 participants fall in the group of low bias risk trials. This highlights the validity of our results…
Our estimates of increased mortality in low-bias risk trials increased significantly when we excluded factorial trials as well as other trials with collateral interventions. These trials may all suffer from potential confounding from the collateral interventions. This highlights the potential dramatic public health consequences of our results.
A large number of unpublished trials on supplements may exist. Their results are more likely to have been either neutral or negative than to have shown beneficial effects…Accordingly, our estimates of increased mortality are likely to be conservative. [pg. 10 document; pg. 13 pdf. Emphasis added]
It is all the more remarkable when one considers that one of the most ubiquitous industry spokespeople was Patrick Holford. During the extended coverage on Radio 5, it seems as if Holford criticised the Cochrane Review while solemnly, and with no apparent sense of irony, offering his own anecdote of the benefits of supplements; a short-term ‘study’ with two participants that relied upon self-report. You can not parody this because it defies exaggeration.
Kruger and Dunning explain why it might be too much to expect Holford to form an appropriate assessment of his own level of comparative competence.
Sarah Bosely has put together an overview of the vitamin pill industry faux outrage over the Cochrane Review. She acknowledges the expertise of the Cochrane reviewers and drily observes:
[their] findings have been met with howls of protest from the vitamin and health-food lobby who make a living by promoting supplements. Their denials are likely to give comfort to the large numbers of people who want to think vitamins make them healthier.
Her concluding paragraph is rather miserable but probably more accurate than I care to admit.
And so the debate goes on, with large numbers of people convinced a bit more of what is good for you can’t do you any harm and scientists using methodological techniques that only scientists understand to prove them wrong – only to be disbelieved. In the end, maybe the safest thing is just to eat a better diet.
My outstanding and unanswered question is whether it is possible to have a debate if one side refuses to try and understand why the stats of the other side make sense and constitute a fair test? Along with other industry spokespeople, Holford might have embraced an opportunity to educate the public on the merits of this form of systematic review and the importance of its findings for public health. They might have discussed their plans to upgrade their own education for clients and practitioners. However, none of this happened. Instead, the industry commenters seemed to refuse to accept the demonstrable merits of a systematic review in favour of their own interpretation of it. Beyond this, some of the commenters appeared to question the competence and independence of the Cochrane reviewers although the grounds for this were never made clear.
Yet again, is this an example of what Professor Holford will be teaching to the impressionable minds of students at the University of Teesside? Contriving a controversy and passing it off as informed criticism? Proferring an anecdote in preference to a systematic review that follows careful, agreed, and well-specified procedures?
Update: Added reference to Big Tobacco’s strategy of Doubt is their product and manufactured uncertainty.
Update 2: Effectively, the systematic review process is geared towards evaluating whether or not a particular study was a ‘fair test’. One of the best explanations of ‘fair tests’ is Evans, Thornton and Chalmers’ Testing Treatments (available for free download). We strongly recommend this generalist and entertaining book to interested parties as it explains all the ins-and-outs of what makes a fair test and the fair way in which to interpret the results.
Update 3 (18 April): We should hush our mouths because the celebrities have spoken.
Sir Cliff Richard, Gloria Hunniford, Jenny Seagrove and Carole Caplin have joined health industry experts in rejecting the widely publicised antioxidant review and reassuring consumers that concerns over these supplements are unfounded.
So, that should be that then. What do you mean they’re speaking a load of nonsense about the number of trials and that they are obviously reading someone else’s crib notes rather than read the paper for themselves?
Update 27 April: A Holford canard about the excluded studies is gaining surprising popularity: Are supplements harmful?
In this review, which is a rehash of their paper published last year in the Journal of American Medical Association (JAMA), they first excluded over 400 trials, many of which had positive outcomes, but had no deaths. They then decided which trials they liked (low risk bias) and did not like (high risk bias), a factor that has received criticism in mainstream medical journals
We responded earlier and a very patient Professor Gluud corrected Holford in the course of a Radio 5 item.
Essentially, Bjelakovic et al discuss their exclusion of 405 trials. These trials involved 40,000 participants: they were mostly small Phase I or II trials and short-term with little or no follow-up and no clinical outcome measures (pg. 8 (pdf), pg. 5 document). Bjelakovic et al contacted all of the authors to enquire whether there were any deaths. Around 20% of the authors replied and reported that there were no deaths in either arm of their studies. The authors did not exclude studies that were positive for the supplement arm but negative for the placebo or control arm.
Several people have expressed concern that eliminating the 405 studies “create[s] a bias in which the therapy on trial appears to be associated with deaths?”. However, these trials were, as the authors described above and only involved 40,000 rather than the 232,550 participants in the 67 trials that were assessed: 164,439 of these participants were healthy. You can read about the characteristics of the excluded studies in the review (pp. 118-42 (pdf), pg. 115-39 document).
The authors did take account of the 405 trials by performing a sensitivity analysis.
Therefore we performed exploratory analyses adding an imagined trial with 1 death and 20000 participants in each intervention group. The influence of zero events trials on our final result was not noticeable.
Their findings didn’t change.
 Some of the criticism might be more thoughtful but not as reported to date. Both selective7 and Woobegone direct our attention to Bernadine Healy’s remarks about the 2007 JAMA review. She mischaracterises the reviewers’ careful work as “blenderizing” and this has been adopted by others who have use it as a glib dismissal without explaining it further. And without explaining why such a criticism is sufficient grounds to dismiss the findings of this review.
Oddly enough, one of the recurrent criticisms of this review is that it concentrated on randomised controlled trials and did not include observational studies. There are some commenters such as Truswell who question whether the Cochrane Review process is appropriate for assessing what are essentially observational and cohort studies. These are legitimate issues that are soberly and thoughtfully discussed. On the other hand, we have Holford and other industry spokespeople, criticising the authors on the basis of their understanding or interpretation of this review.
 The Paranoid Style in American Science: Doubt is their product. Daniel Engber discusses Big Tobacco’s skilful use of the corporate strategy of “manufactured uncertainty” to deny that “scientific evidence has ever been produced, presented or submitted to prove conclusively that cigarette smoking causes cancer”.
 You may be interested to read an overview of the Justin Kruger and David Dunning paper that discusses how difficulties in understanding one’s own incompetence can lead to inflated self-assessments.
What’s even more amazing is that when they then shared the performance of other participants with the people who performed poorly (hoping that they would then adjust their self-perception downward) people who scored poorly failed to adjust their self-perception of their performance. In other words, they are completely unaware of their own [in]competence, and can’t detect competence in others.
It really it a very helpful paper that explains many otherwise inexplicable actions.
A Photon in the Darkness likewise offers a helpful discussion of this paper: The Arrogance of Ignorance