Patrick Holford and Contriving a Controversy: the Cochrane review of antioxidant supplements

Professor Patrick Holford of Teesside University and Head of Science and Education at Biocare has been out and about in the mainstream media, providing the industry response to the Cochrane Review of Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases (pdf). In this, he was joined by assorted other representatives of the supplement industry in an interesting example of manufactroversy or an artificially contrived controversy. Bertrand Russell wrote:

A stupid man’s report of what a clever man says can never be accurate, because he unconsciously translates what he hears into something he can understand.

A disappointing amount of the criticism reflected Russell.[1] Various critics implied that the Cochrane reviewers had been wrong to exclude various studies although the reviewers had followed a transparent, well-specified process and explained their reasoning within the review. They seem to have been criticised for only selecting trials that contained mortality data although mortality was the topic of the review.

Oddly, the reviewers were also responsible for all of the calculations and findings that industry critics accused them of distorting or overlooking. When discussing these findings, the industry spokespeople were careful to ignore the findings that even when you gather together all of the trials, and include those with poor methodology, the results indicate that the vitamins do not show any positive benefit for reducing mortality; the best that can be said is that there is no evidence of harm (the status of selenium is equivocal as the studies that reported reduced mortality were at high risk of bias influencing the outcome).

However, not withstanding the Russell-like misinterpretations, there seemed to be some concerted and purposeful strategy of confusion. Using tactics that are reminiscient of those used by Big Tobacco,[2] the critics were uncritically allowed to muddy the waters as to the quality of the review and were rarely challenged to specify a worked-out and substantiated specific criticism. Amidst the disinformation about the number of participants and the trials included or excluded in various analyses, the critics seemed to have overlooked the reviewers’ helpful elaborations.

More than two thirds of the included trials with more than 180,000 participants fall in the group of low bias risk trials. This highlights the validity of our results…

Our estimates of increased mortality in low-bias risk trials increased significantly when we excluded factorial trials as well as other trials with collateral interventions. These trials may all suffer from potential confounding from the collateral interventions. This highlights the potential dramatic public health consequences of our results.

A large number of unpublished trials on supplements may exist. Their results are more likely to have been either neutral or negative than to have shown beneficial effects…Accordingly, our estimates of increased mortality are likely to be conservative. [pg. 10 document; pg. 13 pdf. Emphasis added]

It is all the more remarkable when one considers that one of the most ubiquitous industry spokespeople was Patrick Holford. During the extended coverage on Radio 5, it seems as if Holford criticised the Cochrane Review while solemnly, and with no apparent sense of irony, offering his own anecdote of the benefits of supplements; a short-term ‘study’ with two participants that relied upon self-report. You can not parody this because it defies exaggeration.

Kruger and Dunning explain why it might be too much to expect Holford to form an appropriate assessment of his own level of comparative competence.[3]

Sarah Bosely has put together an overview of the vitamin pill industry faux outrage over the Cochrane Review. She acknowledges the expertise of the Cochrane reviewers and drily observes:

[their] findings have been met with howls of protest from the vitamin and health-food lobby who make a living by promoting supplements. Their denials are likely to give comfort to the large numbers of people who want to think vitamins make them healthier.

Her concluding paragraph is rather miserable but probably more accurate than I care to admit.

And so the debate goes on, with large numbers of people convinced a bit more of what is good for you can’t do you any harm and scientists using methodological techniques that only scientists understand to prove them wrong – only to be disbelieved. In the end, maybe the safest thing is just to eat a better diet.

My outstanding and unanswered question is whether it is possible to have a debate if one side refuses to try and understand why the stats of the other side make sense and constitute a fair test? Along with other industry spokespeople, Holford might have embraced an opportunity to educate the public on the merits of this form of systematic review and the importance of its findings for public health. They might have discussed their plans to upgrade their own education for clients and practitioners. However, none of this happened. Instead, the industry commenters seemed to refuse to accept the demonstrable merits of a systematic review in favour of their own interpretation of it. Beyond this, some of the commenters appeared to question the competence and independence of the Cochrane reviewers although the grounds for this were never made clear.

Yet again, is this an example of what Professor Holford will be teaching to the impressionable minds of students at the University of Teesside? Contriving a controversy and passing it off as informed criticism? Proferring an anecdote in preference to a systematic review that follows careful, agreed, and well-specified procedures?

Update: Added reference to Big Tobacco’s strategy of Doubt is their product and manufactured uncertainty.

Update 2: Effectively, the systematic review process is geared towards evaluating whether or not a particular study was a ‘fair test’. One of the best explanations of ‘fair tests’ is Evans, Thornton and Chalmers’ Testing Treatments (available for free download). We strongly recommend this generalist and entertaining book to interested parties as it explains all the ins-and-outs of what makes a fair test and the fair way in which to interpret the results.
Update 3 (18 April): We should hush our mouths because the celebrities have spoken.

Sir Cliff Richard, Gloria Hunniford, Jenny Seagrove and Carole Caplin have joined health industry experts in rejecting the widely publicised antioxidant review and reassuring consumers that concerns over these supplements are unfounded.

So, that should be that then. What do you mean they’re speaking a load of nonsense about the number of trials and that they are obviously reading someone else’s crib notes rather than read the paper for themselves?
Update 27 April: A Holford canard about the excluded studies is gaining surprising popularity: Are supplements harmful?

In this review, which is a rehash of their paper published last year in the Journal of American Medical Association (JAMA), they first excluded over 400 trials, many of which had positive outcomes, but had no deaths. They then decided which trials they liked (low risk bias) and did not like (high risk bias), a factor that has received criticism in mainstream medical journals

We responded earlier and a very patient Professor Gluud corrected Holford in the course of a Radio 5 item.

Essentially, Bjelakovic et al discuss their exclusion of 405 trials. These trials involved 40,000 participants: they were mostly small Phase I or II trials and short-term with little or no follow-up and no clinical outcome measures (pg. 8 (pdf), pg. 5 document). Bjelakovic et al contacted all of the authors to enquire whether there were any deaths. Around 20% of the authors replied and reported that there were no deaths in either arm of their studies. The authors did not exclude studies that were positive for the supplement arm but negative for the placebo or control arm.

Several people have expressed concern that eliminating the 405 studies “create[s] a bias in which the therapy on trial appears to be associated with deaths?”. However, these trials were, as the authors described above and only involved 40,000 rather than the 232,550 participants in the 67 trials that were assessed: 164,439 of these participants were healthy. You can read about the characteristics of the excluded studies in the review (pp. 118-42 (pdf), pg. 115-39 document).

The authors did take account of the 405 trials by performing a sensitivity analysis.

Therefore we performed exploratory analyses adding an imagined trial with 1 death and 20000 participants in each intervention group. The influence of zero events trials on our final result was not noticeable.

Their findings didn’t change.

Notes

[1] Some of the criticism might be more thoughtful but not as reported to date. Both selective7 and Woobegone direct our attention to Bernadine Healy’s remarks about the 2007 JAMA review. She mischaracterises the reviewers’ careful work as “blenderizing” and this has been adopted by others who have use it as a glib dismissal without explaining it further. And without explaining why such a criticism is sufficient grounds to dismiss the findings of this review.
Oddly enough, one of the recurrent criticisms of this review is that it concentrated on randomised controlled trials and did not include observational studies. There are some commenters such as Truswell who question whether the Cochrane Review process is appropriate for assessing what are essentially observational and cohort studies. These are legitimate issues that are soberly and thoughtfully discussed. On the other hand, we have Holford and other industry spokespeople, criticising the authors on the basis of their understanding or interpretation of this review.
[2] The Paranoid Style in American Science: Doubt is their product. Daniel Engber discusses Big Tobacco’s skilful use of the corporate strategy of “manufactured uncertainty” to deny that “scientific evidence has ever been produced, presented or submitted to prove conclusively that cigarette smoking causes cancer”.
[3] You may be interested to read an overview of the Justin Kruger and David Dunning paper that discusses how difficulties in understanding one’s own incompetence can lead to inflated self-assessments.

What’s even more amazing is that when they then shared the performance of other participants with the people who performed poorly (hoping that they would then adjust their self-perception downward) people who scored poorly failed to adjust their self-perception of their performance. In other words, they are completely unaware of their own [in]competence, and can’t detect competence in others.

It really it a very helpful paper that explains many otherwise inexplicable actions.

A Photon in the Darkness likewise offers a helpful discussion of this paper: The Arrogance of Ignorance

Advertisements

86 Comments

Filed under antioxidants, Holford, patrick holford, supplements

86 responses to “Patrick Holford and Contriving a Controversy: the Cochrane review of antioxidant supplements

  1. gaius

    You may describe the reviewers’ work as “careful” rather than “blenderizing”, but it is hard to argue with Dr Healy’s actual point, which is that:

    “…this study violates a cardinal rule of meta-analysis. Pooled studies must be compatible. That means combining apples and apples—or at the least, similar patients and comparable doses and duration of treatments.”

    To offset any accusation that I am misunderstanding my own incompetence, I freely admit to being statistically-challenged. I am so incompetent that I can’t even see that this matters in this case. Why? Because the reviewers “carefully” chose as an outcome for selection of the trials death — from any cause.

    Does this not have ANY bearing on their conclusion?

  2. gaius

    “There’s this vague idea – which has been going around for the past few centuries – that statistics is quite difficult. But in reality the maths is often the least of your problems: the tricky bit comes way before the number crunching, when you are deciding what to measure, how to measure it, and what those measurements mean.”
    — Dr Ben Goldacre
    http://www.badscience.net/?p=653#more-653

  3. Woobegone

    gaius : You seem to be jumping from one criticism to another. I think there is a sensible case to be made that combining data on the different antioxidants cannot be justified a priori, although in this case all of the results (except selenium) were very consistent so it probably worked out all right in the end. See my remarks linked to above…

    But then you attack the reviewers for using as their outcome all-cause mortality. They wanted to know whether antioxidants would increase or reduce your chance of dying, so they looked at all the studies pertinent to this question. Yes, it’s another question entirely whether vitamin C reduces your chances of getting a cold or whatever, this review wasn’t aiming to measure that, that wasn’t the purpose. I think most people would agree that an increased risk of death is a reason to avoid something, no matter how many colds you avoid before you prematurely die.

  4. gaius

    Thanks Woobegone,

    Not “jumping”, more like staggering non-sequiturally (?) from one to another; sorry about that.

    Dr Healy doesn’t say anything about combining different antioxidants in the sense of it not being valid to compare, say, vitamin C and vitamin E. What she does query is the validity of including trials that… tested a single vitamin as well as others that used multiple nutrients; used doses ranging from below the recommended daily intake to way beyond the tolerable upper limit; reported on people using the antioxidants for time scales ranging from days to years; and included people whose ages ranged all the way up to 103!

    On my second point, as I said, I am a statistical infant, but your comment has nothing to do with my question. To stumble towards an even simpler way of putting it… if I take 20 trials on antioxidants and from those pick the 10 trials in which there was at least one death (from ANY cause, even old age), then meta-analyse only them…isn’t it inevitable that one of my conclusions will be, hey, when people take antioxidants they die?

    I’m not grinding any axes here, just trying to increase my understanding.

  5. UK dietitian

    Ah, Holfordwatch

    You have managed to out-quote Holford, too

    well done

  6. Gaius

    There are several relevant points:

    1. Most of the trials they authors left out for not reporting mortality data were relatively small trials. They actually tell us approx total no. of subjects in all these 400 odd trials put together: about 40,000 people. And these trials didn’t REPORT mortality data; that doesn’t necessarily mean no-one in them died.

    The trials the authors analysed (mostly big ones) included many more people than that, close to 240,000.

    The bigger the trial, the more reliable the results.

    The authors actually also did the extra calculation “What if there had been a measurable low death rate in the populations in the trials we left out? How would it change the analysis and conclusions?”. The answer is – hardly at all.

    Anyway, the claims that the analysis “ignored 90% of the available evidence” is thus factually wrong. It would be more accurate to say:

    “The analysis included data from around 85% of all the test subjects who had participated in trials of antioxidants. ”

    2. The trials left out were also usually short-term. Some of the supplement lobby gang were complaining that this analysis only included trials that had 3 -year follow up, saying loudly “that’s obviously not long enough”. Well, the studies that they are complaining were “unfairly excluded” were largely much shorter-duration than 3 yrs. Err… shurely shome mistake?

    3. ALL the trials included in the analysis were randomized controlled trials; so each trial had a group of people taking antioxidants AND a group not. Thus the trials were ALWAYS comparing death rates between the two groups. There were deaths in both groups in the vast majority of the trials analysed. The overall death rates were about 13% of enrolled subjects in the taking-antioxidants gp and about 10% of enrolled subjects in the control group. Given the numbers of subjects in the big trials (see above) this means most studies will have been counting enough deaths to quantify population death rates with good accuracy.

    A word about small trials. They are useful for initial safety studies, and for experimental scientists and theory-builders. But to answer, with a reasonable degree of certainty, the overall question “Are supplements at the kind of doses people commonly take them good for you – yes or no?” small trials are utterly useless. Sometimes smaller trials can be combined, which is what meta-analysis does. But – if the small trials were not done carefully and in such a way as to exclude bias (double blinding, proper randomization) then the data are so shaky that including then in a bigger analysis just makes the results of the bigger analysis so “all over the place” that you can’t work out anything. It is an example of “garbage in, garbage out”.

    The not-so-hidden truth here is that lots of sloppily done small trials help the supplement people, as this allows them to claim positive effects of supplements (“Look at all these 100s of trials which report positive effects!”). But most of these apparent positive results are illusory, and disappear in bigger trials which rigorously exclude bias.

  7. gaius

    Dr Aust: thank you. You totally cleared up one of my major not-understandings with this:

    “ALL the trials included in the analysis were randomized controlled trials; so each trial had a group of people taking antioxidants AND a group not. Thus the trials were ALWAYS comparing death rates between the two groups. There were deaths in both groups in the vast majority of the trials analysed. The overall death rates were about 13% of enrolled subjects in the taking-antioxidants gp and about 10% of enrolled subjects in the control group.”

  8. Gaius – you might be interested in reading the Testing Times book we mention in update 2; it is well-written and available as a free download.

    I’m interested in general impressions of the media coverage of the Cochrane Review and the influence of the industry spokespeople and the truncated reports in the media. As a matter of interest, was it the commentary that you heard that led to the impression that the authors had, for no good reason, decided to cut out some trials without discussing their reasoning?

  9. Thank you, Dr Aust. Just to add to your point 1. The review authors wrote to the authors of all the studies that didn’t mention mortality data and they heard back from around 20%. They take account of these data.

    And, as you say in your discussion of the small trials, most of the trials in those small studies were either Phase 1 or Phase 2, both with negligible or short follow-up and without assessment of clinical outcome measures (pg 5 doc; pg 8 pdf).

    As for your point about the impression of 100s of positive trials, I found this observation to be particularly interesting:

    Our findings contradict the findings of observational studies, claiming that antioxidants improve health…Considering that more than 10% to 20% of the adult population(80 million to 160 million people) in North America and Europe may consume the assessed supplements…the public health consequences may be substantial. We are exposed to intense marketing. This is also reflected by the high number of publications per included randomised trial found in the present review. [pg 11 doc; pg 14 pdf)

  10. Professor UK dietitian – always an honour to have you stop by our small corner of the blogosphere.

  11. tifosi246

    From the substantial quotes and references I certainly get the strong impression that HolfordWatch and folk like Dr. Aust have actually read the Cochrane Review, or at least substantial chunks of it. No mean feat given its considerable length.

    On the other ear, having listened to Holford on Radio 5 and others railling against it, they don’t actually seem to have bothered to read it first before complaining; just snipped a few comparative figures, like the number of trials excluded, to support their a priori view and strapped on their megaphones.

    This systematic review was a serious, well considered and diligently executed piece of research and Dr. Gluud et al deserve more than just a perfunctory rejection that it is a cynical “stitch-up”.

  12. Wulfstan

    Not gaius – but I do wonder if some people have been unduly influenced by the canards put about by WDDTY and the like? ‘Antony’ cuts and pastes them over at LCN. This is a snippet.

    Despite its claims to be an independent review, the Cochrane study excluded 405 studies into vitamins because there were no deaths, and another 69 because they weren’t ‘randomised’ trials. As it is, the review looked only at studies that involved sick people, taking very high-dose synthetic vitamins, and which had participants dying. This does not replicate average use, and does not give the researchers the authority to claim that supplements shouldn’t be taken by healthy people.

    As far as I can tell, you and Dr Aust have dealt with most of that except for the same old nonsense about synthetic v. nature identical products.

  13. On the other ear, having listened to Holford on Radio 5 and others railling against it, they don’t actually seem to have bothered to read it first before complaining; just snipped a few comparative figures, like the number of trials excluded, to support their a priori view and strapped on their megaphones.

    I think that gets to the heart of how this has reported; it is just like a political story.

    The front-men, like PH, do not have the know-how to even read the study intelligently. They rely on a critique from someone more knowledgable. The originator for a lot of the criticism being bandied about by the pro-antioxidant folk is Prof Balz Frei of the Linus Pauling Institute at Oregon Health Sciences U. Prof Frei is a scientist of note, but it is fair to say he is also thoroughly pro-antioxidant – that is why he works for the Pauling Institute, which is “ab initio pro- supplement” . Anyway, his and one or two other folks’ (like Bernadine Healy’s) comments have been recycled ad nauseam by people like Patrick H who are, in effect, spokespersons – hired mouthpieces. Or, in my view, acting exactly like politicians, who do not understand the detail but are good with soundbites, and are “briefed” on what soundbites to use by the civil servants and media advisors at their elbow.

    A classic example of this is that some of the “criticisms” they use work directly against one another – e.g. the “they left loads of studies out” and “the analysis wasn’t long term enough”, as detailed above. The rationality of the critique is not the point – it is PR soundbites all the way.

  14. Wulfstan – we may have something to say on the topic of synthetic v. nature identical vitamins. But yes – a positive flock or gaggle of canards there. One wonders if they:
    understand randomised controlled trials and why they are A Good Thing;
    actually read the report where they would have learned that 21 of the trials included 164,439 healthy participants;
    read the reviewers’ transparent explanation for how they handled the 405 trials with zero mortality in both the experimental and control groups and for which the total number of participants in those 405 trials was about 40000.
    Finally, the reviewers wrote (with supporting references): “We confirmed that trials with inadequate bias control significantly overestimate intervention effects”. It doesn’t seem that the critics (should that be doubt-mongers?) want to engage with this fundamental argument that there is a general impression of a vast literature of studies with positive results (actually, a comparatively small number, but referenced over and over again and salami’d into different papers) and that many of the positive findings are grounded in a methodology that inflates the intervention effects.

  15. tifosi246 and Dr Aust – you are both right. There are striking correspondences between this and the Big Tobacco fomenting of scientific controversy that spawned the infamous ‘Doubt is our product’.

    The idea that Holford was spearheading this offensive was offensive. His own work is riddled with errors and he frequently displays an inability to handle even basic numbers yet he was casting aspersions on the value of this systematic review. Again, this stuff is beyond parody.

    As per Update 3 – now the celebrities are dispensing words of wisdom, we should all hush our mouths and cleanse our ears.

  16. Talking of Bernardine Healy, she has just recently used her column to stick her boots into the MMR / mercury / autism nightmare.

    *sigh*

    It is a shame since in other circumstances she is a good writer and has been known to talk significant sense, e.g. here.

    I think Healy’s brand of libertarian Republican “individualism” probably underlies her position on EBM and some other contemporary medical issues. I suspect she has an instinctive distrust of aspects of EBM because she sees bits of it as “creeping neo-statist authoritarianism from averages” and “dictating to individual patients and doctors”. But her critique of EBM, as others have pointed out, is full of flaws, and she seems to be losing the plot a bit – read her comments here about the infamously ludicrous Dave Holmes et al. “microfascism” argument.

    [NB – a reasoned riposte to Healy’s shots at EBM can be found here].

    The danger is that as an ex-Johns Hopkins Professor of Medicine, ex Dean of a medical school, and ex-Head of the NIH (although the latter is a primarily political appointment), Healy makes an attractive authority figure for the supplement crowd, as in:

    “The lid comes off! Mainstream authority figure speaks out to tell us anti-vaxers / antixoxidant freaks we were RIGHT ALL ALONG!!”.

    – read the comments after her column on mercury and vaccines and you will see what I mean.

  17. I’ve just read through the comments to the mercury-autism article and have been overcome by sadness.

    Next stop, Aids-HIV+ debunking plus 9/11 troofism?

  18. Pingback: Irish Association of Nutritional Therapy on the Cochrane Review of Antioxidant Supplements « Holford Watch: Patrick Holford, nutritionism and bad science

  19. Mike Maybury

    I am neither a highly qualified scientist nor expert statistician, so all you qualified people can stop reading now, if you wish.
    I have a simple point to make, not as complicated as a study or group of studies.
    ****************************************
    If a similar study was undertaken to see what link existed between mortality and high intake of medical drugs from doctors and hospitals, I guess that the conclusion might well be, in simple terms:
    The higher the quantity and variety of drugs and interventions used ,per day,the higher the death rate.
    ******************************************
    Some time limits would have to be applied, as some survivors might continue to take cocktails of drugs and treatments ( to the glee of the pharmaceutical industry!) for ages and ages.
    These patients might survive with some sort of existence, which could hardly be called living.
    As some people say, you can provide proof of almost anything with statistics.

  20. Wulfstan

    Mike – you probably know that your example isn’t a good analogy or related to this post.
    Nicking from another post about this:

    Effectively, the systematic review process is geared towards evaluating whether or not a particular study was a ‘fair test’. One of the best explanations of ‘fair tests’ is Evans, Thornton and Chalmers’ Testing Treatments (available for free download).

    I read it and it explains fair tests really well and still managed to make it a very enjoyable read.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s