Patrick Holford and His Own Reality: Part 2, estimating risk bias in Cochrane reviews

Professor Patrick Holford of Teesside University and Head of Science and Education at Biocare regularly describes his own special reality when handing out his criticisms of skilled researchers. As such, Holford’s response to the latest Cochrane Review of Antioxidants (pdf) is a classic: Are supplements harmful?

This is the fourth time Bjekalovic and his group have reviewed the effects on selected studies on antioxidants. Anyone following the science of antioxidants over the past 20 years will be aware of a vast number of studies reporting positive results. So, how do you end up with a headline that implies antioxidants increase mortality?

In this review, which is a rehash of their paper published last year in the Journal of American Medical Association (JAMA), they first excluded over 400 trials, many of which had positive outcomes, but had no deaths. They then decided which trials they liked (low risk bias) and did not like (high risk bias), a factor that has received criticism in mainstream medical journals

Yes, yet again, Holford has displayed his astonishing ignorance of just how rigorous the specification is for a systematic review. What does Holford think that they do? Drag themselves into an office and throw darts or flip coins? Does it sound even vaguely plausible to you that the much-admired Cochrane Reviews depend upon such subjective assessments as ‘liking’ a paper? “Yes – I like the sound of this one but not this? The vowels in the authors’ names, they are too louche. We shall describe those papers as high risk bias.”

Is the process of assessing risk bias an esoteric art that is concealed from non-initiates? Well, it is detailed and laborious but there is a rigorous and transparent process for deciding risk of bias and you can read all about on pg. 4 of the review (pg. 7 in the pdf page counter).

Trials with adequate generation of the allocation sequence, adequate allocation concealment, adequate blinding, and adequate follow-up were considered low-bias risk trials (high methodological quality)…We appended ’Low’ to the references of these trials. Trials with one or more unclear or inadequate quality components were classified as high-bias risk trials (low methodological quality)…

You will find a fuller description and more detail in Chapters 8, 12 and elsewhere (particularly 13.5 onwards) in the Cochrane Handbook. Anyone can look at the Handbook, it is freely available to all interested parties and you don’t even have to register to look through it.

Chapter 8 provides a detailed discussion of study-level assessments of risk of bias in the context of a Cochrane review. It describes an approach to assessing the risk of bias that affect outcome across studies: these are classified as ‘low risk of bias’, ‘unclear risk of bias’ and ‘high risk of bias’ (Chapter 8, Section 8.7). For example, ‘low risk of bias’ would indicate ‘no limitation’; ‘unclear risk of bias’ would indicate either ‘no limitation’ or ‘serious limitation’; and ‘high risk of bias’ would indicate either ‘serious limitation’ or ‘very serious limitation’.

It is true that appropriately experienced researchers must use their informed judgement to decide between these categories, depending on the likely magnitude of the potential biases. However, it is because the reviewers are aware of a risk of subjectivity that such assessments are carried out independently by at least two people who then meet, typically with another researcher, and come to a decision on the papers where there is a difference of assessment.

It is a given that any study addressing a particular outcome will not be perfect: e.g., it may be improbable that some research can ever be done definitively because you would never obtain clearance from an ethics review board; some external factor might change during the period of a study such as the fortification of food which might alter your assessments. So all studies differ, to some degree, in the risk of bias. Review authors make an overall judgement as to whether the quality of evidence for an outcome should is such that one would not place too much confidence in it. Authors of a systematic review use well-defined, transparent and open procedures. We will look into this more but it is notable that Holford has not given a single example where he could specify his point of disagreement of risk bias where his evaluation differs from that of the authors. (We would also note that the NHS NLH finds the Cochrane Review to be “well conducted and thorough”.)

Effectively, the systematic review process is geared towards evaluating whether or not a particular study was a ‘fair test’. One of the best explanations of ‘fair tests’ is Evans, Thornton and Chalmers’ Testing Treatments (available for free download). We strongly recommend this book to Holford as it explains all the ins-and-outs of what makes a fair test and the fair way in which to interpret the results. Once Holford actually understands what is involved and just how rigorous the process is, perhaps he will be less glib about criticising the competence or integrity of researchers. Kruger and Dunning explain why it might be too much to expect Holford to form an appropriate assessment of his own level of comparative competence.[1] Yet again, is this an example of what Professor Holford will be teaching to the impressionable minds of students at the University of Teesside?

Notes

[1] You may be interested to read an overview of the Justin Kruger and David Dunning paper that discusses how difficulties in understanding one’s own incompetence can lead to inflated self-assessments.

What’s even more amazing is that when they then shared the performance of other participants with the people who performed poorly (hoping that they would then adjust their self-perception downward) people who scored poorly failed to adjust their self-perception of their performance. In other words, they are completely unaware of their own [in]competence, and can’t detect competence in others.

It really it a very helpful paper that explains many otherwise inexplicable actions.

A Photon in the Darkness likewise offers a helpful discussion of this paper: The Arrogance of Ignorance

Advertisements

19 Comments

Filed under Holford, patrick holford, supplements

19 responses to “Patrick Holford and His Own Reality: Part 2, estimating risk bias in Cochrane reviews

  1. Pingback: Catherine Collins: “Patrick has [given] an absolutely perfect example of why one should be wary of nutritional therapists.” « Holford Watch: Patrick Holford, nutritionism and bad science

  2. tifosi246

    Hmm, so yet another “I don’t like the result so I’ll question the integrity of the researchers” from PH, who’s own so-called research is of such a uniform high-quality and written up in a wonderful clear and concise style that makes the basis for his recommendations so transparently obvious – NOT!

    Throwing stones and living in glasshouses are clearly complementary activities for him.

  3. Pingback: Patrick Holford and Contriving a Controversy: the Cochrane review of antioxidant supplements « Holford Watch: Patrick Holford, nutritionism and bad science

  4. LeeT

    Mrs Holford: Hallo, have you had a nice day at the institute?
    Mr Holford: No, I had a great media opportunity that was messed up by Catherine Collins. Apparently a guy called Cochrane has just written a paper claming my life’s work is a sham. I mean one person’s point of view!
    Mrs H: umm … no … actually the criticism came from The Cochrane Collaboration a global network of volunteers dedicated to promoting evidence-based health care.
    Mr H: Sounds like a bunch of micro-fascists to me trying to stifle the little man. I’ll mention that in the next email to my supporters.
    Mrs H: Don’t think that’s a good idea. Professor Archie Cochrane (1908-1988) dropped out of medical school to join the International Brigade and fight the fascists in the Spanish Civil War. He dedicated his life to evidence-based health care. Don’t you read Bad Science http://www.badscience.net/?p=284?
    Mr H: Who funds them?
    Mrs H: They rely on grants and donations, and don’t accept conflicted funding.
    Mr H: What’s conflicted funding? Do they receive funding from the EU?
    Mrs H: Err possibly.
    Mr H: Most large European cities have a Burger King branch. Therefore The Cochrane Collaboration is funded by Burger King.
    Mrs H: You’ve lost me.
    Mr H: Anyway, only one thing for it. We have to invite a journalist from The Times or The Guardian over here again to show what a normal family we are. I’ll put some rabbit droppings down in the kitchen. (http://www.guardian.co.uk/society/2003/apr/01/lifeandhealth.healthandwellbeing) That should help increase supplement sales amongst concerned parents?

  5. Wulfstan

    OK – I had a tea – nose – monitor moment. Glossing over that, you seem to have answered one of the questions reproduced in the BCNH comment:

    Just out of curiosity, could you clarify who funds the Cochrane Collaboration?

    If only they take notice of your answer.

  6. LeeT

    Here is a list of how the UK section funds its activities:
    http://www.cochrane.co.uk/en/support.html

    and here is the document on conflicted funding:
    http://www.cochrane.org/docs/commercial_sponsorship_revised.htm

    All very transparent. How refreshing. Can’t see any mention of Big Pharma or Big Burger.

  7. LeeT

    “OK – I had a tea – nose – monitor moment”

    Sorry, but could you explain what a tea, nose monitor moment is …?

  8. Lee, have you never been drinking something (such as tea) when something made you laugh, your breathing lost its co-ordination, your delicious drink ended up in your nose and it ended up being splattered over your keyboard and monitor?

  9. LeeT

    Ahh, that’s okay. It is a positive thing then. I thought possibly I was being told off.

    If I was being told off I was going to point out that: (1) in real life Patrick Holford does not appear to understand what the Cochrane review is (2) an academic once accused the Cochrane collaboration of being fascist (3) Patrick Holford really does invite journalists in to his home to watch him picking up rabbit droppings and (4) Patrick Holford’s supporters really DO think we all love Burger King and unhealthy eating [See Tony Bishop-Weston of Basingstoke Nutritionists or whatever they are called]

    Seriously though, has anyone had any experience of alternative medicine people doing satire the other way? It just doesn’t work does it!?

  10. Don’t think so. I think that some of us have much messier laptops or desktops than we would wish when we inadvertently come across something funny :-)

    And, yes – lots of substantiation for your points.

    The only examples that come to mind of satire weren’t. It was some wee soul who repeated, “Voldemort, oops Ben Goldacre” every 2 lines or so in her posts. There are the usual accusations of sciencism (I think, could be ‘scientism’), the hilarious fantasies about us being Pharma Shills etc.. Stuff like that. So, not too inspirational.

  11. LeeT

    I think some of Mr Holford’s chums confuse abuse with humour.

  12. Hm – they are probably near each other in the brain wiring (so to speak) and explain something profound about the general level of number-handling skills and logic.

  13. Pingback: Sir Cliff Richard offers conclusive refutation of Cochrane review on antioxidants and mortality « Holford Watch: Patrick Holford, nutritionism and bad science

  14. Pingback: Irish Association of Nutritional Therapy on the Cochrane Review of Antioxidant Supplements « Holford Watch: Patrick Holford, nutritionism and bad science

  15. Pingback: Irish Association of Nutritional Therapy: Giving the Facts About the Cochrane Review of Antioxidant Supplements « Holford Watch: Patrick Holford, nutritionism and bad science

  16. Pingback: Dear Terence Kealey, About Natural v. Synthetic Vitamins « Holford Watch: Patrick Holford, nutritionism and bad science

  17. Pingback: Patrick Holford Refers to Himself As An Expert (No-One Else Was Volunteering) « Holford Watch: Patrick Holford, nutritionism and bad science

  18. Arlof

    How can he not have withdrawn that article after these errors have been highlighted?

    Yes, it’s much easier to think there’s a huge conspiracy theory that nobody’s ever whistleblown, with documents, rather than you don’t know what you’re talking about. Still, he seems to have convinced a lot of people who are willing to pay him money for his advice and views.

  19. Pingback: Patrick Holford Promotes Error: Does This Explain His Continuing Support for Opposing MMR and Supporting Andrew Wakefield’s Research? « Holford Watch: Patrick Holford, nutritionism and bad science

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s