Professor Patrick Holford of Teesside University and Head of Science and Education at Biocare regularly describes his own special reality when handing out his criticisms of skilled researchers. As such, Holford’s response to the latest Cochrane Review of Antioxidants (pdf) is a classic: Are supplements harmful?
This is the fourth time Bjekalovic and his group have reviewed the effects on selected studies on antioxidants. Anyone following the science of antioxidants over the past 20 years will be aware of a vast number of studies reporting positive results. So, how do you end up with a headline that implies antioxidants increase mortality?
In this review, which is a rehash of their paper published last year in the Journal of American Medical Association (JAMA), they first excluded over 400 trials, many of which had positive outcomes, but had no deaths. They then decided which trials they liked (low risk bias) and did not like (high risk bias), a factor that has received criticism in mainstream medical journals
Yes, yet again, Holford has displayed his astonishing ignorance of just how rigorous the specification is for a systematic review. What does Holford think that they do? Drag themselves into an office and throw darts or flip coins? Does it sound even vaguely plausible to you that the much-admired Cochrane Reviews depend upon such subjective assessments as ‘liking’ a paper? “Yes – I like the sound of this one but not this? The vowels in the authors’ names, they are too louche. We shall describe those papers as high risk bias.”
Is the process of assessing risk bias an esoteric art that is concealed from non-initiates? Well, it is detailed and laborious but there is a rigorous and transparent process for deciding risk of bias and you can read all about on pg. 4 of the review (pg. 7 in the pdf page counter).
Trials with adequate generation of the allocation sequence, adequate allocation concealment, adequate blinding, and adequate follow-up were considered low-bias risk trials (high methodological quality)…We appended ’Low’ to the references of these trials. Trials with one or more unclear or inadequate quality components were classified as high-bias risk trials (low methodological quality)…
You will find a fuller description and more detail in Chapters 8, 12 and elsewhere (particularly 13.5 onwards) in the Cochrane Handbook. Anyone can look at the Handbook, it is freely available to all interested parties and you don’t even have to register to look through it.
Chapter 8 provides a detailed discussion of study-level assessments of risk of bias in the context of a Cochrane review. It describes an approach to assessing the risk of bias that affect outcome across studies: these are classified as ‘low risk of bias’, ‘unclear risk of bias’ and ‘high risk of bias’ (Chapter 8, Section 8.7). For example, ‘low risk of bias’ would indicate ‘no limitation’; ‘unclear risk of bias’ would indicate either ‘no limitation’ or ‘serious limitation’; and ‘high risk of bias’ would indicate either ‘serious limitation’ or ‘very serious limitation’.
It is true that appropriately experienced researchers must use their informed judgement to decide between these categories, depending on the likely magnitude of the potential biases. However, it is because the reviewers are aware of a risk of subjectivity that such assessments are carried out independently by at least two people who then meet, typically with another researcher, and come to a decision on the papers where there is a difference of assessment.
It is a given that any study addressing a particular outcome will not be perfect: e.g., it may be improbable that some research can ever be done definitively because you would never obtain clearance from an ethics review board; some external factor might change during the period of a study such as the fortification of food which might alter your assessments. So all studies differ, to some degree, in the risk of bias. Review authors make an overall judgement as to whether the quality of evidence for an outcome should is such that one would not place too much confidence in it. Authors of a systematic review use well-defined, transparent and open procedures. We will look into this more but it is notable that Holford has not given a single example where he could specify his point of disagreement of risk bias where his evaluation differs from that of the authors. (We would also note that the NHS NLH finds the Cochrane Review to be “well conducted and thorough”.)
Effectively, the systematic review process is geared towards evaluating whether or not a particular study was a ‘fair test’. One of the best explanations of ‘fair tests’ is Evans, Thornton and Chalmers’ Testing Treatments (available for free download). We strongly recommend this book to Holford as it explains all the ins-and-outs of what makes a fair test and the fair way in which to interpret the results. Once Holford actually understands what is involved and just how rigorous the process is, perhaps he will be less glib about criticising the competence or integrity of researchers. Kruger and Dunning explain why it might be too much to expect Holford to form an appropriate assessment of his own level of comparative competence. Yet again, is this an example of what Professor Holford will be teaching to the impressionable minds of students at the University of Teesside?
 You may be interested to read an overview of the Justin Kruger and David Dunning paper that discusses how difficulties in understanding one’s own incompetence can lead to inflated self-assessments.
What’s even more amazing is that when they then shared the performance of other participants with the people who performed poorly (hoping that they would then adjust their self-perception downward) people who scored poorly failed to adjust their self-perception of their performance. In other words, they are completely unaware of their own [in]competence, and can’t detect competence in others.
It really it a very helpful paper that explains many otherwise inexplicable actions.
A Photon in the Darkness likewise offers a helpful discussion of this paper: The Arrogance of Ignorance