A blog comment recently suggested that we’re not giving enough attention to the Journal of Orthomolecular Medicine, and as it happens I have recently been reading Holford et al’s article in the journal on low glycemic load diets. So, I wanted to analyse the small open trial of the Holford GL diet reported on in the article*. However, I face a problem: the trial is so badly reported that I can’t work out how to interpret the result. As Ben Goldacre argues in his book Bad Science (p. 50) – he’s focusing on homoeopathy here – “as a general rule it’s always worth worrying when people don’t give you sufficient detail about their methods and results.” I’m going to give some examples of what worries me about the Holford et al article.
Firstly, the trial is not clear as to the goal of the intervention used. The 20 people on the trial contained 8 people with a ‘normal’ BMI (p. 75). The article (p. 75) reports that 12% of the group
had been medically diagnosed as suffering hypertension, hypercholesterolemia, or showing early signs of impaired glucose management. The remainder were in apparent good health, but had specific goals relating to weight management.
Fair enough – but what were these goals. Were those with a ‘normal’ BMI seeking to maintain, gain or lose weight? While the study reports weight loss (12 of those who completed the trial lost at least 7lb) it is not clear whether weight loss was a goal for 8/20 subjects. This leaves me wondering how one can assess whether or not the intervention was successful.
Secondly, the 20 people who started on the trial contained only 3 men (p. 75). 4 of the participants dropped out before the end of the trial, and it’s not made clear whether any men made it to the end of the trial. It is therefore not clear whether any men’s results are reported on in the trial (the results section (p. 76) only includes those who completed the trial) and it is unclear to if the trial results (insofar as they are valid at all) include any male subjects.
Thirdly, a number of tests were used in an attempt to assess the health of the participants. The researchers (p. 75) tested homocysteine levels in participants by submitting blood samples “for laboratory testing”. However, in the absence of information on which laboratory was used, the protocols followed in blood storage, etc. it is hard to know whether this was done accurately. Researcher also used Bioelectrical Impedance Analysis scales in an attempt to measure the body fat percentage of participants. The article (p. 75) states that “studies purporting the accuracy of the appliance used indicate a tolerance factor of +/- 1%”. Unfortunately, though, there is no reference to the studies in question. It also remains unclear whether this tolerance factor is for the measurement of Bioelectical Impedance or for body fat percentage: given that the readings on such scales can be seriously affected by hydration levels etc, I suspect the former; however, it is hard to know and therefore hard to assess these results.
A fourth concern is the reporting of drop outs (who weren’t included in the final results reported in the study). The article state (p. 76) that “Four of the 20 participants dropped out due to personal complications hindering their dietary and exercise regime”. They don’t specify what the ‘complications’ were: of course, if we are talking about broken limbs or suchlike, this would not be down to the Holford GL Diet; however, if the ‘complications’ were things like chocolate cravings or poor mood, this could tell us important things about the efficacy or otherwise or the diet.
A fifth concern (acknowledged by the authors (p. 76)) is exercise: “subjects were encouraged to excercise but no significant controld or measure of any changes in exercise was recorded.” I’m therefore left unsure how much of the recorded weight loss was down to increased exercise, and how much was down to the Holford GL diet.
All of this leaves me unable to get much from this study. This is a wasted opportunity, and I am unclear why the referees and journal editors allowed it the article to be published in its current form. This is not an issue of funding, or of big pharma suppressing research into lifestyle interventions: conducting and reporting one’s research to a decent standard does not cost much more than doing this badly. The Holford GL diet may or may not be effective – but this article does not allow one to make a worthwhile assessment of its efficacy.
The article (p. 77) calls for a longer, controlled study of the Holford GL diet. However, when a small study like this has been reported so badly, I would be cautious about allowing the same research team to conduct a larger study.
Postscript: Holford’s decades of nutritional research
As a postscript, it’s worth remembering that Holford has been working in the field of nutrition – and attempting to conduct trials in this field – for decades. For example, his 1985 book Vitamin Vitality (see page 34 and 62, and following text) has ‘trials’ of nutritionism for cyclists and those trying to lose weight. On the positive side, Holford’s claims for his pills and nutritional interventions are somewhat more restrained nowadays: for example he previously argued (p. 61) that
I used to treat overweight people with extra vitamin supplements alone with a success rate of around 40 per cent. With the five-factor diet I believe 90 per cent of people will be guaranteed to lose weight
However, the reporting of these 1980s trials still makes it nigh-on impossible to draw any useful information from them. For example, with the weight loss trial Holford fails to tell us what percentage of participants completed the trial. The cycling trial tells us little except for the – rather unsurprising – fact that cyclists’ performance tends to improve after three months of training.
Having stuck at studying a subject for decades is impressive. Having studied nutrition for decades – and been involved in trials for decades – while consistently making serious, very basic mistakes in your work is also impressive, though perhaps not in the way that Holford and his employers and supporters might hope.
As we might have mentioned before, Holford is the Head of Science and Education at Biocare and CEO of the ‘educational’ charity Food for the Brain; he also founded and lectures at the Institute for Optimum Nutrition, shaping the nutritionists of the future. Holford was – until recently – Visiting Professor at Teesside University. Failing to even write up a trial properly – after decades of working in the field – might seem bad. However, if one can still acquire such positions notwithstanding these failings, maybe it’s actually a perfectly rational strategy: perhaps those working in ‘alternative’ nutrition can gain more from using their time and energy to focus on important areas such as publicity, marketing etc., rather than fussing about boring details like research methods?
Certainly, us science bloggers can mock the likes of Holford for their sloppy science but, as Goldacre says of Equazen in Bad Science (p. 160), after pulling apart the Durham/Equazen fish oil (non)trial in some detail
If you think I have been overly critical, I would invite you to notice that they win.
* Holford, Patrick, Kerry Torrens and Deborah Colson (2006) “The Effects of a Low Glycemic Load Diet on Weight Loss and Key Health Risk Indicators” in Journal of Orthomolecular Medicine 21:2, pp71-8.