The Holford Low GL Diet trial: how was it conducted, and what were the results?

A blog comment recently suggested that we’re not giving enough attention to the Journal of Orthomolecular Medicine, and as it happens I have recently been reading Holford et al’s article in the journal on low glycemic load diets. So, I wanted to analyse the small open trial of the Holford GL diet reported on in the article*. However, I face a problem: the trial is so badly reported that I can’t work out how to interpret the result. As Ben Goldacre argues in his book Bad Science (p. 50) – he’s focusing on homoeopathy here – “as a general rule it’s always worth worrying when people don’t give you sufficient detail about their methods and results.” I’m going to give some examples of what worries me about the Holford et al article.

Firstly, the trial is not clear as to the goal of the intervention used. The 20 people on the trial contained 8 people with a ‘normal’ BMI (p. 75). The article (p. 75) reports that 12% of the group

had been medically diagnosed as suffering hypertension, hypercholesterolemia, or showing early signs of impaired glucose management. The remainder were in apparent good health, but had specific goals relating to weight management.

Fair enough – but what were these goals. Were those with a ‘normal’ BMI seeking to maintain, gain or lose weight? While the study reports weight loss (12 of those who completed the trial lost at least 7lb) it is not clear whether weight loss was a goal for 8/20 subjects. This leaves me wondering how one can assess whether or not the intervention was successful.

Secondly, the 20 people who started on the trial contained only 3 men (p. 75). 4 of the participants dropped out before the end of the trial, and it’s not made clear whether any men made it to the end of the trial. It is therefore not clear whether any men’s results are reported on in the trial (the results section (p. 76) only includes those who completed the trial) and it is unclear to if the trial results (insofar as they are valid at all) include any male subjects.

Thirdly, a number of tests were used in an attempt to assess the health of the participants. The researchers (p. 75) tested homocysteine levels in participants by submitting blood samples “for laboratory testing”. However, in the absence of information on which laboratory was used, the protocols followed in blood storage, etc. it is hard to know whether this was done accurately. Researcher also used Bioelectrical Impedance Analysis scales in an attempt to measure the body fat percentage of participants. The article (p. 75) states that “studies purporting the accuracy of the appliance used indicate a tolerance factor of +/- 1%”. Unfortunately, though, there is no reference to the studies in question. It also remains unclear whether this tolerance factor is for the measurement of Bioelectical Impedance or for body fat percentage: given that the readings on such scales can be seriously affected by hydration levels etc, I suspect the former; however, it is hard to know and therefore hard to assess these results.

A fourth concern is the reporting of drop outs (who weren’t included in the final results reported in the study). The article state (p. 76) that “Four of the 20 participants dropped out due to personal complications hindering their dietary and exercise regime”. They don’t specify what the ‘complications’ were: of course, if we are talking about broken limbs or suchlike, this would not be down to the Holford GL Diet; however, if the ‘complications’ were things like chocolate cravings or poor mood, this could tell us important things about the efficacy or otherwise or the diet.

A fifth concern (acknowledged by the authors (p. 76)) is exercise: “subjects were encouraged to excercise but no significant controld or measure of any changes in exercise was recorded.” I’m therefore left unsure how much of the recorded weight loss was down to increased exercise, and how much was down to the Holford GL diet.

All of this leaves me unable to get much from this study. This is a wasted opportunity, and I am unclear why the referees and journal editors allowed it the article to be published in its current form. This is not an issue of funding, or of big pharma suppressing research into lifestyle interventions: conducting and reporting one’s research to a decent standard does not cost much more than doing this badly. The Holford GL diet may or may not be effective – but this article does not allow one to make a worthwhile assessment of its efficacy.

The article (p. 77) calls for a longer, controlled study of the Holford GL diet. However, when a small study like this has been reported so badly, I would be cautious about allowing the same research team to conduct a larger study.

Postscript: Holford’s decades of nutritional research
As a postscript, it’s worth remembering that Holford has been working in the field of nutrition – and attempting to conduct trials in this field – for decades. For example, his 1985 book Vitamin Vitality (see page 34 and 62, and following text) has ‘trials’ of nutritionism for cyclists and those trying to lose weight. On the positive side, Holford’s claims for his pills and nutritional interventions are somewhat more restrained nowadays: for example he previously argued (p. 61) that

I used to treat overweight people with extra vitamin supplements alone with a success rate of around 40 per cent. With the five-factor diet I believe 90 per cent of people will be guaranteed to lose weight

However, the reporting of these 1980s trials still makes it nigh-on impossible to draw any useful information from them. For example, with the weight loss trial Holford fails to tell us what percentage of participants completed the trial. The cycling trial tells us little except for the – rather unsurprising – fact that cyclists’ performance tends to improve after three months of training.

Having stuck at studying a subject for decades is impressive. Having studied nutrition for decades – and been involved in trials for decades – while consistently making serious, very basic mistakes in your work is also impressive, though perhaps not in the way that Holford and his employers and supporters might hope.

As we might have mentioned before, Holford is the Head of Science and Education at Biocare and CEO of the ‘educational’ charity Food for the Brain; he also founded and lectures at the Institute for Optimum Nutrition, shaping the nutritionists of the future. Holford was – until recently – Visiting Professor at Teesside University. Failing to even write up a trial properly – after decades of working in the field – might seem bad. However, if one can still acquire such positions notwithstanding these failings, maybe it’s actually a perfectly rational strategy: perhaps those working in ‘alternative’ nutrition can gain more from using their time and energy to focus on important areas such as publicity, marketing etc., rather than fussing about boring details like research methods?

Certainly, us science bloggers can mock the likes of Holford for their sloppy science but, as Goldacre says of Equazen in Bad Science (p. 160), after pulling apart the Durham/Equazen fish oil (non)trial in some detail

If you think I have been overly critical, I would invite you to notice that they win.

* Holford, Patrick, Kerry Torrens and Deborah Colson (2006) “The Effects of a Low Glycemic Load Diet on Weight Loss and Key Health Risk Indicators” in Journal of Orthomolecular Medicine 21:2, pp71-8.

BPSDB

19 Comments

Filed under GL diet, patrick holford, supplements, University of Teesside

19 responses to “The Holford Low GL Diet trial: how was it conducted, and what were the results?

  1. nice post. problem is, holford and his ilk are likely to squeal that when people expose such glaring methodological flaws (i.e. was any change due to the ‘intervention,’ or because participants exercised more), we’re objecting to ‘technicalities.’ in other words, we’re splitting hairs, mining for faults. this gives them the ability to bleat to the press that The Establishment is finding ways to discount their research. Still, that doesn’t mean it isn’t fun and worthwhile to pull apart this study…!

  2. “I am unclear why the referees and journal editors allowed it the article to be published in its current form.”

    err… ‘cos the journal is a joke (hence no PubMed listing) – “hobbyists’ journal” would be one apt description – and any referees they use will be more people with the same critical standards as Patrick?

    As has been frequently pointed out, “Peer review” indeed.

    The same “Shaky Woo, by Woo-ser and for Woo-sers” effect applies at PubMed-listed Cargo Cult Journals like the dire Journal of Alternative and Complementary Medicine. For a recent example of the “Cargo Cult Effect” in action, see this interesting post of Orac’s.

  3. superburger

    Do you think the journal would accept a paper containing these criticisms if it was submitted to them?

  4. Teek- sure, in a sense ‘they’ definitely win. Notwithstanding the problems with his own research, Holford is taken seriously by the MSM when he offers pretty poor criticisms of excellent quality research such as Cochrane reviews, was intended to teach students and supervise research at Teesside University, etc. I’m not sure if there’s a better way of approaching this…

    draust- fair enough, but I hadn’t read much of this journal previously. Even if it is just a ‘hobbyist’ journal, I’d have hoped that the reviewers would at least have asked about some of the info contained in the paper.

    superburger – not sure, though an interesting idea.

    OT, but I was impressed to see that the late Bernard Rimland is still serving on the journal’s Editorial Review Board.

  5. Wulfstan

    Of course raising issues such as the number of drop-outs and the actual data relating to weight is picky. Science is, in many ways, picking away at the details until something makes sense. It’s just that Holford seems to pick at the wrong details and have lots of difficulties related to his own lack of understanding (e.g., the infamous blobbogram).

    I was impressed to see that the late Bernard Rimland is still serving on the journal’s Editorial Review Board.

    You understand nothing about the remarkable benefits of mega-vitamins where the benefits extend even after death. [/snark]

    “Shaky Woo, by Woo-ser and for Woo-sers” sounds like an anthem/dance/pudding all in one.

  6. Wulfstan

    I should have said that it is ludicrous that anyone can boast of 30 years experience in research and yet show such unremitting lack of competence in conducting any. I’m drawing a line that starts with the ‘trials’ you reference and continues through the GL stuff and the horror that was the Food for the Brain Child Survey.

    I hope that nobody ever gives Holford or his Scientific Advisory Board tax-payer funding for their vanity projects – it would be a waste of money that could be spent on genuine research.

  7. “Shaky Woo, by Woo-ser and for Woo-sers” sounds like an anthem/dance/pudding all in one.

    See what you mean.

    Talking of songs, can I put it a vote for the homeopaths’ favourite anthem, “Whole Lotta Shakin’ Goin’ On”?

    PS In the science mainstream, it is possible to continue publishing after one’s demise – e.g. the late great Stanley J Korsmeyer has published more in the 3.5 yrs since his death than many of us mere mortals manage in a career.

    Of course, it is a little more difficult to review work from beyond the grave, since reviewing does not usually involve living co-workers. But perhaps Bernard Rimland uses the same mega-vitamins and anti-oxidants as Patrick’s and my old friend Count Dracula.

  8. “[H]ow was it conducted, and what were the results?” Well, the answers seem to be: i) not frightfully well ii) who knows but seems less than impressive?

    As to the beyond the grave reviews – if only Darwin and Wodehouse had had some foresight, they might not now be being outed as a supporters of homeopathy. [/D.Ullman quip.]

    Different sects of homeopaths favour different songs. Shakin’ all over. Shake it. Shake the tree (for those who like to expose remedies to the sun by sticking them up a tree). We can’t possibly overlook Shake Rattle and Roll.

    Allegedly, there is a break-away group that will only succuss to the Chica Chica Boom Chic of Carmen Miranda. When succussing for children, there is the Wiggles Shimmy Shake.

  9. Interesting playlist there, dvn.

    I’ve now penned my own homeopathic anthem to the tune of Supercalifragilistic-expialidocious from Mary Poppins.

    Good to see a link to Johnny Kidd and the Pirates’ Shaking All Over… a song which is even older than me. The original version (no visuals though) is here. The Guess Who’s version is a pretty straight cover – lots of other version on Youtube, too.

  10. Hah – I have an uncle and cousin who talk about little other than various standards and who covered them and when and what the B-sides were. They can name most songs and tell you who is the cover for it in 3 notes at most.

    I had to use Johnny Kidd and the Pirates or I would never have heard the end of it.

    I had another uncle with uncanny listening skills and an encyclopaedic knowledge of stage and film musicals.

    OT – if I were a better listener, I would have a truly extensive knowledge of both Motown and musicals as opposed to nodding in recognition from time to time when something comes on the radio.

    Even further OT – you’ve seen the weird but interesting cartoons by John Wilson of Fine Art Films to various 70s hits? E.g., Cher’s Dark Lady, The Kinks’ Demon Alcohol and Helen Reddy’s Angie Baby. The songs were probably a bit before your time but some of the cartoons are very unsettling in an interesting way (as far as I can tell Leroy Brown and Big Yellow Taxi look straightforward).

  11. UK dietitian

    “I am unclear why the referees and journal editors allowed it the article to be published in its current form”

    I think its totally clear –
    the referees and journal editors are as clueless about nutrition research as those submitting such Key Stage 1 (oops- foundation degree) level nutrition ‘research’.

  12. UK dietician – I think you’re being a bit harsh. In my experience, Key Stage 1 children are *very* good at asking questions ;)

  13. kenanddot

    A mild comment: I was interested to see this site, as a couple of years ago my husband and I went on the Holford Diet. It worked pretty well for us, but we ignored the stuff about vitamin supplements, so what it really amounted to was eating proportionately lots more vegetables and fewer carbs, giving up alcohol and (most of the time) caffeine, and watching the fat. Which is to say, a more balanced diet. Not very controversial. Holford’s research may be bunk but his recipes are useful.

  14. Wulfstan

    I don’t think that anyone has any objections to a Portfolio, Mediterranean or other diet that is full of vegetables and otherwise well-balanced.

    If the recipes are useful then the book should have been sold in the cookery section rather than the factual-based health section.

    It is not trivial that his own research or his interpretation of research does not stack up when he claims that as his USP.

  15. Sure – a healthy diet is a perfectly good idea. It’s when people use dubious research to promote a diet (and supplementation programme) that we object.

  16. Pingback: Patrick Holford Visited All the Major Nutritional Research Centres in the United States: or so he claimed in 1985 « Holford Watch: Patrick Holford, nutritionism and bad science

  17. Stella Scire

    I have just ordered one of your books on weight loss. I have been told by a close friend that you sell a Glycemic Load Indicator Booklet which lists all foods and the points attributed to each. Can you let me know where I could order this from as I have not found it on the Website as yet. Apparently the booklet costs about £3.00.

    I look forward to hearing from you.

    With kind regards,

    Stella Scire

    Admin edit: you might want to look at these.

    She mean this?

    http://www.patrickholford.com/content.asp?id_Content=1554

    http://www.patrickholford.com/content.asp?id_Content=1707

    They’ve taken away the section that used to give the unit values for foodstuffs.

  18. Pingback: Patrick Holford on Science Friction and the Limitations of RCTs and Meta-analyses « Holford Watch: Patrick Holford, nutritionism and bad science

  19. Pingback: Patrick Holford: Why Did BBC Oxford Radio Give Him Free Advertising? « Holford Watch: Patrick Holford, nutritionism and bad science

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s