Chair of FFTB Scientific Advisory Board acknowledges that their research hasn’t been “a proper job” and hasn’t been “rigorous”

Two members of the Food for the Brain (FFTB) Scientific Advisory Board were asked about the FFTB child survey we’ve been analysing here, and they gave a very interesting response. The SAB Chair – Prof David Smith – replied that “the report is more hypothesis-generating for future research than a rigorous scientific study….Find us some money and we will do a proper job.” Here at HolfordWatch, we’re pleased that Prof Smith acknowledges that this report was neither “rigorous scientific study” nor “a proper job”: this entirely concurs with our own analyses of this work. We also agree with SAB-member Prof Cowen’s statement that the Survey is “not informative about causality”. However, we’re disappointed that – if the Scientific Advisory Board (SAB) knew that this report was not done properly, and does not provide useful information about policy – they allowed it to be represented as something on which dietary and policy decisions could be based. We would also note that – even as an exercise in hypothesis-generation – the Survey is quite woefully inadequate.

The Survey’s literature review is riddled with errors and wholly inadequate: the authors thus fail to give due consideration to the work of other researchers when developing their hypothesis. The methodology is deeply flawed: significant compounding factors are largely ignored, and the questionnaire used has not been validated and is extremely hard to interpret.

The reporting of results is also inadequate: this is partial, and the representation and statistical analysis of the results is also full of errors. Basic statistical and mathematical errors are also not a helpful addition to an exercise in hypothesis generation. All of these problems means that the Survey does not support the conclusions drawn from it (and it is hard to see how such poor quality work could lead to any useful conclusions).

The literature review is riddled with flaws; the data interpretation and summary is flawed and there is something seriously amiss with the claims made for the data. With this in mind, we can only wonder what it would take to convince FFTB that it is time to withdraw this report and issue some corrections, or to convince the SAB to withdraw their support from this report.

At HolfordWatch, we are happy to accept that hypotheses can be generated from this survey: this has already taken place. However, the quality of this work is such that we fail to see how it is any more useful for hypothesis generation than, for example, fairy tales or dice-throwing; we suspect that you would get a much more sensible response if you simply asked your Gran what you should eat. We’re therefore surprised that Professors Cowen, Holford, and Smith all appear to see this report as useful.

To conclude, we were also struck by Prof Smith’s novel approach to research funding applications: he appears to be arguing that failing to do the initial research properly is a valid reason to be given more money for additional research. This does not fit with our knowledge of how research funding works. We believe that many would view such inadequate preliminary research as a good reason not to provide an organisation with additional research funding, at least until funding bodies can be assured that a competent Principle Investigator and appropriate Supervision have been put in place.

On the other hand, if Prof Holford’s less-than-stellar research record lead to him being awarded a Chair at Teesside University, and one sees the nominally peer-reviewed Nutrition Journal publishing such dubious research, perhaps we are lagging behind the times… We watch with interest to see whether any bona fide peer-reviewed journal is prepared to lower its standards sufficiently to publish an article based on the Survey.

Further Reading

Food for the Brain Child Survey 2007: The Promotion
Holford Watch looks at the literature review:
Food for the Brain Child Survey 2007: Review Part 1
Food for the Brain Child Survey 2007: Review Part 2
Food for the Brain Child Survey 2007: Review Part 3
Food for the Brain Child Survey 2007: Review Part 4
Food for the Brain Child Survey 2007: Review Part 5

Holford Watch appeals for help to Professor Holford and two members of the Scientific Advisory Board who approved this report and then looks at the data and analyses:
Food for the Brain Child Survey 2007: Review Part 7
Food for the Brain Child Survey 2007: Review Part 8
Why Don’t Food for the Brain Report Their Survey Results on Supplement Pills Survey: Review Part 9
Food for the Brain Child Survey 2007: Review Part 10

Advertisements

22 Comments

Filed under Food for the brain, patrick holford

22 responses to “Chair of FFTB Scientific Advisory Board acknowledges that their research hasn’t been “a proper job” and hasn’t been “rigorous”

  1. Kat

    Your penultimate paragraph is absolutely right. As I read that plea to “give us the funding” I was fuming and thinking that it wouldn’t have cost significantly more to do this “not a proper job” of “research” to a much more rigorous standard in the first place.

    But then, the objective doesn’t seem to have been to generate genuine research data, but to puff up Holford, FFTB and be an income-generating marketing tool.

    Maybe, if lack of funding is the issue, Holford could donate some of his profits from supplements sales and/or media appearances for -preferably independent – research. or would that not give the answers required for those promotional purposes?

  2. superburger

    the thing is, it wouldn’t have cost anything more or less to do a proper job of this bit of ‘research’.

    A good, thourough lit. review doesn’t cost anything. Especially for Prof. Holford who no doubt has institutional journal access (and access to British Library) via his chair at Teeside.

    I’m certainly know expert on questionnaire design, but it took me all of 2 seconds to spot the immediate and glaring flaws in the design of the questions.

    Perhaps Prof Holford et al might subcontract HolfordWatch to help him design more effective questionnaires and perform a rigorous lit. review.

    Perhaps this could be the start of a beautiful relationship?

  3. Dr Aust

    Yes, the “Gissa cash” line contains plenty of chutzpah, to put it mildly.

    I suspect people like David Smith are making a Faustian bargain here, but a dangerous one. Surveys like this certainly DO cost significant money; if one were running it out of a Univ Department I suspect you would be talking a research assistant to help devise and run it, and compile and analyse data, plus some bought-in expertise about web-based questionnaire design… and then Univs want overheads and indirect costs too. So I would suspect you would need 100 grand plus in grant money, and probably nearer 150, even if the whole thing barely took a year. This kind of cash does not wash around in Univs and to screw it out of the Govt or the research charities is a non-trivial problem.

    In contrast, when it is getting done by booster types and PR men it paradoxically probably comes out cheaper for the (commercial) sponsor. This is because some of those involved will be giving their time at a discount rate – or even free – since it is effectively business promotion for them

    For the academics involved , then, it is tempting to view this as a choice between “a proper study that will never get funded and thus doesn’t get done” or “a PR exercise which might just be splashy enough that it could allow me to “lever” the cash for a proper study”. So you can talk yourself into the latter as “the only way forward”.

    The downside, of course, is that boosters like PH don’t need a proper study if the PR exercise serves them just as well. And they get to dress up the PR as science by having David Smith (or similar) “on board”. And if the commercial sponsor has already paid for the PR, they are hardly going to want to fund a (more expensive) real study unless they have to. So any real study would have to be Govt or charity funded; but reviewers don’t regard surveys like this as being terribly good supporting evidence , for all the scientific reasons discussed on sites like this.

    So in the end the scientist is in danger of finding themself saying:

    “Please can someone fund me to do a proper study with a major rationale being to correct the misstatements of things like the PR project I just associated myself with”

    Like I say, a Faustian bargain.

  4. Thanks for the comments. I can sympathise with the problem of how expensive it can be to do research in a University setting. However, there are alternatives: a number of people do much lower-budget research for NGOs (and even blogs ;) ) and they can produce some good quality outputs. Certainly – assuming competent researchers are involved, and are willing to donate their time or work for a modest wage – things like a short lit review should not be expensive…

  5. As people have pointed out, it would not have cost any more money to write an accurate and relevant literature overview that includes papers that actually exist. It would have been more courteous to the people who have done relevant research in this field.

    Organix sponsored this study – it is not lack of money that caused these problems. Kat suggests that Prof Holford might consider donating money from his supplement sales but even this would have been unnecessary, he might just choose to forbear to draw the current salary that he does for being FFTB’s CEO – no actual donation necessary and probably a nice tax write-off.

    There are research instruments that already exist for food frequency and children – there is no good reason for FFTB to have devised their own questionnaire when validated instruments exist. Such research instruments are typically free of charge or available for nominal sums to not-for-profit institutions for research.

    There is no justification for choosing the wrong graph to display the summary data and drawing completely erroneous conclusions from that. There is no excuse for affecting to comment on the impact of food on behaviour and using the groups of children rated as very good for behaviour to make assumptions about the children rated as poor/very poor – completely wrong calculation and data set.

    I have every sympathy for the difficulties in obtaining research funding but there is no excuse for wrong data analysis even in a PR-flashy study. It is offensive that this report received such lavish and uncritical media coverage at its launch despite the patent absurdity of some of its findings.

    The errata slips would be longer than the current report – it should be withdrawn. The Scientific Advisory Board should either re-affirm their endorsement and state why or withdraw their support for this report. I would like to see GMTV run a retraction after their puff piece on this wretched report and Dr Jones’ reassurance that we could trust the report because he had read it right through and it has a robust Scientific Advisory Board.

    (My first version of this comment was lost due to an enforced Windows Update that wouldn’t postpone so that is probably just as well as it was a little intemperate. My sole mitigation is the brain-destroying effect of reviewing that wretched report.)

  6. Nice deconstruction, DVN. I wasn’t defending David Smith & Co – more pointing out a possible thought process and illustrating the slippery slope. It is fair to say that Smith has a long-standing interest in this area, though it is equally fair to say that he seems to have always had some odd “associates” in this connection – Holford, Kim Jobst etc. The question would seem to be: do you, as the “involved scientist”, have enough influence to get the study “doers” to do it right? And if not, should you associate yourself with it? And how dodgy do the people have to be before you dissociate yourself from them altogether?

    I am hearing echoes of Dr Alex Richardson and Durham Council / Equazen here as I type….!

    And in some ways, of course, this parallels the sort of issues with academics who”co-author” studies of trials done by PharmaCos, though there you would at least expect the studies to be soundly designed.

    Speaking of the data analysis, I wonder how far DVN would get if he asked for FFTB’s data so it could be submitted to a SERIOUS analysis?

  7. LeeT

    Let’s not forget many charities do fund their own research. The Charities Commission website very helpfully give the gross income of registered charities. It is not difficult to see how Oxfam (£290million) and the NSPCC (£148million) fund their operations. However, Food for the Brain’s annual income is only £162K which won’t go very far.

    Perhaps they should raised the money then done the research? I see they are making an attempt to get people to become members or “friends” as they put it: http://www.foodforthebrain.org/content.asp?id_Content=566 I wonder how many friends they have?

    Regarding DVN’s point about reviewing existing literature The World Cancer Research Fund did that back in November. (The fact their conclusions contradicted Holfordism is proably why Mr H has not quoted from it.)

    I believe some research has been done about the role of selenium and oily fish in promoting mental health. However, as far as I know no one has ever looked at the influence of supplements on mental health so perhaps he thinks he is doing us all a favour ….?

  8. Speaking of the data analysis, I wonder how far DVN would get if he asked for FFTB’s data so it could be submitted to a SERIOUS analysis?

    afaik, and we seem to have this confirmed from some FOIA documents that we shall discuss in the near future, some hapless psychologist is writing this up for a journal paper so there is no chance that we could run this through SPSS or some such and create some sensible summary data. One of the really annoying things is their use of the phrase “effect size” when discussing some of their tabulated results. I ran a stupid number of calculations trying to guess at reasonable SDs to come up with their numbers. I finally realised that they were using a statistical term in a colloquial way – there were no effect sizes involved.

    otoh, the questionnaire was so flawed and the methodology so questionable, that it is not clear that the data can yield anything useful although it would be correct as opposed to completely wrong. If they had used a validated instrument then it might be worth re-running the analysis but this is such a skewed population that I doubt it.

    And, of course, you know how far we got when we asked some questions about the data analyses etc. for the report – you’ve seen the response that Prof Colquhoun received from Profs Cowen and Smith and we didn’t receive any response to our enquiries that we made to Prof Holford.

    Usually, report authors (particularly those hoping to publish in a peer-reviewed place) welcome reviews from other people, particularly ones that highlight problems with the statistics or their summary. This wouldn’t be so irritating if they were not so frequently accusing other people of lack of scientific rigour.

  9. Yes, Lee – there are a number of good quality literature reviews out there that would have been an appropriate reference but, as you say, not supportive of the claims that the authors wanted to make here.

    There is some ‘research’ on supplements and mental health but it is generally poor quality or good quality but conducted in very specific, non-generalisable populations.

  10. Can I perhaps suggest an extended study for Holfordwatch?

    1. Scan all the media coverage you can find of the F4TB report, and also a representative selection of where it is referred to on other “health websites” (i.e. not PH’s ones).

    2. Then ask how many of these stories or quotes repeat things from the study that are arguably or even demonstrably not true.

    3. Then write a paper for a nutrition or science and society journal explaining (i) why the study and shite and (ii) exploring how often it’s mis-statements are repeated in the secondary coverage.

    Orac over at Respectful Insolence has been blogging about a study of inaccuracies on breast cancer websites . I would love to see someone do the same for PH and Co.

    The whole F4TB thing certainly makes one rather cynical about Scientific Advisory Boards. It would be interesting to know if the F4TB one had ever met, or whether it all happens by occasional email (more likely, I suspect). And I would love to be able to ASK the journalists who write about tripe like F4TB whether the existence and identity of the “Scientific advisory board” makes any difference to how reliable they view the findings as being.

  11. Professor UKdietitian

    Rather like BrainGym, it appears a common theme for educationalists to pick up on practices that seem to have a plausible rationale behind them, even if to the scientist the methods are so flawed as to render the results useless.

    It deeply concerns me that in bona fide nutrition research requires ethical approval, sufficient funding to complete and summarise the study, informed consent from clients, client confidentiality and a recognised nutritional approach using validated nutrition tools and careful monitoring – especially in children – as absolute prerequisites before the research begins.

    What F4TB do appears to fall well short of the above- whether or not they are supported by eminent psychiatrists or educationalists.

    Would you expect your childs teacher to give recommendations as to the surgeon you may need to remove a troublesome gallbladder? Of course not.

    What is it that makes teachers – or psychiatrists for that matter- assume they have sufficient nutritional knowledge to warrant support of what is, essentially, an amateur approach to nutrition research?

  12. Can I perhaps suggest an extended study for Holfordwatch?

    Suggest away because we totally don’t do that sort of thing on Holfordwatch (this being part 10 of the review but #13 in the series to date). :D

    However, I take your point and maybe we might be able to agree a methodology that would be suitable for publication.

    Interesting about the journalists and whether they are influenced by the SAB. I know that Dr Hilary Jones (pundit and commentator rather than journalist) claimed to be convinced that the SAB is a guarantee of the merits of the report. The mere existence of the SAB wouldn’t bother me so much if they had not been explicitly thanked for their help with the data and analyses in the report.

  13. Good to see you back, Professor UKdietitian.

    It deeply concerns me that in bona fide nutrition research requires ethical approval, sufficient funding to complete and summarise the study, informed consent from clients, client confidentiality and a recognised nutritional approach using validated nutrition tools and careful monitoring – especially in children – as absolute prerequisites before the research begins.

    This has been bothering us. FFTB reports that they are currently running 3 projects. Did they obtain the informed consent by quoting the results from this survey and their previous studies? If so, with the extent of serious errors in these data and the erroneous assertions that have flowed from that, do they need to re-obtain that consent?

  14. Pingback: Barefoot, Sex, Sleaze and Life’s 4 Living « Holford Watch: Patrick Holford, nutritionism and bad science

  15. Pingback: Patrick Holford and His Alternative to Anti-Depressants « Holford Watch: Patrick Holford, nutritionism and bad science

  16. Pingback: Why Do Mainstream Media Promote Patrick Holford? « Holford Myths: what is the problem with Nutritionist Patrick Holford?

  17. sue

    The May Newsletter of the ‘Food for the Brain ‘ (of which PH is the chief executative officer ) gave news of a third school project which plans to test the impact of diet only, supplements only, or diet and supplements . They have also been working with Hertfordshire County Council Catering to help them ‘tweak’ thir menu in line with FFTB principles.
    FFTB board of experts have also set up an accreditation process to recognise products that they approve as beneficial for brain function. The key criteria will be
    low GI and sugar
    high in essential fats
    high in vitamins and minerals
    free from harmful or unnecessary additives!
    Well that should not be too difficult as there are plenty of staple foods meeting that criteria, but I suspect that is not the intention!

  18. Pingback: Visiting Professor Patrick Holford, Not Even for a Thousand Days « Holford Watch: Patrick Holford, nutritionism and bad science

  19. Pingback: The Economist: The End of a Childhood Illusion « Holford Watch: Patrick Holford, nutritionism and bad science

  20. Pingback: Patrick Holford on Science Friction and the Limitations of RCTs and Meta-analyses « Holford Watch: Patrick Holford, nutritionism and bad science

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s