You might recall that in early January we had mail from Professor Patrick Holford of Teesside University. We responded to him and took the opportunity to ask some questions about the Food for the Brain Child Survey 2007 (FFTB) (pdf) for which he is co-author. Well, we wrote that email 3 weeks ago and have received no response.
We have published our extensive (some say mercilessly thorough) examination of the FFTB Report in several parts: Part 1, Part 2, Part 3 and Part 4. We have many, many questions about the data and analysis in the FFTB Report. We wanted to clarify our understanding of several matters so that we could put together an appropriately rigorous comment on the main substance of this report. The queries that we have are essential to understanding and interpreting the report, yet, as of today, we have not received any response.
The extensive publicity and promotion of the survey emphasised that it collects data on more than 10,000 children, however, in the data analyses, it is not clear to us when the authors are presenting the data from the 10,222 children or just the 3139 for whom there are SATs scores. This is a very important clarification but despite approaching Holford and two members of the Scientific Advisory Board (SAB) (pg. 4 of report; named in 1.1 and acknowledged in 1.3 for checking the data and analyses), we still don’t have an answer. Holford has not responded to us; one of the members of the SAB was unwilling to discuss the report and referred us to the report’s authors; the other SAB member professed himself unable to answer our questions.
We shall embark upon our analysis of the data but before we unleash it upon you, we reproduce part of our email to Patrick Holford on January 11 (the numbers in bold refer to section, table or graph numbers in the report).
We’re also in the process of analysing the science of the September 2007 Food for the Brain Child Survey, and it would be very helpful if you could answer a couple of questions and respond to a couple of
criticisms. If FFTB is looking to publish this as a journal article, I suspect that these points will be among those raised by reviewers anyway – so hopefully shouldn’t create too much additional work:
- In most of the report, it’s not clear whether the nutritional data are drawn from the data collected on all 10,222 children or just those for whom their SAT scores are known (the 3,139). 3.5 states that the data for assessing food/food group associations were generated from the 10,222 respondents. However, it is unclear whether this data were comparable to those of the sub-group of the 3,139 for whom there were SAT scores or not; it is this latter group that is represented in later analyses. Could you clarify this?
- The graph 4.2.1 is represented as showing a “direct and consistent increase in SAT scores with improving diet”, yet it doesn’t. The improved diet is based on a score that is derived from the survey results of these children with a substantially below average SAT score. It is difficult to tell from the data but it seems as if the no. of children in the ‘very good diet’ category may be as low as 120 or so – is this the case?
- Our problem with table 4.3 is that we don’t understand if the data in the first 3 columns are drawn from the analysis of the 10,000+ or just the 3,000+ that are represented in the SAT scores of column 4. Beyond that, we don’t understand what is meant by the effect size mentioned in the notes. The report does not mention how FFTB calculated the effect size or any useful information such as SDs. We have never seen an equivalence drawn between a % and the effect size: an effect size of 8% is said to indicate that a high consumer of a food has an 8% higher score than a low/non consumer. It also seems quite novel to calculate an effect size for an observational survey without any intervention
- 4.4 has ideal diet recommendations for children. The recommendations are a little tricky to interpret; do the 5+ fruit and 5+ vegetables include the portion of nuts and seeds and the dark green veg or are these last two in addition?
- We don’t have any helpful gender breakdowns. Given the better academic performance reports for girls than boys (6.3), it is a little odd that we don’t see analysis of different SAT scores, particularly the higher ones that are drawn from comparatively small numbers that may be disproportionately female?
- Those completing the FFTB questionnaire were asked whether the children used essential fat and multivitamin supplements, but the results of this are not included in the report. Was there a particular reason for this omission, and are these results available?
- 7.3 reports improved academic performance of children who eat a portion of nuts/seeds per day. However, in the table for 8 we discover that 1% of the sample achieved this. Is this 1% derived from the 10,000+ sample or the 3000+ set?
- For the whole of 9, we are not clear whether the data are from the 10,000+ sample or just the 3000+ SAT set – perhaps you could clarify. It also seems a little odd to put the 2 lowest groups together as one class and contrast them with a small group of the comparatively well-behaved or high academic performers: what was the rationale for this?
- Re: nuts/seeds, how does 9:16 relate to the recommendation for a daily serving?
- 10.1 is interesting because it would have been helpful if they had indicated how many children fell into each group so that we might have some idea of the samples. It seems as if the ‘very good diet’ group would have to be part of the subset of children who are in the 6% who eat oily fish 3+ times per week, 2% who frequently eat dark green veg, and the report refers to 1% who eat nuts/seeds daily: is this correct? What is n for this group – it looks like it might be very small?
We had many more questions but had hoped that the selection above would initiate a dialogue so that we might discuss them. We were mistaken. It is very unusual for the authors of such a publication to fail to respond to enquiries. It is regrettable that the members of the FFTB SAB were unable to assist us with our enquiries. Holford Watch feels very strongly that research that might influence public policy or research agenda should be robust and capable of surviving the sort of scrutiny that is common to the peer-review process.
We shall publish our extensive analysis of this report – we would far rather do so with the cooperation of FFTB than without it.
Food for the Brain Child Survey 2007: The Promotion
Holford Watch looks at the literature review:
Food for the Brain Child Survey 2007: Review Part 1
Food for the Brain Child Survey 2007: Review Part 2
Food for the Brain Child Survey 2007: Review Part 3
Food for the Brain Child Survey 2007: Review Part 4
Food for the Brain Child Survey 2007: Review Part 5
Holford Watch appeals for help to Professor Holford and two members of the Scientific Advisory Board who approved this report and then looks at the data and analyses:
Food for the Brain Child Survey 2007: Review Part 7
Food for the Brain Child Survey 2007: Review Part 8
Why Don’t Food for the Brain Report Their Survey Results on Supplement Pills Survey: Review Part 9
Food for the Brain Child Survey 2007: Review Part 10