Dr Ben Goldacre of badscience.net has posted a summary of the Durham Fish Oil
Trials Initiative and the latest update to the saga in which Durham Council has released the data showing spectacularly successful outcomes. Except, it hasn’t. Enquiring minds want to know if the data are being held over to be released with a fanfare at the Food for the Brain conference and the session where Dr Madeleine Portwood is scheduled to announce the outcome of the trials initiative.
Durham Council has been fielding a number of Freedom of Information requests related to the initiative. We have previously noted that this has led them to complain about their language being interpreted literally and that this spawned a remarkable mutation in the press release on the topic.
As recently as April, Durham Council responded to a FOIA request by stating that:
I can now confirm and reiterate that there was no contract or formal agreement between Equazen and Durham County Council and/or Dr Madeleine Portwood.
As regards the gifts/services or products given or agreed to, obviously Equazen did supply the capsules to be used in the initiative. However, all unused capsules were to be returned to Equazen on the conclusion of the initiative.
Any collection, collation and analysis of data would be an internal affair for Durham County Council and would not involve Equazen. However, I should reiterate that it was never intended that there would be scientific analysis of the data. All that was being sought was the number of children taking up the offer of capsules and a comparison of GCSE results from the previous year. This information will be published on the Durham County Council website later this year and will be readily available for you to look over.
Call us picky, but does this look like an adequate description of what Durham Council has released?
Initially, just over 3,000 Year 11 pupils began the study, taking the Omega-3 tablets at school and at home.
By the time GCSE examinations came around, 832 pupils had 80 per cent or greater compliance.
Mr Ford and his colleagues then sought to identify the same number of Year 11 pupils who had not taken the supplement and match them to those who had, according to school, gender, prior attainment and social background.
The GCSE results of 629 ‘matched pairs’ – fish oil takers and non-fish oil takers – were then analysed.
“ To reach comparative levels of their attainment prior to the study, we used a nationally accepted system which took into account the results for each pupil at Key Stage 2 and Key Stage 3,” said Mr Ford.
“ At both stages, the difference in predicted GCSE outcomes between the groups was on average less than three points.
“ However, following the fish oil initiative, the difference in Key Stage 4 (GCSE) results between those who had taken the supplement and those who had not rose to 17.7 points.
“ If there had been no difference in attainment between the two groups, we would be tempted to dismiss the benefits of Omega-3,” said Mr Ford.
“ However there seem to be some very clear indications that pupils taking the supplement do significantly better.”
Mr Ford said the Council made no claim that the results of its GCSE study could be attributed only to Omega-3 supplementation.
A few random observations. That is a more than 70% attrition rate for children who managed to stay the course and take 80% or more of their essential fatty acid capsules over the duration of the project. However, the people analysing the results have not accounted for that by attempting an intention to treat or numbers needed to treat analysis. We have no idea how or why 76% of the capsule compliant children were selected for the match pair assessment. Were the non-compliant children non-compliant from the outset in which case we are comparing two sets of extremes? Or had some of the children taken, say, 75% of the capsules for the duration but they were being compared in this match pairing?
Were the match pairs from comparable schools or from schools involved in the project where the children might have been caught up in the same media excitement, and the Hawthorne Effect of extra attention and high expectation? As Goldacre points out, both in his columns on the topic and in his book, what happened at Durham was the case study for how to maximise the Hawthorne Effect. The sense of expectation and media razzmataz conjured up a modern-day remake of James Stewart’s Magic Town set in Durham rather than Grandview. Camera, sound, lighting and technical crews shot B-roll in the classrooms. Famous and/or attractive reporters interviewed children for mainstream media. The parents spoke about their hopes for the initiative; the teachers provided their opinions. Dr Madeleine Portwood and Dave Ford, head of education, described their expectation of positive outcomes.
The sort of set-up will almost certainly produce a false positive result and distort any outcomes. It is unfortunate that this seems to be the standard nutritionism research protocol for any research that involves children, schools and the interest of the media. (You may recall the ludicrously over-hyped work of Patrick Holford and Food for the Brain that was reported so credulously throughout the mainstream media).
For now, we can only speculate about the results until such time as Durham Council or their partners publish them. Publication in an appropriate journal would be the preferred route. One might piously wish that this isn’t going to be another example of scientific breakthrough by press release with lots of publicity and excitement and little scrutiny until later – by which time, a more sober interpretation of the actual findings has little to no chance of being reported.
The only way of definitively rehabilitating this zombie is by publication and appropriate scrutiny. Releasing insufficiently detailed press releases is not good enough. Announcing the results at a conference without releasing the data would also not be good enough.