Dr Ben Goldacre of badscience.net has posted a summary of the Durham Fish Oil Trials Initiative and the latest update to the saga in which Durham Council has released the data showing spectacularly successful outcomes. Except, it hasn’t. Enquiring minds want to know if the data are being held over to be released with a fanfare at the Food for the Brain conference and the session where Dr Madeleine Portwood is scheduled to announce the outcome of the trials initiative.
Durham Council has been fielding a number of Freedom of Information requests related to the initiative. We have previously noted that this has led them to complain about their language being interpreted literally and that this spawned a remarkable mutation in the press release on the topic.
As recently as April, Durham Council responded to a FOIA request by stating that:
I can now confirm and reiterate that there was no contract or formal agreement between Equazen and Durham County Council and/or Dr Madeleine Portwood.
As regards the gifts/services or products given or agreed to, obviously Equazen did supply the capsules to be used in the initiative. However, all unused capsules were to be returned to Equazen on the conclusion of the initiative.
Any collection, collation and analysis of data would be an internal affair for Durham County Council and would not involve Equazen. However, I should reiterate that it was never intended that there would be scientific analysis of the data. All that was being sought was the number of children taking up the offer of capsules and a comparison of GCSE results from the previous year. This information will be published on the Durham County Council website later this year and will be readily available for you to look over.
Call us picky, but does this look like an adequate description of what Durham Council has released?
Initially, just over 3,000 Year 11 pupils began the study, taking the Omega-3 tablets at school and at home.
By the time GCSE examinations came around, 832 pupils had 80 per cent or greater compliance.
Mr Ford and his colleagues then sought to identify the same number of Year 11 pupils who had not taken the supplement and match them to those who had, according to school, gender, prior attainment and social background.
The GCSE results of 629 ‘matched pairs’ – fish oil takers and non-fish oil takers – were then analysed.
“ To reach comparative levels of their attainment prior to the study, we used a nationally accepted system which took into account the results for each pupil at Key Stage 2 and Key Stage 3,” said Mr Ford.
“ At both stages, the difference in predicted GCSE outcomes between the groups was on average less than three points.
“ However, following the fish oil initiative, the difference in Key Stage 4 (GCSE) results between those who had taken the supplement and those who had not rose to 17.7 points.
“ If there had been no difference in attainment between the two groups, we would be tempted to dismiss the benefits of Omega-3,” said Mr Ford.
“ However there seem to be some very clear indications that pupils taking the supplement do significantly better.”
Mr Ford said the Council made no claim that the results of its GCSE study could be attributed only to Omega-3 supplementation.
A few random observations. That is a more than 70% attrition rate for children who managed to stay the course and take 80% or more of their essential fatty acid capsules over the duration of the project. However, the people analysing the results have not accounted for that by attempting an intention to treat or numbers needed to treat analysis. We have no idea how or why 76% of the capsule compliant children were selected for the match pair assessment. Were the non-compliant children non-compliant from the outset in which case we are comparing two sets of extremes? Or had some of the children taken, say, 75% of the capsules for the duration but they were being compared in this match pairing?
Were the match pairs from comparable schools or from schools involved in the project where the children might have been caught up in the same media excitement, and the Hawthorne Effect of extra attention and high expectation? As Goldacre points out, both in his columns on the topic and in his book, what happened at Durham was the case study for how to maximise the Hawthorne Effect. The sense of expectation and media razzmataz conjured up a modern-day remake of James Stewart’s Magic Town set in Durham rather than Grandview. Camera, sound, lighting and technical crews shot B-roll in the classrooms. Famous and/or attractive reporters interviewed children for mainstream media. The parents spoke about their hopes for the initiative; the teachers provided their opinions. Dr Madeleine Portwood and Dave Ford, head of education, described their expectation of positive outcomes.
The sort of set-up will almost certainly produce a false positive result and distort any outcomes. It is unfortunate that this seems to be the standard nutritionism research protocol for any research that involves children, schools and the interest of the media. (You may recall the ludicrously over-hyped work of Patrick Holford and Food for the Brain that was reported so credulously throughout the mainstream media).
For now, we can only speculate about the results until such time as Durham Council or their partners publish them. Publication in an appropriate journal would be the preferred route. One might piously wish that this isn’t going to be another example of scientific breakthrough by press release with lots of publicity and excitement and little scrutiny until later – by which time, a more sober interpretation of the actual findings has little to no chance of being reported.
The only way of definitively rehabilitating this zombie is by publication and appropriate scrutiny. Releasing insufficiently detailed press releases is not good enough. Announcing the results at a conference without releasing the data would also not be good enough.
Notes
Our thanks to toastmonster for her permission to use the wonderful zombie fish picture.
Holford Watch: Patrick Holford, nutritionism and bad science 
“Were the non-compliant children non-compliant from the outset in which case we are comparing two sets of extremes? Or had some of the children taken, say, 75% of the capsules for the duration but they were being compared in this match pairing?”
A very good point – No doubt Durham will claim that this emphasises the benefits of fish oil as if some of the children cross matched were, say 50% compliant then they would presumably have improved as well – thus reducing the difference that would have been expected between compliant and non-compliant children.
I am not sure about the Hawthorne effect – all children were presumably given the opportunity to partake and were provided with information by Durham and saw the press releases.
I am sure Durham would have done their best to match children up in terms of gender/ school/ social status and results at earlier key-stage results. I wonder how well they did?
Incidentally, if you peruse the programme and speaker info for the FFTB conference you will see that Dear Patrick’s bio sketch (on the speakers’ page) says:
Visiting Professor? As the audience cries “oh no he isn’t”, they suddenly remember that the FFtB conference is not supposed to be a pantomime and Patrick is not Widow Twankey. But then, the Durham fish oil initiative was not meant to be a trial and for some reason people forgot that it was an initiative and assumed it was a trial of some sort. That must have been very frustrating for the people who launched the initiative, having their idea misrepresented as a ‘trial’.
Oh, hang on a mo. Ben Goldacre checked Durham Council’s own press release for it. They called it a “trial” twice, and a “study” once. http://www.badscience.net/?p=297
This is awful. You would have thought Dr Portwood would have been able to find an excuse NOT to attend the Food for the Brain Conference. Obviously, she and those in charge at Durham County Council do not think they have done anything wrong. It does not like they are back down in the slightest. The saga will run and run until they get the result they want.
Does anyone know the attitute of Durham political parties to this initiative? Perhaps some one local could find out. The Liberal Democrats control the soon-to-abolished Durham City Council and have been running them close for the parliamentary seat: http://ukpollingreport.co.uk/guide/seat-profiles/durhamcityof
Obviously, scientific truth should not be a political issue but if I were an elector or opposition politician in County Durham I would be none too pleased at how the local political leadership were turning the area in to a laughing stock. I think there are local elections next year.
It is the whole bogus PR circus aspect of it that gets up my nose. And the credulity with which sections of the press parrot the press releases
(see multiple past Holfordwatch stories).
As alluded to above, it stretches credulity that the Fish Oil Zombie has re-appeared a mere three weeks before the FFTB meeting, where la Portwood is due to speak to the assembled Nutritionistas.
Note that the FFTB conference, apart from being run by Dear Patrick’s acolytes (FFTB and Biocare), is supported by Equazen (and Yorktest of ill repute). The gang’s all here, as it were.
Or… evidence, once again, that everything Evil Big Pharma can do – run “vanity PR” meetings with hand-picked speakers, all designed to act as a surreptitious plug for the company’s drug(s) and message(s) – “Big Quacka” can, and does, do too.
@Frank Spencer – the annoying thing is that we simply don’t know. We know about the schools that agreed to participate in the initiative and some of the individuals but we are in the dark as to some very important detail. For instance, others may, but I know nothing about the demographics of how many children in the relevant schools are vegetarian (maybe for faith/cultural reasons) and would therefore not participate in the trial and might be less likely to read the literature. (The over-hyped Lever Park School study in Greater Manchester did use a flax oil formula for their Omega 3 supplementation.)
And yes – it is entirely possible that using matched pairs in the scenario that you outline would reduce the effect size. Again, because of the lack of detail, we simply don’t know.
@Dr Aust, jdc – at some point I expect the story about the professorship to be transformed into some amazing account that either proves censorship or something that redounds to Holford’s integrity.
@Lee – I don’t know when, but at some point I will finish researching my history of fish oil and unleash it upon the world. Suffice it to say that in addition to the rich folkloric history of fish oils, the first ‘hospital’ trial of it for a specific disorder was conducted in Manchester in 1705 (so far, the earliest that I’ve found). For all sorts of unbelievably interesting reasons that tumble across the history of science, politics, fortunes of war, scientific cul de sacs, the treatment of the first formally-qualified female research academics, the history of fish-oil research has been remarkably tendentious – and 300 years on there is no change in that.
Oddly enough fish oil – as what we would call a nutritional supplement – has been available in the UK for hundreds of years.
@Dr Aust, on the question of supporters, backers etc., it is particularly irritating to see charities such as Allergy UK, Mental Health Foundation and Mind listed among the credulous.
Then again, I feel that way about some of the speakers.
If any journalists have been offered freebie tickets – it would be interesting to read what they make of it. I would expect some of the academics to give appropriately nuanced interpretations of the state of the art and their findings. I would expect other, less qualified, speakers and enthusiasts to display no such reticence and to over-claim hopelessly. It would be interesting to see if the other speakers have any qualms about that.
I see there’s a raffle with “healthy” prizes…
I work in one of the schools which ‘participated’ in the trial/study/initiative.
The pupils had the bottles of fish oils doled out to them at intervals and were expected to take them at home. We didn’t monitor compliance. We have absolutely no way of knowing which pupils actually took them and which didn’t, nor am I aware of any follow up (pupil questionnaires, etc.) to monitor compliance.
The school prefers to think that our good GCSE results were a result of good teaching; and, indeed, our results have continued to improve in subsequent years without the ‘benefit’ of fish oils.
I would like to see the details of the study/trial/initiative.
The whole affair is very fishy…
(Sorry…)
Pingback: Durham choosing how to measure success in their fish oil (non)trial: third time lucky? « Holford Watch: Patrick Holford, nutritionism and bad science
I think that you all need some bottles of Omega3! It makes you think better! ;)
I belive in facts! I belive in what I can see with my own eyes!
My son – 5 years – hav got a NEW LIFE thanks to Omega3 and EyeQ!
He is NOT the only one! Lot’s of parents in Sweden have noticed remarkable effekts on their children.
My son has ADHD. Children’s with ADHD, Aspergers, Tourettes and Autism CAN get a “new life” thanks to EyeQ (omega3)!
I say: MORE OMEGA3 to the people! If you cares for your children: give him Omega3!
sorry for my bad english!
Beckan in Sweden
Like Beckan I have observed a considerable difference in a child with dyspraxic characteristics after taking a fish oil supplement. I would like to see good quality trials conducted to see if this helps children more generally or if it was a random change, coincidentally at the time they were given fish oil.
Bad science does not mean that fish oil may not be beneficial to some children, only that it is not proven.
Few people would argue otherwise. However, it is not established that it is and before recommending something as a therapeutic intervention, it should be assessed not only for clinical efficacy but for real world practicality.
Yes – there is a strong need for trials. Until such time as those trials are done, there should not be misleading information on fish oil advertising, and parents should be fully informed as to the state of the clinical evidence.
Sure – there’s some interesting research around fish oil, and more good quality research would be valuable. That just makes some of the poor research and overblown marketing we’ve seen even less excusable, though.