In a fine example of convergent evolution, some information posted on HolfordWatch, the digging by Letting off Steam and Ben Goldacre’s tenacity have a welcome airing as the Durham fish oil zombie once again swims through the nutrient-rich waters of the Bad Science blog. This week, Ben Goldacre returns again to the Durham fish oil (non)trial:
And so an epic saga comes to a close. You will remember the Durham Fish Oil tale – don’t switch off now, the punchline’s funny. The county council said it was doing a “trial” of fish oil pills in children, but the trial was designed so that it couldn’t possibly give useful information – not least because it had no placebo group – and was very likely to give a false positive result.
However, we don’t feel that the story is over quite yet. There is still plenty of information to come out: although any hopes of meaningful results from this trial are long dead, the Durham fish oil zombie still swims this morning and will return to swim another day. For starters, we would like to tell you the story of the fabulous mutating press release to come out.
Goldacre notes that
Durham has finally announced, in a formal response to a written question to the county council, that the trial in 2,000 children was never intended to produce any data on children’s performance. Specifically they said this: “It was never intended, and the county council never suggested, that it would use this initiative to draw conclusions about the effectiveness or otherwise of using fish oil to boost exam results.”
[But] Durham county council’s own press release from the beginning of the “trial” reads: “Education chiefs in County Durham are to mount a unique back-to-school initiative today which they believe could result in record GCSE pass levels next summer.”
This press release has an interesting history: it rather mutated as Durham county council changed their story about this (non)trial. As Goldacre notes on an earlier post, the press release originally said that
The County-wide trial will continue until the pupils complete their GCSE examinations next June, and the first test of the supplement’s effectiveness will be when they sit their mock exams this December…The trial has won the backing of Durham County Councillors…All Year 11 pupils at Durham County Council’s 36 comprehensive schools are to be offered omega-3 fish oil supplements to see whether the proven benefits it has already brought children and young people in earlier trials can boost exam performances too. [Emphasis added]
Sounds like a trial to find out if fish oil improves exam results, right? However, the press release rather mutated following controversy about this (non)trial: I noticed that the current version on Durham’s website refers to an initiative instead of a trial:
The County-wide initiative will continue until the pupils complete their GCSE examinations next June, and the first test of the supplement’s effectiveness will be when they sit their mock exams this December…The initiative has won the backing of Durham County Councillors [Emphasis added]
I therefore submitted a Freedom of Information Act Request to Durham. In response, Durham’s Press Officer Fraser Davie states that:
I used the word ‘trial’ in its loosest sense that I know journalists have for years readily understood, commonly accepted and regularly employed themselves
If you look back through newspaper libraries you will find countless similar examples, where the word trial has been used in this sense and has been taken to mean a trial run or a test.
Similarly, when journalists and newspapers use the word ‘experiment’ or ‘experimental’ , they do not necessarily mean its more scientific application where the people are working in laboratories with scientific apparatus.
Perhaps my reasoning is best summed up in the Concise Oxford Dictionary, where the word ‘trial’ can be taken to mean “an investigation by means of experience”…which was precisely what the Year II Fish Oil ‘trial’ or ‘initiative’ was intended to be.
Unfortunately, my use of the word [in the original version of the press release] was seized upon in some more specialist scientific’ quarters and criticized for being misleading at least, and devious at worst.
In hindsight, I accept that perhaps, in those quarters, the use of the word ‘trial’ carries with it a different and more precise definition…Anxious to avoid any further misunderstanding among the more scientific minds – and for no other reason – I changed the word ‘trial’ to ‘initiative’ in the version of the press release that was mounted on our website after the embargo expired on the day of the launch itself…I would point out that both words – ‘trial’ and ‘initiative’ – were used in the original embargoed release to describe what we were doing.
All we wanted to show, and all we will be able to state, is that pupils who continued taking the fish oil in the run-up to their exams achieved X and those who did not elect to be involved or dropped out achieved Y.
We’re quite happy for councils to try out new food ideas in their schools – for example, to work to improve the quality of school meals – although we would prefer to see food-based initiatives rather than pill-based (non)trials. However, I would reiterate that Durham’s press release states that they
wanted to “see whether the proven benefits it has already brought children and young people in earlier trials can boost exam performances too”. You need a proper trial to do that – not a (non)trial with no placebo group and no randomised control group. If children, parents and taxpayers were told that the (non)trial would achieve this, this was – perhaps unintentionally – misleading.
Now, most normal people might see this as the end of the matter. However, to my mind, the zombie of this fishy trial that never was is still flapping around on the banks of the river Wear. As Davie acknowledged in the above quote, Durham
wanted to show, and…will be able to state…that pupils who continued taking the fish oil in the run-up to their exams achieved X and those who did not elect to be involved or dropped out achieved Y.
As Goldacre notes, “the GCSE results for Durham were rather disappointing this year”. It will therefore be interesting to see how the results of this (non)trial are reported: will Durham and Equazen still publicise the results of the (non)trial as widely as they publicised the (non)trial itself if the use of the fish oil pills in Durham is not correlated with improved exam performance, or if it is correlated with impaired performance? And how will the media report the results of the (non)trial that they publicised so lavishly last year?
Update 1 April: A local paper takes up the story of curious absence of any report of the GCSE results: Silence on effect of fish oil for students
Update 3 April: The Advertiser has picked up the story in Durham