Joanna Blythman: Please Read the Data Appendices About Organic Food Before Conjuring ‘Cancerous Conspiracies’: Part 1

Dear Daily Mail Editors: congratulations on a very dramatic headline. A cancerous conspiracy to poison your faith in organic food: that is pure genius, building nicely on the recent reprimand to ‘the authorities’ for making us Scared to death? The REAL worry is today’s culture of fear. You will understand how many readers chuckled to read that the Daily Mail, of all newspapers, is accusing others of scare-mongering.

Joanna Blythman seems to have uncritically included the figures that were quoted in the Soil Association’s press release. Unfortunately, the Soil Association made the somewhat odd decision to quote some figures that were not statistically relevant and to ignore the figures that are.[a] E.g., both Blythman and the Soil Association seize upon what they see as inconsistencies.

Yet even in the context of the latest report from the FSA, the spin does not match the reality. For, contrary to all the hype this week, the Agency’s own published research shows that organic foods are clearly far better for the consumer even just in nutritional terms. [Blythman]

“Although the researchers say that the differences between organic and non-organic food are not ‘important’, due to the relatively few studies, they report in their analysis that there are higher levels of beneficial nutrients in organic compared to non-organic foods. [Soil Association]

HolfordWatch has not seen the AJ Clin Nutr publication: Nutritional quality of organic foods: a systematic review but we have consulted the published data from the well-specified reviews[b] that are freely available from the Food Standards Agency (FSA).

Blythman writes:

According to the FSA’s findings, organic vegetables contain 53.6 per cent more betacarotene – which helps combat cancer and heart disease – than non-organic ones.
Similarly, organic food has 11.3 per cent more zinc, 38.4 per cent more flavonoids and 12.7 per cent more proteins.
In addition, an in-depth study by Newcastle University, far deeper than the one conducted by the FSA, has shown that organic produce contains 40 per cent more antioxidants than non-organic foods, research the FSA appears to have overlooked.

No. The report clearly states that the differences were not statistically significant for Beta-carotenes or proteins. The differences for zinc and flavonoids only hold if you were to include all of the included studies and comparisons, regardless of their quality[c] but were not relevant if the reader is only interested in the results from studies that are of ‘satisfactory quality’.[d]

Consider the Beta-carotenes example. Accepting even the non-significant difference figure for all of the included studies would be the equivalent of selecting one carrot from a barn that is filled with organic carrots of different cultivars, grown in different countries, in different soils, with different growing methods, of different stages of maturity, analysing that and then claiming that the nutrient profile for that carrot is true for that of all other carrots that you deem to be comparable.

For the zinc and flavonoids example, consider this. Drawing a retirement pension is strongly associated with mortality, based on reports from all sources, including newspaper articles. However, neglecting to mention that retirement pensions are normally drawn by people aged 60 or more and that older people do die would be foolish for many reasons. Most readers would realise that perhaps the association doesn’t hold once age is taken into account and perhaps it would be more relevant, in this instance, to assign credibility to figures and interpretations from appropriate sources such as the Office of National Statistics.

LSHTM very carefully detailed that they were presenting data from both all of the included studies and those that were objectively assessed as satisfactory-quality studies.[d] However, most readers will already know that there is little interest in statistical results that yield a p value greater than 0.05, particularly in a study that includes multiple comparisons for which it is not practical to apply a correction for multiple analyses.

So, does the FSA-commissioned review overlook the nutrient differences that it has reported? No.

The results are summarised in Table 2 (Comparison of content of nutrients and other substances in organically and conventionally produced crops) on pg. 19 and Table 3 (Comparison of content of nutrients and other substances in organically and conventionally produced livestock products) on pg. 20 and are very clear. However, for greater detail, I refer interested readers to what the authors report which makes it clear that Blythman and the Soil Association are quoting some non-significant results:

Analysis suggests that there is a significantly greater zinc content in organically than in conventionally produced crops ([11.3 mean percentage difference (se: 4.9)] p=0.03 for all comparisons). Analysis of data from
satisfactory quality studies suggests that there is no difference in zinc content between organically and conventionally produced crops ([10.1 mean percentage difference (se: 5.6)] p=0.11).
[Emphasis added: Appendix 12-46]

Leaving aside the issue of the flavonoids and whether or not the purported differences would be clinically relevant to an adequately-nourished population, it seems that this is another finding that wasn’t statistically significant when the analysis was based solely on studies of suitable quality.

Analysis suggests that there is a significantly greater flavonoid content in organically than in conventionally produced crops ([38.4 mean percentage difference (se: 10.6)] p=0.002 for all comparisons). Analysis of data from satisfactory quality studies suggests that there is no difference in flavonoid content between organically and conventionally produced crops ([32.9 mean percentage difference (se: 21.0)] p=0.22 for comparisons from satisfactory quality studies). [Emphasis added: Appendix 12-12]

How about Beta-Carotenes? With the drumbeat of inevitability, the authors report that there is no significance for the differences for either the all-studies analysis or that in which only the sufficient-quality studies are included.

Analysis suggests that there is no difference in β-carotene content between organically and conventionally produced crops ([53.6 mean percentage difference (se: 37.0)] p=0.18 for all comparisons; [20.7 mean percentage difference (se: 38.8)] p=0.65 for comparisons from satisfactory quality studies). [Appendix 12-4.]

Proteins? Well, this is getting embarrassing.

Analysis suggests that there is no difference in specific proteins content between organically and conventionally produced crops ([12.7 mean percentage difference (se: 8.0)] p=0.12 for all comparisons; [-2.0 mean percentage difference (se: 4.6)] p=0.68 for comparisons from satisfactory quality studies). [Appendix 12-34]

We will have more to say on Blythman’s article as we have emailed the LSHTM team in connection with some of her assertions.

However, as the FSA stated when they commissioned this report, this is a matter of public interest. It is inappropriate to put figures about purported nutritional advantages of organic food into the public debate in the absence of any context – such as that the figures in question are not statistically significant. Perhaps wrongly, we had thought that even the Daily Mail would not be party to such misrepresentation of verifiable figures.

Updates

Aug 1: Ben Goldacre has written an elegant response to the Soil Association’s objections to the FSA-commissioned systematic review: Argument is about capitalism, not food.

Notes

[a] Nutrient content of organically and conventionally produced foodstuffs.

We used t-tests with robust standard errors (to account for clustering caused by multiple nutrient comparisons within studies) to test the null hypothesis of no evidence of a difference between organically and conventionally produced food in content of nutrients and other substances. P-values were calculated to determine the significance of observed differences; p-values of less than 0.05 were used as a basis for evidence of significant differences between organically and conventionally produced foodstuffs. It should be noted that a large number of statistical tests were undertaken which increases the possibility of finding a significant difference where there is in fact no evidence of a difference between organically and conventionally produced food in content of nutrients and other substances.

To convey the totality of evidence, primary analysis was based on all included studies. A subsequent analysis only considered satisfactory quality studies. Statistical analysis was conducted separately for crops and livestock products. [pp. 12-13]

[b] The Food Standards Agency (FSA) commissioned the London School of Hygiene and Tropical Medicine (LSHTM) Nutrition and Public Health Intervention Research Unit to report on the public interest issue of whether or not there are significant differences in the nutrient content of organic v. conventionally produced foods and whether there are health benefits associated with these.

[The team] reviewed all papers published over the past 50 years that related to the nutrient content and health differences between organic and conventional food. This systematic review is the most comprehensive study in this area that has been carried out to date.

The FSA commissioned this research as part of its commitment to giving consumers accurate information about their food, based on the most up-to-date science.

This research was split into two separate parts, one of which looked at differences in nutrient levels and their significance, while the other looked at the health benefits of eating organic food. [Emphasis added.]

[c] The LSHTM laid out their search strategy, data extraction and data analysis for their systematic review in some detail in the report and had provided a consultation period. It is clear that the Soil Association and other interested parties had an opportunity to object to any part of this, had they so wished.

To date, there has been no explicitly systematic review of the available literature on this topic. In contrast to non-systematic reviews which can be biased and incomplete, the prime purpose of systematic reviews of literature is to provide a comprehensive display of all available evidence in a common format. Systematic reviews have clear principles for their conduct. First, the process of the review should be carried out according to a prespecified method. Second, the proposed method should be open to public scrutiny and peer review. Third, the review should be comprehensive within its pre-specified criteria. A systematic approach offers clear advantages in terms of reducing bias, where for instance inclusion or exclusion of studies may be influenced by preconceived ideas of the investigators. Systematic reviews cannot improve the quality of published data, but can provide details of the characteristics and quality of studies. [pg. 5]

In line with accepted guidelines, the review process was initiated by the preparation of a protocol which pre-specified the method to be used for the conduct of the review. The protocol was reviewed by an independent review panel. The review panel comprised a subject expert, Dr. Julie Lovegrove (University of Reading), and a public health clinician with systematic review expertise, Professor Martin Wiseman (World Cancer Research Fund International and University of Southampton). The review panel provided feedback on the protocol which was incorporated into a final version, posted on-line on 18th April 2008 at http://www.lshtm.ac.uk/nphiru/research/organic/. Relevant subject experts and external bodies were alerted to the review process and the availability of the review protocol. A draft of the final report was reviewed and approved by the independent review panel. [pg. 8]

HolfordWatch has not, as yet, seen any on-point criticism of that other than the Soil Association’s somewhat odd declaration that they had asked for publication of the report to be delayed for some considerable time so that the results of a project that ended this year might be included.
[d] Alan Dangour and his team detail their criteria for evaluating the quality of a study for the purposes of these reports.

Study quality was categorised based on concordance with five fundamental factors which were defined a priori as essential to answer the research question (i.e. comparison of nutrient and other substance composition of organically and conventionally produced foodstuffs). Study quality was grouped into two categories: satisfactory quality and unsatisfactory quality.
Satisfactory quality publications provided the following:

  • a clear definition, in the Introduction or Methods section of the paper, of the organic production methods of the crop or livestock product analysed (including the name of any certifying body)
  • specification of the cultivar of crop, or breed of livestock
  • a statement of which nutrient(s) and other substance(s) were assessed for content
  • a description of the laboratory analytical methods used to test for the content of the named nutrients and other substances
  • a statement of the statistical methods used for data analyses.

Unsatisfactory quality publications were those that do not specify all of the above. [Section 4.6, pg. 10]

If Blythman or the Soil Association have a well-founded objection to the LSHTM authors’ specified protocol for reviewing the literature, including or excluding studies and how the authors evaluated the quality of studies, then it would be useful if they were to publish this.

Advertisements

15 Comments

Filed under Current events, nutrition

15 responses to “Joanna Blythman: Please Read the Data Appendices About Organic Food Before Conjuring ‘Cancerous Conspiracies’: Part 1

  1. I actually have one of Joanna’s books on my shelf at home. Think I might burn it.

    • I’ve just been wondering about this. I’d been fairly well-disposed towards Blythman as a sort of warm, fuzzy feeling for Real Food campaigns but I have to admit that I haven’t read her books. I’ve been a little irritated that some of her articles have some nuggets of sense but seem to go a little bit off-message when it comes to verifiable facts: see, eg., Be wary of eating by nutrient and number.

      It seems odd that only a few months ago Blythman was cautioning:

      We don’t need any more advice that exhorts us to eat by nutrient and number. The principles of healthy eating can be distilled in one sentence: eat as little processed food as possible and base your diet on home-cooked meals made from scratch from raw ingredients.

      However, it seems the FSA’s transparently-commissioned, well-specified systematic review should be ignored in favour of misquoted mean differences of…nutrients.

      I would be sorry if there were similar lapses in her books because they have been endorsed by writers whom I admire, such as Nigel Slater, who is very sound on the subject of potato and particularly so on the topic of mashed potato and other foods that account for a fair few of my favourite foods.

  2. The comments on the article are possibly more depressing than the article itself: (list by approval rating) http://tinyurl.com/luh593

  3. Pingback: The crunchies get their panties in a bunch « The Cock Bucket

  4. James Hodson

    http://tinyurl.com/luh593 is fair enough as long one goes no further than let’s all be nice to little animals – or big ones, for that matter. And carrots.

    Joanna Blythman mentioned “common sense” a couple or times in her article of times and this reminded me of a line I came across in a blog about a year ago. It was along the lines of “Common sense is the best press blind faith ever received.”

    • Bill Jones

      Anything wrong with being nice to animals? Or are you happy with what Big Agriculture puts aimals through? Just asking.

  5. Pingback: Investigative journalism on 03 Aug 09 « The Centre for Investigative Journalism News Blog

  6. Pingback: Zone5 » The Real Dirt on Organic Food

  7. James Turner

    The reports on organic foods were rubbish. Both should be taken with a pinch of salth like everything else. Reason for non organic food (as in whole foods) its cheaper than organic, contains same amounts of nutrients and you may get lucky, some non organic farmer use the bare minimum like when insets mate and lay eggs (this come from a farming friend). Reason against non-organic farming, it contains “harmful chemicals” which could increase cancer or other health risk over time, etc. Reason for organic farming it maybe expensive but contains no or very insignificant traces of chemical (from down wind or other factors etc). Finished

    I was alittle annoyed at the organic study because of the things it missed personally. I was waiting to hear the full brunt of people showing it me and laughing, then me having to explain that I do it to eliminate the added chemicals.

    But what really drove me nuts was the follow up story. Yeah alot of these chemicals use to treat our food could have a serious long term effects on our health. But to absolutely slam a report like the writer did is beyond stupidity. People like this should rammed into the ground for scaring people like that. There are a hell of alot more serious issue which can cause cancer and/or other health risks which need to be address. All these bad things all add up but if your avoiding smoking, drinking, too much sun, crap diet, etc which contributes then eating foods with these chemicals is going to contribute to your health and if your doing it in the first place (I mean all this bad life styles combined) then you’ve probably given up on life and not expects to have a long life span.

    If the good out ways the bad then your laughing to see grand children.

    As for the mentality side of this, then it’s up to you guys to slam the writer even more, however though before you scare off people from listening, take the parts you see sense (if any) and bring them out. But then alot of people can repeat, so if there isn’t anything new then I’ll forgive :)

    Keep up the good work

  8. Bill Jones

    This is all very interesting. I am not a nutritionist and I really have no idea whether organic food has more nutritional benefit than non-organic. However, I do know the following:
    (a) Organic food will have no pesticide residue. I don’t know whether pesticide residue will kill me or not but I’d rather not eat it.
    (b) Organic food tastes better. Really, it just does!
    (c) If I buy a joint of beef from Tesco and roast it, it will shrivel down to nearly half its original size. An organic joint will not do this because it hasn’t been pumped full of water. It may be more expensive but at least it’s all meat. I don’t want to pay for water just for it to evaporate in my oven.
    (d) If I buy organic produce from a local producer, my money is going straight back into the local economy, not into the pockets of some big corporation. Eat organic and stand up to The Man!

  9. Pingback: How Joanna Blythman got it wrong, part 4 « Quant suff

  10. Pingback: UK Food Standards Agency: organic not healthier. Reactions from the press and blogosphere. | PI Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s