Dear Daily Mail Editors: congratulations on a very dramatic headline. A cancerous conspiracy to poison your faith in organic food: that is pure genius, building nicely on the recent reprimand to ‘the authorities’ for making us Scared to death? The REAL worry is today’s culture of fear. You will understand how many readers chuckled to read that the Daily Mail, of all newspapers, is accusing others of scare-mongering.
Joanna Blythman seems to have uncritically included the figures that were quoted in the Soil Association’s press release. Unfortunately, the Soil Association made the somewhat odd decision to quote some figures that were not statistically relevant and to ignore the figures that are.[a] E.g., both Blythman and the Soil Association seize upon what they see as inconsistencies.
Yet even in the context of the latest report from the FSA, the spin does not match the reality. For, contrary to all the hype this week, the Agency’s own published research shows that organic foods are clearly far better for the consumer even just in nutritional terms. [Blythman]
“Although the researchers say that the differences between organic and non-organic food are not ‘important’, due to the relatively few studies, they report in their analysis that there are higher levels of beneficial nutrients in organic compared to non-organic foods. [Soil Association]
HolfordWatch has not seen the AJ Clin Nutr publication: Nutritional quality of organic foods: a systematic review but we have consulted the published data from the well-specified reviews[b] that are freely available from the Food Standards Agency (FSA).
According to the FSA’s findings, organic vegetables contain 53.6 per cent more betacarotene – which helps combat cancer and heart disease – than non-organic ones.
Similarly, organic food has 11.3 per cent more zinc, 38.4 per cent more flavonoids and 12.7 per cent more proteins.
In addition, an in-depth study by Newcastle University, far deeper than the one conducted by the FSA, has shown that organic produce contains 40 per cent more antioxidants than non-organic foods, research the FSA appears to have overlooked.
No. The report clearly states that the differences were not statistically significant for Beta-carotenes or proteins. The differences for zinc and flavonoids only hold if you were to include all of the included studies and comparisons, regardless of their quality[c] but were not relevant if the reader is only interested in the results from studies that are of ‘satisfactory quality’.[d]
Consider the Beta-carotenes example. Accepting even the non-significant difference figure for all of the included studies would be the equivalent of selecting one carrot from a barn that is filled with organic carrots of different cultivars, grown in different countries, in different soils, with different growing methods, of different stages of maturity, analysing that and then claiming that the nutrient profile for that carrot is true for that of all other carrots that you deem to be comparable.
For the zinc and flavonoids example, consider this. Drawing a retirement pension is strongly associated with mortality, based on reports from all sources, including newspaper articles. However, neglecting to mention that retirement pensions are normally drawn by people aged 60 or more and that older people do die would be foolish for many reasons. Most readers would realise that perhaps the association doesn’t hold once age is taken into account and perhaps it would be more relevant, in this instance, to assign credibility to figures and interpretations from appropriate sources such as the Office of National Statistics.
LSHTM very carefully detailed that they were presenting data from both all of the included studies and those that were objectively assessed as satisfactory-quality studies.[d] However, most readers will already know that there is little interest in statistical results that yield a p value greater than 0.05, particularly in a study that includes multiple comparisons for which it is not practical to apply a correction for multiple analyses.
So, does the FSA-commissioned review overlook the nutrient differences that it has reported? No.
The results are summarised in Table 2 (Comparison of content of nutrients and other substances in organically and conventionally produced crops) on pg. 19 and Table 3 (Comparison of content of nutrients and other substances in organically and conventionally produced livestock products) on pg. 20 and are very clear. However, for greater detail, I refer interested readers to what the authors report which makes it clear that Blythman and the Soil Association are quoting some non-significant results:
Analysis suggests that there is a significantly greater zinc content in organically than in conventionally produced crops ([11.3 mean percentage difference (se: 4.9)] p=0.03 for all comparisons). Analysis of data from
satisfactory quality studies suggests that there is no difference in zinc content between organically and conventionally produced crops ([10.1 mean percentage difference (se: 5.6)] p=0.11). [Emphasis added: Appendix 12-46]
Leaving aside the issue of the flavonoids and whether or not the purported differences would be clinically relevant to an adequately-nourished population, it seems that this is another finding that wasn’t statistically significant when the analysis was based solely on studies of suitable quality.
Analysis suggests that there is a significantly greater flavonoid content in organically than in conventionally produced crops ([38.4 mean percentage difference (se: 10.6)] p=0.002 for all comparisons). Analysis of data from satisfactory quality studies suggests that there is no difference in flavonoid content between organically and conventionally produced crops ([32.9 mean percentage difference (se: 21.0)] p=0.22 for comparisons from satisfactory quality studies). [Emphasis added: Appendix 12-12]
How about Beta-Carotenes? With the drumbeat of inevitability, the authors report that there is no significance for the differences for either the all-studies analysis or that in which only the sufficient-quality studies are included.
Analysis suggests that there is no difference in β-carotene content between organically and conventionally produced crops ([53.6 mean percentage difference (se: 37.0)] p=0.18 for all comparisons; [20.7 mean percentage difference (se: 38.8)] p=0.65 for comparisons from satisfactory quality studies). [Appendix 12-4.]
Proteins? Well, this is getting embarrassing.
Analysis suggests that there is no difference in specific proteins content between organically and conventionally produced crops ([12.7 mean percentage difference (se: 8.0)] p=0.12 for all comparisons; [-2.0 mean percentage difference (se: 4.6)] p=0.68 for comparisons from satisfactory quality studies). [Appendix 12-34]
We will have more to say on Blythman’s article as we have emailed the LSHTM team in connection with some of her assertions.
However, as the FSA stated when they commissioned this report, this is a matter of public interest. It is inappropriate to put figures about purported nutritional advantages of organic food into the public debate in the absence of any context – such as that the figures in question are not statistically significant. Perhaps wrongly, we had thought that even the Daily Mail would not be party to such misrepresentation of verifiable figures.
Aug 1: Ben Goldacre has written an elegant response to the Soil Association’s objections to the FSA-commissioned systematic review: Argument is about capitalism, not food.
We used t-tests with robust standard errors (to account for clustering caused by multiple nutrient comparisons within studies) to test the null hypothesis of no evidence of a difference between organically and conventionally produced food in content of nutrients and other substances. P-values were calculated to determine the significance of observed differences; p-values of less than 0.05 were used as a basis for evidence of significant differences between organically and conventionally produced foodstuffs. It should be noted that a large number of statistical tests were undertaken which increases the possibility of finding a significant difference where there is in fact no evidence of a difference between organically and conventionally produced food in content of nutrients and other substances.
To convey the totality of evidence, primary analysis was based on all included studies. A subsequent analysis only considered satisfactory quality studies. Statistical analysis was conducted separately for crops and livestock products. [pp. 12-13]
[b] The Food Standards Agency (FSA) commissioned the London School of Hygiene and Tropical Medicine (LSHTM) Nutrition and Public Health Intervention Research Unit to report on the public interest issue of whether or not there are significant differences in the nutrient content of organic v. conventionally produced foods and whether there are health benefits associated with these.
[The team] reviewed all papers published over the past 50 years that related to the nutrient content and health differences between organic and conventional food. This systematic review is the most comprehensive study in this area that has been carried out to date.
The FSA commissioned this research as part of its commitment to giving consumers accurate information about their food, based on the most up-to-date science.
This research was split into two separate parts, one of which looked at differences in nutrient levels and their significance, while the other looked at the health benefits of eating organic food. [Emphasis added.]
[c] The LSHTM laid out their search strategy, data extraction and data analysis for their systematic review in some detail in the report and had provided a consultation period. It is clear that the Soil Association and other interested parties had an opportunity to object to any part of this, had they so wished.
To date, there has been no explicitly systematic review of the available literature on this topic. In contrast to non-systematic reviews which can be biased and incomplete, the prime purpose of systematic reviews of literature is to provide a comprehensive display of all available evidence in a common format. Systematic reviews have clear principles for their conduct. First, the process of the review should be carried out according to a prespecified method. Second, the proposed method should be open to public scrutiny and peer review. Third, the review should be comprehensive within its pre-specified criteria. A systematic approach offers clear advantages in terms of reducing bias, where for instance inclusion or exclusion of studies may be influenced by preconceived ideas of the investigators. Systematic reviews cannot improve the quality of published data, but can provide details of the characteristics and quality of studies. [pg. 5]
In line with accepted guidelines, the review process was initiated by the preparation of a protocol which pre-specified the method to be used for the conduct of the review. The protocol was reviewed by an independent review panel. The review panel comprised a subject expert, Dr. Julie Lovegrove (University of Reading), and a public health clinician with systematic review expertise, Professor Martin Wiseman (World Cancer Research Fund International and University of Southampton). The review panel provided feedback on the protocol which was incorporated into a final version, posted on-line on 18th April 2008 at http://www.lshtm.ac.uk/nphiru/research/organic/. Relevant subject experts and external bodies were alerted to the review process and the availability of the review protocol. A draft of the final report was reviewed and approved by the independent review panel. [pg. 8]
HolfordWatch has not, as yet, seen any on-point criticism of that other than the Soil Association’s somewhat odd declaration that they had asked for publication of the report to be delayed for some considerable time so that the results of a project that ended this year might be included.
[d] Alan Dangour and his team detail their criteria for evaluating the quality of a study for the purposes of these reports.
Study quality was categorised based on concordance with five fundamental factors which were defined a priori as essential to answer the research question (i.e. comparison of nutrient and other substance composition of organically and conventionally produced foodstuffs). Study quality was grouped into two categories: satisfactory quality and unsatisfactory quality.
Satisfactory quality publications provided the following:
- a clear definition, in the Introduction or Methods section of the paper, of the organic production methods of the crop or livestock product analysed (including the name of any certifying body)
- specification of the cultivar of crop, or breed of livestock
- a statement of which nutrient(s) and other substance(s) were assessed for content
- a description of the laboratory analytical methods used to test for the content of the named nutrients and other substances
- a statement of the statistical methods used for data analyses.
Unsatisfactory quality publications were those that do not specify all of the above. [Section 4.6, pg. 10]
If Blythman or the Soil Association have a well-founded objection to the LSHTM authors’ specified protocol for reviewing the literature, including or excluding studies and how the authors evaluated the quality of studies, then it would be useful if they were to publish this.