We were shocked to see a Sunday Express story by Laura Clout, stating that:
Research by natural health firm BetterYou [which sells nutritional supplements] found that more than eight in ten of us do not eat fruit and vegetables in our daily diet.
These is a really striking figure, so we asked BetterYou for the research behind it. They, and their PR agency Lucre Communications, were very helpful with our questions: the research was carried out online by The Leadership Factor, with a total sample of 1,000 adults. However, we found a number of issues with this research. As you’ll see in the above Yes Minister clip, there is a noble British tradition of surveys which reliably give certain answers and – while this may be entirely accidental – The Leadership Factor and BetterYou appear to have followed this great tradition.
The 8 in 10 figure came from responses to the question
Out of the following, what do you mainly rely on to keep you going?
– “Lots of water” (24.6%)
– “Endless cups of tea. coffee [sic]” (33.5%)
– “Plenty of fruit and vegetables” (19.7%)
19.7% of respondents picked the latter, and the Express article and BetterYou press release therefore reported that 8/10 people did not include fruit and veg in their daily diet. This is clearly unjustified: drinking lots of coffee clearly does not prevent one from eating lots of fruit and veg as well.
The BetterYou press release used the same question to report that
Worryingly, only a quarter drink water regularly
However, once again, lifestyle choices such as eating lots of fruit and veg or drinking lots of coffee do not prevent one from drinking ample water alongside this.
Subjects were also asked
When it comes to following the ‘Five a Day’ rule, which of these statements most applies to you?
The only options available to them were:
– “What is ‘Five a Day’?” (4.7%)
– “I know what ‘Five a Day’ is but don’t follow it” (35.2%)
– “It’s too expensive for me to stick to every day” (21.5%)
– “I try to follow it but I/my family don’t really like fruit and veg” (20.4%)
– “I forget to follow it because I’m too busy” (18.2%)
Subjects apparently had no other options: all 1,000 picked one of these options, and no other questions are listed in the spreadsheet reporting the research results. There was apparently no option to say ‘I know what 5-a-day is, and eat at least that much fruit and veg’. In order to generate valid results, it would also have been good to see the survey seeking further information – asking, for example, about subjects’ understanding of what constitutes a portion. It would be most surprising if not one of the 1,000 people surveyed both knew what 5-a-day is and ate that much fruit and veg.
However, despite these serious issues, the press release and Express article used this research to claim that (to quote from the press release, rather than Express)
an astounding 60 per cent don’t follow the Government’s Five A Day guidelines because they either can’t afford to, don’t like fruit and vegetables, or because they are simply too busy
A final point to note is an interesting comment from a market research professional on Gimpy’s blog – claiming that
“in the halls of academia” researchers do not need to be governed by the rules of the Market Research Society? So research from a reputable commercial agency is more likely to have the checks and balances in place to ensure fair reporting
The Leadership Factor – responsible for carrying out this research – is a member of the Market Research Society. The flaws in this survey design are so significant that any competent researcher should have spotted them. I wonder where the Market Research Society stands on such dismal research from its members – and if this Society, along with other reputable market research companies, will speak out against this embarrassingly poor research?
One can also note that academics who have been through the rigorous ethics review process that is generally required for research on human subjects might be surprised to hear about how unregulated they are. I wonder how the Market Research Society’s approach to regulation compares, and what the consequences are for members who fail to ensure fair reporting?