BIS and Science: So What’s definition of “rigorous and credible” research

I had high hopes for the government’s Science: So What? So Everything campaign – a good science communication initiative would be very welcome. However, its recent Shape of Jobs to Come PR splash has heavily promoted strikingly poor research – even quoting Gordon Brown and Lord Drayson in support of this research! Most worryingly, the Department for Business, Innovation and Skills (BIS) – who run Science: So What? – continue to insist that this research is ‘rigorous and credible’, even after major flaws in the research have been made clear to them. This raises serious questions about whether BIS understand what constitutes good research and/or whether they care about this. Promoting bad research in the name of science communication is likely to be counterproductive.

The Shape of Jobs to Come promotion was based on a 150 page report [PDF] from Rohit Talwar and Tim Hancock from Fast Future. BIS commissioned the report for only £7,500 so there were always going to be limitations on how much could be achieved but, even given budgetary restraints, this report is strikingly poor quality. There are too many problems with the report to discuss here, but some examples are:

  • As Gimpy notes, the report fails to provide appropriate evidence for its arguments about limb regeneration.
  • James Hayton points out that the report’s claim regarding “Advances in nanotechnology for creating sub-atomic devices and treatments” is “so utterly impossible it defies belief”.
  • A social scientist’s review of the report finds that: the methodologies used are inadequate and poorly implemented; some of the ‘future jobs’ suggested have been around for a good while some are implausible; the report fails to back up many of its assertions with good quality evidence; and the referencing of the report is inadequate (including references to Wikipedia, among other concerns).

Despite the problems with the report, it has been promoted throughout the media: achieving coverage (among many outlets) in the Guardian and Mail and discussed on the Today Programme. Gordon Brown was quoted [PDF] saying that

The shape of jobs to come shows what might be on offer for the next generation. I hope it will inspire young people to gain the skills and training they will need to succeed.

while Lord Drayson responded to the report by stating that

These jobs are no longer the stuff of dreams

Given the problems with the report and its heavy promotion, I asked Science: So What? for a comment. The BIS comment is reproduced in full below. When BIS made this comment, they were aware of the concerns discussed above.

The Science: [So what? So everything] campaign commissioned future research company, Fast Future, to conduct The shape of jobs to come study to show the types of jobs we might be doing in 2030. The purpose of the study is to engage the public with science and highlight the wealth of jobs scientific developments could create in the future.

This work is a speculative look to the future and does not inform policy or impact on government funding decisions. The report was checked by the Department for Business, Innovation and Skills and felt to be rigorous and credible for the purposes we required.

For the reasons given above, it is entirely inappropriate to describe this research as ‘rigorous and credible’. I am not sure what the checking of the report involved, but it appears that it failed to spot some significant problems. Given that some of these issues are not subtle – relying on Wikipedia, for example! – and one therefore has to worry about whether those checking the research were competent and motivated to do so.

Even if the BIS review process was inadequate, one should note that they continue to stand by the research even after its flaws were pointed out: insisting that it is “rigorous and credible”. We have previously criticised Science: So What? on this blog: we have tried to be constructive, as we were hoping to see real improvements. However, with the mis/use of research in this Science: So What? campaign and BIS’ response to criticism, I would argue that things have considerably deteriorated – not only are Science: So What? getting things seriously wrong, but they are refusing to acknowledge when mistakes are made.

Good science communication should be based upon and use good research: there is no need to promote bad research in order to get ‘the kids’ interested. Science and social science are exciting and important: they help us to understand and to think about the world around us. If BIS cannot find some good current or future research that is considerably more compelling than the Shape of Jobs to Come report – and if BIS believe that it is appropriate to use research as poor as the Shape of Jobs to Come in order to promote UK science – they should not be running a science communications campaign.

Edit: now also covered by Gimpy.

Advertisements

1 Comment

Filed under Science: So What?

One response to “BIS and Science: So What’s definition of “rigorous and credible” research

  1. Pingback: Science So What – better at headlines than science « gimpy’s blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s