Oxford University, Food for the Brain, Alzheimer’s Disease and a Curious Test

The People’s Medical Journal (aka Daily Mail) has a touching faith in the value of early diagnosis and screening tests. It would be rather charming to note that their history of being wrong has not as yet reduced them to cynicism if it were not for the errors and false hope that they present as verified fact to their readers.

Food for the Brain (CEO Patrick Holford) claims to offer an online test to detect the early symptoms of Alzheimer’s Disease. Alzheimer’s Disease is a subject that excites great concern and, by definition, those who research in this area are aware that they are typically dealing with some vulnerable people.

Dr Margaret McCartney recently examined the claims for Food for the Brain and self administered cognitive tests after some enthusiastic media reports and found them unwarranted, and not in line with available evidence. She evaluated the claims again in the BMJ, An early warning for Alzheimer’s disease, and questioned FFTB about the claims made for the test in the diagnosis of Alzheimer’s Disease or its prodromes and the evidence base for its recommendations.

Patrick Holford, who describes himself as a “nutritionist” and chief executive officer of Food for the Brain, told me, “We, the charity, deemed the evidence to have become substantial enough to warrant the launch of our Alzheimer’s prevention project . . . the primary aim of which is to encourage early screening of cognitive function from age 50, followed by homocysteine testing.” Food for the Brain’s adviser, the pharmacologist David Smith, told me that the online test is “not a diagnostic test, and there is no definitive outcome; it simply tells the user about their cognitive status.

So, media coverage (eg, 15-minute online test for dementia: DIY memory quiz detects early signs of Alzheimer’s in people as young as 50; Online test for early signs of Alzheimer’s) persuaded people to take an online Cognitive Function Test (CFT) developed by an “Oxford research team” as a way of detecting the early symptoms of Alzheimer’s Disease and as a way of persuading them to take/pay for non-evidence-based tests and supplements. However, Professor David Smith, Chairman of FFTB’s Scientific Advisory Board, admits that the test has no diagnostic value. (Holfordwatch readers with a good memory will recall that David Smith has previously admitted that FFTB has not done a “proper job” of research it attempted.)

As is too frequently the case when discussing Patrick Holford and Food for the Brain, it is difficult to outline all of the misunderstandings and errors that accompany their claims. We can’t begin to cover all of the issues which, inevitably, also involve: the inappropriate promotion of the crystal-ball of homocysteine testing as a biomarker; the advice that test takers should ask their GPs for a test that is not available for that purpose on the NHS or have recourse to Yorktest private testing; the promotion of supplements. This is not the time to explore the ethical concerns that must accompany the availability of a direct to consumer test that purports to diagnose such a widely-feared condition and has already caused some distress while also illustrating a worrying (and perhaps unwarranted) confidence in the significance/value of the test.[1]

The following is not a complete account as the story is still unfolding, however, even these items highlight the contested nature of this test and why the marketing/media coverage of it is inappropriate. The inconsistencies and recent redactions must also question whether the test ought still be available until such time as various issues are clarified. It isn’t clear why Oxford is failing to protect its reputation as it can not be to its advantage to be associated with such a questionable test and set of recommendations.

i) Mid-May various media outlets covered FFTB’s Cognitive Function Test (CFT): they reported that it had been developed by an “Oxford research team” and was made available online, direct to consumers, and promoted as diagnostic of Alzheimer’s Disease and its prodromes.

Food for the Brain (FFTB) emphasised the involvement of Oxford University in news stories and the then current version of its own website: Patrick Holford has likewise stressed the involvement of Oxford Uni. in his marketing materials for his own website and for FFTB for which he is the CEO. Oxford researchers are said to have played the role of lead developer.[2]

However, approaches to several people, including Virginia’s Professor Timothy Salthouse and Oxford’s Dr Celeste de Jager, subsequently revealed that neither of them played such a substantial role in the development of the test.

Professor Salthouse reports that he granted permission to the authors to use an adapted version of his perceptual comparison tests. However, he emphasises that that is the extent of his involvement and makes no claims with respect to the role of his tests in assessing the risk of memory decline or the development of Alzheimer’s Disease.

Dr de Jager’s involvement was recently clarified by an amendment to the FFTB website:

The CFT composes three elements:
A Episodic memory, using cued recall and paired associate learning test constructs, developed by Catharine Trustram Eve for FFB, with the advice of Dr Celeste de Jager.

So, it seems as if the test, developed by an “Oxford research team”, was substantially the work of Catharine Trustram Eve who is listed as an “Independent Market Research Professional” albeit that is not made clear in the Letter to GPs that test-takers are advised to give their doctors. Catharine Trustram Eve’s profile does not list any qualifications in neurology, psychology, cognitive science or similar relevant disciplines.

How many test-takers took the test because they were reassured by the much publicised Oxford provenance of the CFT? What, if anything, is Oxford doing to dissociate itself from the CFT? Even today, the Daily Mail is linking Oxford and this test: Test to detect early onset of Alzheimer’s for all over 65s to be introduced within two years.

One of the items in the box insert states:

“Oxford University has devised a memory test that can be taken at home in 15 minutes and can spot the signs of Alzheimer’s in people as young as 50.”

Both Patrick Holford and Food for the Brain promoted the CFT to their mailing lists. Why have neither Patrick Holford nor Food for the Brain issued corrections to their mailing lists to clarify the provenance of the test?

ii) The CFT is said to be validated for a specific age range and it is promoted as diagnostic in news coverage and on the FFTB website.[2], [3] However, as Dr McCartney explains, at the time of the news items, the validation for this test was not available and there were no data relating to sensitivity or specificity (false positives and false negatives).

The Daily Mail story (inter alia) refers to pilot studies:

The researchers cannot put a figure on the test’s accuracy, but in pilot studies it worked as well as tests already used in GP surgeries and specialist memory clinics.

In the absence of detail that would allow for appropriate scrutiny, the claims for validation were premature at best. Irritatingly, in late May the FFTB site was amended to state:

“based on the pilot, it appears that the CFT is sensitive to MCI. A full description of the pilot and analysis will be available from this page by 1st July 2011.”

http://www.foodforthebrain.org/content.asp?id_Content=1825

However, following the latest updates to the FFTB website, we now learn that we are not to be permitted to see the detail that ‘validates’ these tests for some time:

“Previously it was stated that ‘A full description of the pilot and analysis will be available from this page by 1st July 2011.’ However, in light of an expert academic critique of the CFT validation, we have decided to submit the work for publication prior to publicising the results on the website.”

http://www.foodforthebrain.org/content.asp?id_Content=1825

Prof. Salthouse’s contribution to the test is well validated within its usual sphere of use. Prof. Salthouse makes no claims for the integration of his test within the CFT and states that his comparison tests should not be treated as valid predictors of the risk of memory decline or the development of Alzheimer’s Disease. Under the circumstances, it is arguably imperative that the pilot studies and materials that underpin the “validation” of the CFT should be made available. However, the absence of accessible validation is not made clear to the public, nor, perhaps, to some NHS Commissioners who are given funding proposals for which there has not been adequate due diligence.

iii) Media coverage, Patrick Holford and the FFTB promoted the CFT as diagnostic of Alzheimer’s Disease or its prodrome when the CFT has not yet been publicly validated as an appropriate instrument to identify or quantify mild cognitive impairment.

David Smith modified those claims when challenged by Margaret McCartney. FFTB updated its website July 1 and, in line with Smith, has considerably modified its claims for the scope of the CFT:

“Does the CFT diagnose dementia, Alzheimer’s disease or Mild Cognitive Impairment?
No. The CFT is not a diagnostic test, but a test designed to inform/educate the user about their cognitive function. If the result is below a threshold we suggest that they visit their GP who can perform whatever diagnostic tests are required at their discretion.

However, both Patrick Holford and Food for the Brain promoted the CFT to their mailing lists: the former was headed, “15 minute free test to prevent dementia” and the latter “A 15 minute free test could stop you ever getting Alzheimer’s”. Why have neither Patrick Holford nor Food for the Brain issued corrections to their mailing lists to update them as to the reduced scope of claims for the test?

Allegedly, many people have taken this test. Perhaps the test-takers were reassured that it was developed at Oxford University, and that it is a validated test for Alzheimer’s and its prodromes. Patrick Holford claimed (in a later deleted blog post) that 55,000 people took the CFT in 10 days. Since then, FFTB has claimed that more than 70,000 people have taken this test.

A test that plainly was not developed by an “Oxford research team”. A test that is now said not to be a test for what people were told that it was but now “simply tells the user about their cognitive status” albeit the validation for that is not available.

Why hasn’t Oxford contacted media outlets such as Daily Mail and Telegraph to instruct them to correct their stories if Oxford’s involvement is as limited as the current version of the FFTB website implies? Were the pilot studies on which the ‘validation’ rests conducted at Oxford, and, if so, did Oxford oversee their clinical governance?

Why haven’t Patrick Holford or Food for the Brain alerted media outlets that they have modified their claims concerning: the provenance of the test; the scope of the claims for the CFT’s diagnostic purpose; and that the claims for validation ought to be held in abeyance until such time as the details are published?

Why haven’t Patrick Holford or Food for the Brain alerted their mailing lists as to these substantial revisions concerning the CFT? Has there been any attempt to contact GPs who’ve received one of these letters from a patient to inform them of the modified status of these claims? (The GP letter is still available on the FFTB site and still contains claims that are out of date.) If not, why not?

The CFT has created anxiety amongst some users and has given false confidence to others. Rather curiously, FFTB brags of the thank you letters it has received although it now seems as if they were being thanked for reassurances that can no longer carry any weight given the modifications and reduced scope of the claims for that test.

A curious test and a curious business. There will be more to come when more information is available about the ethical approval for this test and other pertinent matters.

Notes

[1] Sample quoted from

http://www.womanandhome.com/forums/showflat.php/Cat/0/Number/726038/Main/725925/

[Northwindrider] Mine said I was at low risk of developing Altzeimers which is quite comforting as I have a Grandmother with it and know that my Mum was in the early stages when she died.

[snowy47] I have completed the test and i have a very low chance of developing Altzeimers, my D passed last year and he put my M through hell with it.

[susieblue] I have just done the test and told I was low risk. But I am pretty sure my mother would have been told the same had she done it at my age too. Read an article about it on Yahoo. Complete rubbish! For starters my mother, aunt and uncle all had/have it.
[aec13cat] Was so curious in the end I took it but it shows I’m at risk -totally depressed now and wish I hadn’t taken it

Following from

http://forum.alzheimers.org.uk/showthread.php?33976-Someone-look-at-this-for-me-please

[Danny] I took the online test myself,I scored 37/110. It was a bit worrying to be told I could be at serious risk of developing Alzheimers. It has worried me to bits.It will teach me to stop researching so much.

[Tony] I just took that test and scored 29 yesterday
had my 6 monthly test for memory clinic scored 30/30 now I’m confused with the results was the memory clinic test 2 easy

[Gill66] I am suitably happy with a score of 88. Use a mouse, it’s a lot easier. With a history of dementia in the family i feel quite content this evening

Quoted from Patrick Holford’s blog:

“The positive response to my test results came as a great relief, as my father, uncles and paternal grandfather all developed symptoms of senile dementia of one form or another, when they were precisely the age I am now. I feel a great weight has been lifted from my shoulders, at least for the foreseeable future.” Yours, Hugh G. “A really informative website – and the opportunity to put my mind at rest by doing the cognitive test was priceless. I found the test itself very well introduced and explained. The examples are particularly helpful. I’m sure I’m not the only one who approached the test with some anxiety but I found I was far less panicked than I expected.” said Marion. Ivor, age 75, said “The nightmare of Alzheimers has been put to sleep by the results of this test. Thank you.”

http://www.patrickholford.com/index.php/blog/blogarticle/951/ – if no longer available, please see

http://www.freezepage.com/1307103965TWDDWXGIGE

It is worth noting that Ivor is outside the age range for the test yet Patrick Holford nonetheless includes this testimonial.

[2] “The Cognitive Function Test assesses three critical areas of cognitive processing associated with cognitive decline leading to Alzheimer’s disease…This test has been developed in collaboration with Dr Celeste de Jager from the University of Oxford, Professor Timothy Salthouse from the University of Virginia and Catharine Trustram Eve.”

Original link for text: http://www.foodforthebrain.org/content.asp?id_Content=1824

Freezepage for the page May 20 2011: http://www.freezepage.com/1305892729EKYIERRJCR

A similar claim is made in the results letter than test-takers are advised to give their GPs:

“Your patient has completed the Cognitive Function Test at http://www.foodforthebrain.org, an educational trust whose mission is to promote the link between mental health and nutrition. This is a validated screening test for those aged 50 and above, designed to detect early cognitive impairment. This test has been developed with Professor Timothy Salthouse and Dr Celeste de Jager, specialists in assessment of cognitive function.”

Original link for text: http://cft.foodforthebrain.org/doctors-letter-r.aspx?name=Patrick%20Holford&dob=4/5/1953

Freezepage for the page May 20 2011:
http://www.freezepage.com/1305892790UAAUYMCITZ

Daily Mail and other accounts credit “Dr Celeste de Jager, [as] the main developer of the Cognitive Function Test”.

http://www.dailymail.co.uk/news/article-1386912/15-minute-online-test-dementia-DIY-memory-quiz-detects-early-signs-Alzheimers-people-young-50.html

Patrick Holford blog post 15 Minute Online Test for Dementia:

“The test, available from http://www.foodforthebrain.org, also tells you how to delay memory decline and possibly reduce Alzheimer’s risk, based on research of people with mild cognitive impairment, the stage before Alzheimer’s, by Oxford University experts Professor David Smith and Dr Celeste de Jager.”

Freezepage for the page 20 May 2011: http://www.freezepage.com/1305892842FQLAHXZRQC

[3] “The Cognitive Function Test assesses three critical areas of cognitive processing associated with cognitive decline leading to Alzheimer’s disease…The test has been validated for the age range of 50 plus. ”

Original link for text: http://www.foodforthebrain.org/content.asp?id_Content=1824

Freezepage for the page May 20 2011: http://www.freezepage.com/1305892729EKYIERRJCR

The Daily Mail account of the CFT (in common with the Telegraph and other news outlets) explicitly claims that the test can detect early signs of Alzheimer’s which implies that the test is diagnostic.

“An early warning test for Alzheimer’s that can be taken online in 15 minutes has been developed by British scientists.
It can spot signs of the debilitating brain disease in people as young as 50.
The computer-based interactive quiz provides an instant result and could help delay or prevent the condition by advising simple diet and lifestyle changes.”…
“But most are still in the early stages of development and none, other than the new Cognitive Function Test, which has been devised by Oxford University scientists, can be taken online in the comfort of a person’s own home.
This is likely to make it popular with those who fear their memory is failing but are too embarrassed to discuss their worries with their doctor.

[B]ecause apparently healthy people have no way of telling if they are among those who could benefit from the vitamin B memory boost, the Oxford research team created the test.
It measures mild cognitive impairment – or the slight memory lapses that can be a precursor to Alzheimer’s – which affects one in six aged 70-plus, or 1.5million Britons. Half will develop dementia within five years of diagnosis.”

http://www.dailymail.co.uk/news/article-1386912/15-minute-online-test-dementia-DIY-memory-quiz-detects-early-signs-Alzheimers-people-young-50.html

Advertisements

4 Comments

Filed under Alzheimer's, Food for the brain, patrick holford

4 responses to “Oxford University, Food for the Brain, Alzheimer’s Disease and a Curious Test

  1. Good post.

    I think I know what’s going on here.

    Holford has a history of…let’s say…profiting from well-meaning academics.

    Here’s how I think it works – he approaches them with an offer of a “collaboration” which will involve allowing him to use their name and/or their tasks.

    Academics are always been asked this, and most people will agree with it almost reflexively because a) it’s usually in good faith and b) most of these promised “collaborations” never happen, so you need to agree to lots of them, if you want any to bear fruit. And the fruit is citations of your papers and a better profile.

    Anyway, Holford wants to use their name (and their institution’s name) to sell something. It’s no coincidence that Oxford academics have been involved in this case but also in past cases e.g. Prof Phillip Cowen was in the firing line last time.

    I think it is rather unfair to blame the academics for not speaking out against it. Almost certainly they are just not aware of what’s going on. they entered into this in good faith and then forgot about it and went back to their real research – that’s what academics tend to do.

  2. @Neuroskeptic Oxford Clinical Governance Dept. and relevant academics were notified of problems relating to the claims made for the FFTB Cognitive Function Test in May and June.

  3. @Neuro

    “I think it is rather unfair to blame the academics for not speaking out against it. Almost certainly they are just not aware of what’s going on. they entered into this in good faith and then forgot about it and went back to their real research – that’s what academics tend to do.”

    I agree with that in general, but one has to say that David Smith (now Emeritus Prof of Pharmacology) has a very long-term association with Holford, and must have been told many times (including by David Colquhoun) just what Holford is about.

    David Smith has had an interest in cognitive function, memory, and nutrition, dating back at least to the founding of OPTIMA in the late 1980s and probably before – so I imagine this interest predates Smith encountering Cher Patrique. But… I imagine they must have run across one another fairly early – amongst the people who worked on OPTIMA was Kim Jobst (who was there at the start, 1988, until 1996), and we all know where he ended up. Given Jobst’s interest in Alt.Med even as a junior doctor (which he has written about several times in editorials for J Alt Comp Med), I reckon it is quite conceivable Smith encountered Holford as much as twenty years ago, or even before that.

    Anyway, one can imagine reasons why David Smith might have been minded to collaborate with Holford over the years (few mainstream people interested in nutrition, access to large number of potential subjects interested in taking vitamin formulae and hence data, promoting interest in OPTIMA and their research / ‘getting the word out’ etc etc). But Smith really cannot possibly have been unaware, at least this last decade or so, what a Faustian Bargain he was making dealing with Patrick.

  4. gerry

    Seems your wasting a huge amount of time and energy fussing about a test that has the potential to be useful
    (Subject to more tests) and which advises you to go and see your GP should you obtain a low score.

    It seems to me that the article assumes anyone coming into contact with the claims about the test is unlikely to be capable at looking at it objectively.

    The article far from being an example of object thought seems to have already made its mind up regarding the tests potential for future benefit.

    Just as galling is the assertion that anything that we do for ourselves without the help of a team of suitably qualified NHS practitioners is bound to be either invalid or too scary to contemplate.

    To put my own comments into context.

    I am not a fan of Mr Holford either, the complete opposite in fact.

    When my mother had a fall when she was 78 The NHS hospital to which she was admitted were concerned about her cognitive ability and subjected her to a series of tests telling her they were concerned she had Alzheimer’s Disease. She found this extremely upsetting but hadn’t been given any choice in the matter. I received a call later in the week telling me my mother had been diagnosed as positive and that she had 3 to 4 years to live. I believed this to be arrant nonsense collected her and arranged for her to see someone privately. The outcome was completely negative. I complained and following an inquiry was told that the individuals concerned no longer worked at the hospital. No apology, explanation or promise to amend processes.

    My own work is around organisational performance and I understand the NHS to be both one of the most wasteful organisations in the country both in terms of physical waste and quality of service. My own very personal experiences would show a success rate (in terms of right first time) of less than 10%. No not a scientific study but my own record over only a few hundred transactions. If this is in any way representitive is it any wonder that people are turning to alternative remedies and so called alternative healers?

    I believe that so called men of science who denigrate everything that doesn’t show itself to be at the correct level of statistical significance or that hasn’t undergone extensive trials of a particular kind every bit as divisive as those on the opposite side of the argument and are equally as dismissive of everything scientific.

    My limited understanding of such leads me to BELIEVE the following and I remain open minded to correction:- Articles often seem to imply that trials can make accurate predictions about how a drug will affect an individual rather than it true nature. They fail to mention that different scientists often interpret the same results very differently. They also fail to point out that a trial is not necessarily an accurate predictor of success in the field or that individuals are not the ‘mean’ and hence may react individually.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s