The People’s Medical Journal (aka Daily Mail) has a touching faith in the value of early diagnosis and screening tests. It would be rather charming to note that their history of being wrong has not as yet reduced them to cynicism if it were not for the errors and false hope that they present as verified fact to their readers.
Food for the Brain (CEO Patrick Holford) claims to offer an online test to detect the early symptoms of Alzheimer’s Disease. Alzheimer’s Disease is a subject that excites great concern and, by definition, those who research in this area are aware that they are typically dealing with some vulnerable people.
Dr Margaret McCartney recently examined the claims for Food for the Brain and self administered cognitive tests after some enthusiastic media reports and found them unwarranted, and not in line with available evidence. She evaluated the claims again in the BMJ, An early warning for Alzheimer’s disease, and questioned FFTB about the claims made for the test in the diagnosis of Alzheimer’s Disease or its prodromes and the evidence base for its recommendations.
Patrick Holford, who describes himself as a “nutritionist” and chief executive officer of Food for the Brain, told me, “We, the charity, deemed the evidence to have become substantial enough to warrant the launch of our Alzheimer’s prevention project . . . the primary aim of which is to encourage early screening of cognitive function from age 50, followed by homocysteine testing.” Food for the Brain’s adviser, the pharmacologist David Smith, told me that the online test is “not a diagnostic test, and there is no definitive outcome; it simply tells the user about their cognitive status.
So, media coverage (eg, 15-minute online test for dementia: DIY memory quiz detects early signs of Alzheimer’s in people as young as 50; Online test for early signs of Alzheimer’s) persuaded people to take an online Cognitive Function Test (CFT) developed by an “Oxford research team” as a way of detecting the early symptoms of Alzheimer’s Disease and as a way of persuading them to take/pay for non-evidence-based tests and supplements. However, Professor David Smith, Chairman of FFTB’s Scientific Advisory Board, admits that the test has no diagnostic value. (Holfordwatch readers with a good memory will recall that David Smith has previously admitted that FFTB has not done a “proper job” of research it attempted.)
As is too frequently the case when discussing Patrick Holford and Food for the Brain, it is difficult to outline all of the misunderstandings and errors that accompany their claims. We can’t begin to cover all of the issues which, inevitably, also involve: the inappropriate promotion of the crystal-ball of homocysteine testing as a biomarker; the advice that test takers should ask their GPs for a test that is not available for that purpose on the NHS or have recourse to Yorktest private testing; the promotion of supplements. This is not the time to explore the ethical concerns that must accompany the availability of a direct to consumer test that purports to diagnose such a widely-feared condition and has already caused some distress while also illustrating a worrying (and perhaps unwarranted) confidence in the significance/value of the test.[1]
The following is not a complete account as the story is still unfolding, however, even these items highlight the contested nature of this test and why the marketing/media coverage of it is inappropriate. The inconsistencies and recent redactions must also question whether the test ought still be available until such time as various issues are clarified. It isn’t clear why Oxford is failing to protect its reputation as it can not be to its advantage to be associated with such a questionable test and set of recommendations.
i) Mid-May various media outlets covered FFTB’s Cognitive Function Test (CFT): they reported that it had been developed by an “Oxford research team” and was made available online, direct to consumers, and promoted as diagnostic of Alzheimer’s Disease and its prodromes.
Food for the Brain (FFTB) emphasised the involvement of Oxford University in news stories and the then current version of its own website: Patrick Holford has likewise stressed the involvement of Oxford Uni. in his marketing materials for his own website and for FFTB for which he is the CEO. Oxford researchers are said to have played the role of lead developer.[2]
However, approaches to several people, including Virginia’s Professor Timothy Salthouse and Oxford’s Dr Celeste de Jager, subsequently revealed that neither of them played such a substantial role in the development of the test.
Professor Salthouse reports that he granted permission to the authors to use an adapted version of his perceptual comparison tests. However, he emphasises that that is the extent of his involvement and makes no claims with respect to the role of his tests in assessing the risk of memory decline or the development of Alzheimer’s Disease.
Dr de Jager’s involvement was recently clarified by an amendment to the FFTB website:
The CFT composes three elements:
A Episodic memory, using cued recall and paired associate learning test constructs, developed by Catharine Trustram Eve for FFB, with the advice of Dr Celeste de Jager.
So, it seems as if the test, developed by an “Oxford research team”, was substantially the work of Catharine Trustram Eve who is listed as an “Independent Market Research Professional” albeit that is not made clear in the Letter to GPs that test-takers are advised to give their doctors. Catharine Trustram Eve’s profile does not list any qualifications in neurology, psychology, cognitive science or similar relevant disciplines.
How many test-takers took the test because they were reassured by the much publicised Oxford provenance of the CFT? What, if anything, is Oxford doing to dissociate itself from the CFT? Even today, the Daily Mail is linking Oxford and this test: Test to detect early onset of Alzheimer’s for all over 65s to be introduced within two years.
One of the items in the box insert states:
“Oxford University has devised a memory test that can be taken at home in 15 minutes and can spot the signs of Alzheimer’s in people as young as 50.”
Both Patrick Holford and Food for the Brain promoted the CFT to their mailing lists. Why have neither Patrick Holford nor Food for the Brain issued corrections to their mailing lists to clarify the provenance of the test?
ii) The CFT is said to be validated for a specific age range and it is promoted as diagnostic in news coverage and on the FFTB website.[2], [3] However, as Dr McCartney explains, at the time of the news items, the validation for this test was not available and there were no data relating to sensitivity or specificity (false positives and false negatives).
The Daily Mail story (inter alia) refers to pilot studies:
The researchers cannot put a figure on the test’s accuracy, but in pilot studies it worked as well as tests already used in GP surgeries and specialist memory clinics.
In the absence of detail that would allow for appropriate scrutiny, the claims for validation were premature at best. Irritatingly, in late May the FFTB site was amended to state:
“based on the pilot, it appears that the CFT is sensitive to MCI. A full description of the pilot and analysis will be available from this page by 1st July 2011.”
http://www.foodforthebrain.org/content.asp?id_Content=1825
However, following the latest updates to the FFTB website, we now learn that we are not to be permitted to see the detail that ‘validates’ these tests for some time:
“Previously it was stated that ‘A full description of the pilot and analysis will be available from this page by 1st July 2011.’ However, in light of an expert academic critique of the CFT validation, we have decided to submit the work for publication prior to publicising the results on the website.”
http://www.foodforthebrain.org/content.asp?id_Content=1825
Prof. Salthouse’s contribution to the test is well validated within its usual sphere of use. Prof. Salthouse makes no claims for the integration of his test within the CFT and states that his comparison tests should not be treated as valid predictors of the risk of memory decline or the development of Alzheimer’s Disease. Under the circumstances, it is arguably imperative that the pilot studies and materials that underpin the “validation” of the CFT should be made available. However, the absence of accessible validation is not made clear to the public, nor, perhaps, to some NHS Commissioners who are given funding proposals for which there has not been adequate due diligence.
iii) Media coverage, Patrick Holford and the FFTB promoted the CFT as diagnostic of Alzheimer’s Disease or its prodrome when the CFT has not yet been publicly validated as an appropriate instrument to identify or quantify mild cognitive impairment.
David Smith modified those claims when challenged by Margaret McCartney. FFTB updated its website July 1 and, in line with Smith, has considerably modified its claims for the scope of the CFT:
“Does the CFT diagnose dementia, Alzheimer’s disease or Mild Cognitive Impairment?
No. The CFT is not a diagnostic test, but a test designed to inform/educate the user about their cognitive function. If the result is below a threshold we suggest that they visit their GP who can perform whatever diagnostic tests are required at their discretion.
However, both Patrick Holford and Food for the Brain promoted the CFT to their mailing lists: the former was headed, “15 minute free test to prevent dementia” and the latter “A 15 minute free test could stop you ever getting Alzheimer’s”. Why have neither Patrick Holford nor Food for the Brain issued corrections to their mailing lists to update them as to the reduced scope of claims for the test?
Allegedly, many people have taken this test. Perhaps the test-takers were reassured that it was developed at Oxford University, and that it is a validated test for Alzheimer’s and its prodromes. Patrick Holford claimed (in a later deleted blog post) that 55,000 people took the CFT in 10 days. Since then, FFTB has claimed that more than 70,000 people have taken this test.
A test that plainly was not developed by an “Oxford research team”. A test that is now said not to be a test for what people were told that it was but now “simply tells the user about their cognitive status” albeit the validation for that is not available.
Why hasn’t Oxford contacted media outlets such as Daily Mail and Telegraph to instruct them to correct their stories if Oxford’s involvement is as limited as the current version of the FFTB website implies? Were the pilot studies on which the ‘validation’ rests conducted at Oxford, and, if so, did Oxford oversee their clinical governance?
Why haven’t Patrick Holford or Food for the Brain alerted media outlets that they have modified their claims concerning: the provenance of the test; the scope of the claims for the CFT’s diagnostic purpose; and that the claims for validation ought to be held in abeyance until such time as the details are published?
Why haven’t Patrick Holford or Food for the Brain alerted their mailing lists as to these substantial revisions concerning the CFT? Has there been any attempt to contact GPs who’ve received one of these letters from a patient to inform them of the modified status of these claims? (The GP letter is still available on the FFTB site and still contains claims that are out of date.) If not, why not?
The CFT has created anxiety amongst some users and has given false confidence to others. Rather curiously, FFTB brags of the thank you letters it has received although it now seems as if they were being thanked for reassurances that can no longer carry any weight given the modifications and reduced scope of the claims for that test.
A curious test and a curious business. There will be more to come when more information is available about the ethical approval for this test and other pertinent matters.
Notes
[1] Sample quoted from
http://www.womanandhome.com/forums/showflat.php/Cat/0/Number/726038/Main/725925/
[Northwindrider] Mine said I was at low risk of developing Altzeimers which is quite comforting as I have a Grandmother with it and know that my Mum was in the early stages when she died.
[snowy47] I have completed the test and i have a very low chance of developing Altzeimers, my D passed last year and he put my M through hell with it.
[susieblue] I have just done the test and told I was low risk. But I am pretty sure my mother would have been told the same had she done it at my age too. Read an article about it on Yahoo. Complete rubbish! For starters my mother, aunt and uncle all had/have it.
[aec13cat] Was so curious in the end I took it but it shows I’m at risk -totally depressed now and wish I hadn’t taken it
Following from
http://forum.alzheimers.org.uk/showthread.php?33976-Someone-look-at-this-for-me-please
[Danny] I took the online test myself,I scored 37/110. It was a bit worrying to be told I could be at serious risk of developing Alzheimers. It has worried me to bits.It will teach me to stop researching so much.
[Tony] I just took that test and scored 29 yesterday
had my 6 monthly test for memory clinic scored 30/30 now I’m confused with the results was the memory clinic test 2 easy
[Gill66] I am suitably happy with a score of 88. Use a mouse, it’s a lot easier. With a history of dementia in the family i feel quite content this evening
Quoted from Patrick Holford’s blog:
“The positive response to my test results came as a great relief, as my father, uncles and paternal grandfather all developed symptoms of senile dementia of one form or another, when they were precisely the age I am now. I feel a great weight has been lifted from my shoulders, at least for the foreseeable future.” Yours, Hugh G. “A really informative website – and the opportunity to put my mind at rest by doing the cognitive test was priceless. I found the test itself very well introduced and explained. The examples are particularly helpful. I’m sure I’m not the only one who approached the test with some anxiety but I found I was far less panicked than I expected.” said Marion. Ivor, age 75, said “The nightmare of Alzheimers has been put to sleep by the results of this test. Thank you.”
http://www.patrickholford.com/index.php/blog/blogarticle/951/ – if no longer available, please see
http://www.freezepage.com/1307103965TWDDWXGIGE
It is worth noting that Ivor is outside the age range for the test yet Patrick Holford nonetheless includes this testimonial.
[2] “The Cognitive Function Test assesses three critical areas of cognitive processing associated with cognitive decline leading to Alzheimer’s disease…This test has been developed in collaboration with Dr Celeste de Jager from the University of Oxford, Professor Timothy Salthouse from the University of Virginia and Catharine Trustram Eve.”
Original link for text: http://www.foodforthebrain.org/content.asp?id_Content=1824
Freezepage for the page May 20 2011: http://www.freezepage.com/1305892729EKYIERRJCR
A similar claim is made in the results letter than test-takers are advised to give their GPs:
“Your patient has completed the Cognitive Function Test at http://www.foodforthebrain.org, an educational trust whose mission is to promote the link between mental health and nutrition. This is a validated screening test for those aged 50 and above, designed to detect early cognitive impairment. This test has been developed with Professor Timothy Salthouse and Dr Celeste de Jager, specialists in assessment of cognitive function.”
Original link for text: http://cft.foodforthebrain.org/doctors-letter-r.aspx?name=Patrick%20Holford&dob=4/5/1953
Freezepage for the page May 20 2011:
http://www.freezepage.com/1305892790UAAUYMCITZ
Daily Mail and other accounts credit “Dr Celeste de Jager, [as] the main developer of the Cognitive Function Test”.
Patrick Holford blog post 15 Minute Online Test for Dementia:
“The test, available from http://www.foodforthebrain.org, also tells you how to delay memory decline and possibly reduce Alzheimer’s risk, based on research of people with mild cognitive impairment, the stage before Alzheimer’s, by Oxford University experts Professor David Smith and Dr Celeste de Jager.”
Freezepage for the page 20 May 2011: http://www.freezepage.com/1305892842FQLAHXZRQC
[3] “The Cognitive Function Test assesses three critical areas of cognitive processing associated with cognitive decline leading to Alzheimer’s disease…The test has been validated for the age range of 50 plus. ”
Original link for text: http://www.foodforthebrain.org/content.asp?id_Content=1824
Freezepage for the page May 20 2011: http://www.freezepage.com/1305892729EKYIERRJCR
The Daily Mail account of the CFT (in common with the Telegraph and other news outlets) explicitly claims that the test can detect early signs of Alzheimer’s which implies that the test is diagnostic.
“An early warning test for Alzheimer’s that can be taken online in 15 minutes has been developed by British scientists.
It can spot signs of the debilitating brain disease in people as young as 50.
The computer-based interactive quiz provides an instant result and could help delay or prevent the condition by advising simple diet and lifestyle changes.”…
“But most are still in the early stages of development and none, other than the new Cognitive Function Test, which has been devised by Oxford University scientists, can be taken online in the comfort of a person’s own home.
This is likely to make it popular with those who fear their memory is failing but are too embarrassed to discuss their worries with their doctor.
…
[B]ecause apparently healthy people have no way of telling if they are among those who could benefit from the vitamin B memory boost, the Oxford research team created the test.
It measures mild cognitive impairment – or the slight memory lapses that can be a precursor to Alzheimer’s – which affects one in six aged 70-plus, or 1.5million Britons. Half will develop dementia within five years of diagnosis.”
Holford Watch: Patrick Holford, nutritionism and bad science 