What is the "accuracy" of the new Alzheimer's Blood test?

Last week an important paper was published that claimed researchers had developed a diagnostic test for predicting Alzheimer's Disease (within 3 years) with "90% accuracy". Does a positive result on this diagnostic mean that the patient has a 90% chance of developing Alzheimer's? The answer is No and illustrates a subtle point about the definition of predictive accuracy. This clarification was raised by David Colquhoun in a blog post.

First some background. Alzheimer's Disease (AD) is a type of dementia that progresses from preclinical symptoms (e.g. memory lapses) to mild cognitive impairment (MCI, e.g. noticeable changes in memory and thinking) to full-blown AD (e.g. confusion, trouble with language, long-term memory loss). Onset typically starts after age 65. In a post last week, I described a new report suggesting that the number of deaths attributed to Alzheimer's Disease may be vastly underestimated. Finally, there is no cure or effective treatment for Alzheimer’s Disease.

Diagnosis of Alzheimer's Disease is complex because of overlapping symptoms with other dementias and mental disorders. Memory and cognitive functions in the patient provide one important indicator. A more rigorous test involves using MRI to image the brain which can show shrinkage in certain regions due to AD. A more recent development is testing for biomarkers (e.g. amyloid protein) in the cerebrospinal fluid, which unfortunately is hard (and painful) to access via spinal tap. Finally, there are more experimental approaches that are being evaluated such as the "peanut butter smell test" described in a previous post. Although there is no cure for Alzheimer's Disease, earlier diagnosis can rule out other disorders and allow for a more focused treatment of the symptoms.

In a new study, researchers at Georgetown University devised a blood test for AD that measures 10 lipids in the blood using mass spectrometry. They first examined patients with AD and identified lipids whose concentrations were different from people without AD. They then used this lipid panel to predict whether a group of people 70 years and older without AD (i.e. normal or preclinical) would later develop MCI or AD. They found that their lipid blood test could predict the onset of Alzheimer's Disease within 3 years with an accuracy of 90%.

In other words, 90% of those with eventually developed AD tested positive on the test, and 90% of those who did not convert to MCI or AD within 3 years were correctly diagnosed as negative. Thus, the true positive rate (TPR or sensitivity) was 90%, and the true negative rate was also 90% (TNR or specificity). The chance of being incorrectly diagnosed as normal but eventually developing AD (false negative rate) was 10% (1 - TPR), and the chance of being incorrectly diagnosed as AD but not developing the disease (false positive rate) was 10% (1-TNR). The overall accuracy is the total number of people being correctly diagnosed which in this study was 90%, i.e. (0.90 * fraction of patients who were affected by AD) + (0.9 * fraction of patients who remained unaffected).

This is an unprecedented result for any AD diagnostic let alone a blood test which is less invasive and easier to perform than other methods. However let's re-ask the questioon from above, "Does a positive result on this diagnostic mean that the patient has a 90% chance of developing Alzheimer's?" The answer is no, which can be revealed by a simple example.

Let's assume that 10% of the population over 70 will be affected by AD in 3 years (the actual number is probably closer to 5%). Then in a population of 100 people that means that 10 people will get AD, and of those, 90% (9 people) will have a correct positive result on the lipid blood test; one affected person will be incorrectly diagnosed as normal (false negative). Among the 90 people who won't get AD, 90% (81 people) will be correctly diagnosed as non-AD; there will be 9 people who will be incorrectly diagnosed as AD (false positives).

Thus, the total number of positive test results from the 100 people will be 9 + 9 = 18. So if you receive a positive result, the chance that you will convert to AD is 9/18 or 50% (precision), and not 90%. Overall however, 90 of the 100 people (90%) will receive the correct positive or negative test result, giving rise to the 90% accuracy figure.

This distinction between accuracy and precision is worth point out. That being said, a potentially highly accurate blood test for Alzheimer's disease is very exciting. Improving Alzheimer's Disease diagnosis is an important medical issue especially if treatments start becoming available in the future.

Finally in Figure 1 below (for those interested in prediction statistics) I gratuitously reproduce a ROC (receiver operating characteristic) curve which "illustrates the performance of a binary classifier system as its discrimination threshold is varied." In other words the blood lipid test classifies between AD or non-AD (within 3 years), and the values of blood lipids are used to make the binary discrimination. ROC curves are good for depicting the tradeoff between capturing the true positives while avoiding the false positives. The straight line (worthless) indicates essentially random guessing in which the TPR = FPR (true positive rate equals false positive rate), and so the classifier offers no diagnostic information. The yellow line indicates a more accurate classifier in which TPR is greater than FPR.

Figure 1. ROC graph shows Excellent, Good, and Worthless prediction curves. A 90% accurate diagnostic (0.9 true positive rate and 0.1 false positive rate) would lie on the yellow "excellent" line.