- Open Access
Cognition comes of age: comments on the new FDA draft guidance for early Alzheimer’s disease
Alzheimer's Research & Therapy volume 10, Article number: 61 (2018)
The FDA have recently published draft guidance for the development of treatments for early Alzheimer’s disease. Key features of this guidance are the advocacy of sensitive cognitive measures and a taxonomy of disease severity. Whilst desirable patterns of cognitive-functional improvement are included, specific measures, and the magnitude of required effects, are not described.
We describe key elements of the guidance content, especially with regard targeting key cognitive domains and the means by which they might be efficiently indexed in the disease stages included in the guidance. We discuss also the opportunities to assess cognitive performance in ‘Stage 2’ and ‘Stage 3’ patients, as well as the possibilities for effectively assessing function in the latter category. In this section we review candidate cognitive assessments that we judge are capable of delivering on the guidance specification for sensitive neuropsychological measures. This includes detailed consideration of the ADCS-PACC and Catch-Cog initiatives. With respect to the magnitude of effects, we propose that standardised effect sizes of 0.3 represent a reasonable level of efficacy based on the observation that already marketed drugs on average deliver this level of improvement.
We propose the use of cognitive measures in stage 2 patients to index the cognitive skills known to be compromised early in the Alzheimer’s disease process. We recommend extending the traditional interest in episodic memory to include sensitive, reliable and valid measures of attention, working memory and aspects of executive function. We propose a focus on these additional cognitive abilities based on evidence that performance on tests of these domains is moderately well related to functional skills.
The FDA has recently published a new guidance note for developing drugs to treat early Alzheimer’s disease (AD) . The guidance is as significant for what it doesn’t contain, as much as its actual content. For example, neither the ADAS-cog  nor the CDR-SB  are mentioned. One interpretation of this is that the agency recognises the risk that the mere mention of a test to illustrate a point will be seen as a specific endorsement of that measure. The absence of the CDR-SB will come as no surprise to those of us who attended the CTAD session at which an agency employee indicated that this would be the case. So much for what is absent, but what is actually in the document? The authors include guidance on the use of biomarkers and other aspects of drug development that will not be discussed here. However, cognitive assessment, and a four-stage taxonomy of disease progression are extensively discussed, and it is these topics that will be the focus of our consideration.
General issues raised in the new guidance
The preamble to the guidance includes a helpful reminder that whilst we have tended to dichotomise cognition and function, the two are inextricably linked:
‘FDA rejects this dichotomy and finds such usage inappropriate, because it implies that an effect on cognition itself, regardless of the nature of the observed effect and the manner in which it is assessed, cannot be clinically meaningful. This is certainly not the case’.
A question this raises is how do we define what might be ‘clinically meaningful’? The guidance includes reference to cognitive changes ‘of a particular character, perhaps defined by magnitude or breadth of effect(s), may represent clinically meaningful benefit’. This issue has been a topic of discussion for some time and was considered by a number of us in a recent paper . We will return to the topic of clinical meaningfulness later.
Types of patients – a proposed taxonomy
Turning to the staging taxonomy, Stage 1 patients are in part defined as having no ‘detectable abnormalities on sensitive neuropsychological measures’. However, Stage 2 patients are defined as those who exhibit ‘detectable abnormalities on sensitive neuropsychological measures’ but in whom ‘there is no evidence of functional impairment’. The authors suggest in the context of Stage 2 patients that the ‘FDA will consider strongly justified arguments that a persuasive effect on sensitive measures of neuropsychological performance may provide adequate support for a marketing approval’. The authors suggest that a persuasive effect could be based on ‘A pattern of putatively beneficial effects demonstrated across multiple individual tests’.
Stage 3 patients are in part defined as exhibiting ‘subtle or more apparent detectable abnormalities on sensitive neuropsychological measures, and mild but detectable functional impairment’. The authors describe Stage 3 as ‘Early AD patients approaching the onset of overt dementia’ who are likely to have relatively mild but noticeable impairments in their daily functioning. The authors suggest that as with Stage 2 patients, ‘studies in this stage of disease will generally include sensitive measures of neuropsychological performance’. However, they recognise that this performance may be ‘of uncertain independent clinical meaningfulness’ and thus in their view ‘it is important to demonstrate that a drug favorably affects these functional deficits’.
The assessment of Stage 2 Patients
So much for the guidance review, what might this mean in practice, and how might drug development sponsors respond? In the context of Stage 2 patients, in whom there are detectable cognitive but not functional deficits, we might sensibly employ a cognitive assessment that indexes the relevant domains of function. A number of such assessments have been developed and include computerised assessments, as well as traditional ‘paper-and-pencil’ (P&P) testing. Irrespective of the delivery platform the proposed assessments have tended to converge on the domains of episodic memory, working memory, and aspects of executive function. This focus has a long history and specific reference to these domains was included in the 2008 European Task Force guidance . A very similar approach was adopted by the EPAD Scientific Advisory Group for Clinical and Cognitive Outcomes . Specific examples of cognitive assessments that include measures of these domains are the Alzheimer Disease Cooperative Study Preclinical Alzheimer Cognitive Composite (ADCS-PACC)  and Catch-Cog  initiatives, both of which will be discussed in detail later in this paper. These assessments are composed of P&P measures, some of which have been combined with computerised assessments, as in the case of recently completed studies such as the one reported by Probiodrug , and in ongoing studies being conducted by companies such as Axon Neuroscience . Whilst the content of these assessments has a similar conceptual basis, the specific composition of these assessments varies markedly. However, they converge with respect to exhibiting respectable levels of reliability, validity and sensitivity, and the domains of cognition they index. The guidance offered by Ferris et al.  and Harrison  provides the means for selecting appropriate measures. In the following section, we will describe current projects focused on developing novel and more comprehensive assessments of cognition. In the course of this description we will appraise these measures in the context of best practice guidance.
The ADCS-PACC is one of a number of composite cognitive measures employed in ongoing studies of early or preclinical AD . It is a response to concerns that standard AD measures are likely to prove insensitive to early stage deficits and to cognitive change, whether as a function of disease progression and/or therapeutic intervention. The tests that comprise the ADCS-PACC have been selected to index primarily episodic memory, hence the inclusion of the Free and Cued Selective Reminding Test (FCSRT ) and the Logical Memory IIa sub-test from the Wechsler Memory Scale . The former test yields a score of 0-48 and the latter from 0 to 25. Memory in the form of orientation is measured using the MMSE . The fourth and final ADCS-PACC measure is the Digit Symbol Substitution Test (DSST) from the WAIS-R . This has been selected by the authors as a measure of executive function, though the DSST is popularly regarded as measuring a number of cognitive functions, including both attention and working memory. The performance range for the DSST varies between 0 and 93.
The four components of the ADCS-PACC are all scored on different ranges and so in order to make performance on all four measures directly comparable a composite score of all ADCS-PACC tests is calculated using a z-score normalization method. Each of the four component change scores is divided by the baseline sample standard deviation of that component, to form standardized z-scores. These z-scores are summed to form the composite.
The tests that comprise the ADCS-PACC have a pedigree of use in various cross-sectional and longitudinal studies of patients with AD. The most notable of these is probably the ADNI group of studies. The cognitive focus of the assessment meets the requirement for measures of episodic memory, working memory and elements of executive function. The inclusion of the DSST also allows for praxis and attention to be assessed. The tests selected meet current best practice guidance and promise much as measures of longitudinal cognitive change. The ADCS-PACC has yet to demonstrate sensitivity to pharmacological treatment effects in studies of putative therapies for AD. However, the DSST has previously shown beneficial treatment effects in MCI . A second key issue is whether the rates of cognitive change seen on these measures, when incorporated in assessments featuring multiple other tests, will be replicated when solely the ADCS-PACC measures are administered. This is a key element of validation and a question that must be asked of all new cognitive assessment tools, whether composed of entirely new tests, or measures that have been employed in previously conducted studies.
A further approach is the possibility of employing measures that comprise traditional AD trial metrics, such as the ADAS-cog, NTB, etc. We have considerable experience with the use of these measures and an abundance of data with which to judge their sensitivity and reliability [18, 19]. A further advantage of this approach is that they are familiar to the drug development community, including trial sites, regulators, and other interested third parties. Repurposing well-known tests is therefore a tempting proposition. Catch-Cog is a composite score that blends memory components of the ADAS-cog with executive function measures from the NTB and ADCS-PACC . Amongst the ADAS-cog subtests Word Recall, Word Recognition, and Orientation are substantially less prone to range restrictions than the other ADAS-cog measures (see Table 1) and have significant potential for capturing progressive decline in episodic memory and treatment effects. A point of note is that in contrast to the other ADAS-cog subtests, the Orientation test is naturally, if only partially, parallel in nature, and through the use of parallel word lists the Word Recall and Word Recognition stimuli can be varied across visits.
An acknowledged deficiency of the ADAS-cog is the lack of quality measures of attention, working memory and executive function [20,21,22]. Valid and sensitive measures of these functions have been extensively employed in other composite measures . For example, Catch-Cog includes the COWAT & CFT from the NTB, both of which are well-known and extensively validated tests of working memory and executive function . These measures have also previously been shown to be sensitive to treatment effects, most obviously in the PBT2  and Encenicline  studies. The final cognitive Catch-Cog measure is the Digit Symbol Substitution Test (DSST). As discussed above, this test has a number of virtues, including brevity (about two minute duration) as well as the capacity to measure a variety of cognitive skills, including attention and working memory. It is also acknowledged as a test of ‘timed executive function’ by the CHMP of the EMA . As mentioned, this measure yielded a significant benefit of treatment with galanthamine in MCI  and has been observed to be a sensitive measure of treatment effects in other CNS indications .
The philosophy underlying selection of the Catch-Cog assessment is that robust, reliable, sensitive and valid measures will yield an efficient, evidence-based assessment of cognitive change. The selection of measures that index performance in clinically relevant skills, known to be compromised early in the Alzheimer’s disease process, meets expert group guidance on cognitive assessment. A further dimension of the Catch-Cog research program is that the Amsterdam Instrumental Activities of Daily Living  is being validated alongside the cognitive composite.
The measures selected for inclusion in Catch-Cog have a rich provenance for capturing treatment effects. The NTB executive function tests have previously performed well in studies of the PBT2 , as well as the Encenicline study mentioned above. A remaining but critical question for Catch-Cog is the extent to which it demonstrates robust psychometric characteristics, such as temporal reliability and the assessment’s capacity to show longitudinal change. Both issues are under active investigation. Given that the measures that comprise the Catch-Cog have independently exhibited robust psychometric attributes, acceptable levels of performance are anticipated in the validation study . A summary of the cognitive domains and processes indexed by Catch-Cog, the ADCS-PACC, and ADAS-cog is shown in Table 1.
So, what would the FDA find ‘persuasive’ enough with respect to these measures to grant marketing approval for Stage 2 patient treatment? This is not specified, though the guidance authors suggest that this requires ‘A pattern of putatively beneficial effects demonstrated across multiple individual tests’ and that ‘A large magnitude of effect on sensitive measures of neuropsychological performance may also increase their persuasiveness’. The former requirement is the pattern of performance seen in recent studies of cognitive benefits of new multimodal anti-depressant drugs , captured using cognitive assessments the content of which is very similar to the above examples. The guidance authors do not provide an indication of how substantive a treatment would have to be. Currently marketed drugs yield positive cognitive effect sizes of about 0.3  and we are necessarily left to speculate what the agency would regard as acceptable evidence. Intuitively we suspect that a standalone cognitive effect of a symptomatic relieving treatment might require positive effect sizes of at least 0.3 across a number of domains, with a 0.5 positive effect in a key area, such as episodic or working memory.
Now we turn to consider Stage 3 Patients, in whom the focus is on cognitive-functional impairment. Cognitive assessment is still ‘front and centre’ and the authors specify that ‘Ideally, the outcome measure used in this stage of disease will provide an assessment of meaningful cognitive function’. With respect to function, the authors point out that ‘Many of the assessment tools typically used to measure functional impairment in patients with overt dementia may not be suitable for use in these early stage patients’. They are presumably referring here to traditionally employed ADL assessments such as the DAD  and ADCS-ADL . Consistent with past advice, the guidance authors allow for the use of ‘An integrated scale that adequately and meaningfully assesses both daily function and cognitive effects in early AD patients’ which would be ‘acceptable as a single primary efficacy outcome measure’. The inclination of sponsors with regard the selection of an integrated scale has been to employ the CDR-SB. This new guidance neither endorses nor prohibits the use of the CDR-SB. However, it is perhaps significant that the authors state that:
‘FDA encourages the development of novel approaches to the integrated evaluation of subtle early AD (predementia) functional deficits/impact that arise from early cognitive impairment (e.g., facility with financial transactions, adequacy of social conversation)’.
One supposes that this exhortation would be unnecessary if the CDR-SB was considered to be the sole acceptable solution to the need for a cognitive-functional measure. EMA has previously cautioned  that ‘The CDR-SB scoring requires extensive training and is subject to variability among ethnicity and languages’. It might be that current initiatives, such as Catch-Cog, in which we are seeking to validate the combined use of the more robust cognitive measures used in AD trials with the Amsterdam IADL will provide helpful alternatives to the use of the CDR-SB.
On a final topic, where does this new guidance leave us with respect to use of the ADAS-cog? A generous interpretation of the ADAS-cog’s utility is that it does a not bad job of measuring episodic memory, but that even after the addition of further measures  it is a wholly inadequate assessment of other important cognitive skills, such as attention, working memory, and aspects of executive function, especially in very early stage patients. Whilst it includes measures of language and praxis, they could in no sense be described as ‘sensitive neuropsychological measures’. Previously reported studies of very early stage patients have highlighted that performance on the majority of ADAS-cog subtests is at ceiling  and likely to stay there for the typical duration of most clinical trials featuring mild stage patients . Abandoning the ADAS-cog for superior measures is long overdue and the quality of our clinical science will likely be enhanced by making the shift to more reliable, sensitive and valid measures.
This new guidance offers welcome and helpful details of the FDA’s thinking with respect to the relationship between function and cognition, as well as the need for sensitive neuropsychological assessment. There is also helpful acknowledgement that the tools needed to determine efficacy will vary as a function of disease severity. Whilst still a draft document, it is encouraging to see the agency offering thoughtful guidance. This guidance appears to endorse the view that judicious cognitive domain targeting, wed to the selection of sensitive neuropsychological and cognitive measures, has the potential to demonstrate treatment efficacy in the very earliest detectable stages of AD.
Alzheimer’s disease Assessment Scale – cognitive subscale
- ADCS ADL:
Alzheimer’s disease Cooperative Study Activities of Daily Living
- ADCS PACC:
Alzheimer’s disease Cooperative Study Pre-Alzheimer’s Cognitive Composite
Activities of Daily Living
Amsterdam Instrumental Activities of Daily Living
Clinical Dementia Rating scale – sum of boxes
Category Fluency Test
Committee for Medicinal Products for Human Use
Controlled Oral Word Association Test
Clinical Trials in Alzheimer’s Disease
Disability Assessment for Dementia
Digit Symbol Substitution Test
European Medicines Agency
Free and Cued Selective Reminding Test
Food & Drug Administration
Mild Cognitive Impairment
Mini-Mental State Examination
Neuropsychological Test Battery
Wechsler Adult Intelligence Scale (Revised)
Wechsler Memory Scale
FDA. Early Alzheimer’s Disease: Developing Drugs for Treatment Guidance for Industry. 2018. https://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM596728.pdf. Accessed 26 Feb 2018.
Rosen WG, Mohs RC, Davis KL. A new rating scale for Alzheimer’s disease. Am J Psychiat. 1984;141(11):1356–64.
Hughes CP, Berg L, Danziger WL, Coben LA, Martin RL. A new clinical scale for the staging of dementia. Br J Psychiatry. 1982;140:566–72.
Cummings J, Scheltens P, McKeith I, Blesa R, Harrison J, Bertolucci PH, et al. Effect size analyses of Souvenaid in patients with Alzheimer’s disease. Alz Res Therapy. 2017;55(3):1131–9.
Vellas B, Andrieu S, Sampaio C, Coley N, Wilcock G. Endpoints for trials in Alzheimer's disease: a European task force consensus. Lancet Neurol. 2008;7(5):436–50.
Ritchie K, Ropacki M, Albala B, Harrison JE, Kay J, Kramer J, et al. Recommended cognitive outcomes in pre-clinical Alzheimer's disease: consensus statement from the European Prevention of Alzheimer's Dementia (EPAD) Project. Alz Dement. 2016;13(2):186–95.
Donohue MC, Sperling RA, Salmon DP, Rentz DM, Raman R, et al. The preclinical Alzheimer cognitive composite: measuring amyloid-related decline. JAMA Neurol. 2014;71(8):961–70.
Jutten RJ, Harrison JE, de Jong F, Aleman A, Ritchie CW, Scheltens P, et al. A composite measure of cognitive and functional progression in Alzheimer's disease: Design of the Capturing Changes in Cognition study. Alz Dement: Trans Res Clin Intervent. 2017;3(1):130–8.
https://clinicaltrials.gov/ct2/show/NCT02389413, Accessed 26 Feb 2018.
https://clinicaltrials.gov/ct2/show/NCT02579252, Accessed 26 Feb 2018.
Ferris SH, Lucca U, Mohs R, Dubois B, Wesnes K, Erzigkeit H, et al. Objective psychometric tests in clinical trials of dementia drugs. Position paper from the International Working Group on Harmonization of Dementia Drug Guidelines. Alz Dis Assoc Dis. 1997;11(S3):34–8.
Harrison JE. Measuring the mind: Detecting cognitive deficits and measuring cognitive change in patients with depression. In: McIntyre RS, Cha D, editors. Cognitive Impairment in Major Depressive Disorder. Cambridge: Cambridge University Press; 2017. p. 229–41.
Grober E, Hall CB, Lipton RB, Zonderman AB, Resnick SM, Kawas C. Memory impairment, executive dysfunction, and intellectual decline in preclinical Alzheimer’s disease. J Int Neuropsychol Soc. 2008;14(2):266–78.
Wechsler D. WMS-R: Wechsler Memory Scale–Revised: Manual. San Antonio: Psychological Corporation; 1987.
Folstein MF, Folstein SE, McHugh PR. “Mini-Mental State”: a practical method for grading the cognitive state of patients for the clinician. J Psychiat Res. 1975;12:189–98.
Wechsler D. Wechsler Adult Intelligence Scale–Revised. San Antonio: Psychological Corporation; 1981.
Winblad B, Gauthier S, Scinto L, Feldman H, Wilcock GK, Truyen L, et al. Safety and efficacy of galantamine in subjects with mild cognitive impairment. Neurol. 2008;70(22):2024–35.
Harrison J, Caveney A. 10 years of the Neuropsychological Test Battery (NTB). PRO Newsletter. 2011;46:21–4.
Harrison JE, Rentz DM, McLaughlin T, Niecko T, Gregg KM, Black RS, et al. Cognition in MCI & Alzheimer’s disease: baseline data from a longitudinal study of the NTB. Clin Neuropsychol. 2016;28(2):252–68.
Harrison JE. Measuring Cognitive Change in Alzheimer’s disease Clinical Drug Trials. J Nutrit Health Aging. 2007;11(4):327–9.
Wesnes KA, Harrison JE. The evaluation of cognitive function in the dementias: methodological and regulatory considerations. Dial Clin Neurosci. 2003;5(1):77–88.
Hobart J, Cano S, Posner H, Selnes O, Stern Y, Thomas R, et al. Putting the Alzheimer's cognitive test to the test I: traditional psychometric methods. Alz Dement. 2013;9(1):S4–9.
Harrison J, Minassian SL, Jenkins L, Black RS, Koller M, Grundman M. A neuropsychological test battery for use in Alzheimer disease clinical trials. Arch Neurol. 2007;64:1323–9.
Lezak MD. Neuropsychological Assessment. 3rd ed. Oxford: Oxford University Press; 2007.
Lannfelt L, Blennow K, Zetterberg H, Batsman S, Ames D, Harrison JE, et al. Targeting Aβ as a modifying therapy of Alzheimer’s disease: safety, efficacy and biomarker findings of a Phase lla Randomised, Double-Blind Placebo-Controlled trial of PBT2. Lancet Neurol. 2008;7(9):779–86.
Hilt D, Gawryl M, Koenig G, EVP-6124 Study Group. EVP-6124: Safety, tolerability and cognitive effects of a novel A7 nicotinic receptor agonist in Alzheimer’s disease patients on stable Donepezil or Rivastigmine therapy. Alz Dement. 2009;5(4):P4–348.
CHMP (2014) Discussion paper on the clinical investigation of medicines for the treatment of Alzheimer’s disease and other dementias. EMA/CHMP/539931/2014.
McIntyre RS, Lophaven S, Olsen CK. A randomized, double-blind, placebo-controlled study of vortioxetine on cognitive function in depressed adults. Int J Neuropsychopharmacol. 2014;30:1–11.
Sikkes SA, Knol DL, Pijnenburg YA, de Lange-de Klerk ES, Uitdehaag BM, Scheltens P. Validation of the Amsterdam IADL Questionnaire©, a new tool to measure instrumental activities of daily living in dementia. Neuroepidemiology. 2013;41(1):35–41.
Faux NG, Ritchie CW, Gunn A, Rembach A, Tsatsanis A, Harrison J, et al. PBT2 rapidly improves cognition in Alzheimer's Disease: additional phase II analyses. J Alz Dis. 2010;20:509–16.
Jutten R, Harrison JE, Lee Meeuw Kjoe PR, Opmeer EM, NSM S, de Jong FJ, et al. A novel cognitive-functional composite measure to detect changes in early Alzheimer's disease: Test-retest reliability and feasibility. Alz Demen: Diag Assess Dis Monit. 2018;10:153–60.
Harrison JE, Lophaven S, Olsen CK. Which cognitive domains are improved by treatment with vortioxetine? Int J Neuropsychopharmacol. 2016;19(10):pyw054
Rockwood K, Black SE, Robillard A, Lussier L. Potential treatment effects of donepezil not detected in Alzheimer's disease clinical trials: a physician survey. Intl J Geriat Psychiat. 2004;19(10):854–60.
Gélinas I, Gauthier L, McIntyre MC, Gauthier S. Development of a functional measure for persons with Alzheimer disease: The Disability Assessment for Dementia. American J Occ Ther. 1999;53:471–81.
Gauthier S, Bodick N, Erzigkeit E, Feldman H, Geldmacher JH, Mohs R, et al. Activities of daily living as an outcome measure in clinical trials of dementia drugs. Position paper from the International Working Group on Harmonization of Dementia Drug Guidelines. Alz Dis Assoc Dis. 1997;11(Suppl 1):6–7.
Mohs RC, Knopman D, Petersen RC, Ferris SH, Ernesto C, Grundman M, et al. Development of cognitive instruments for use in clinical trials of antidementia drugs: additions to the Alzheimer's Disease Assessment Scale that broaden its scope. The Alzheimer's Disease Cooperative Study. Alz Dis Assoc Dis. 1997;11(Suppl 2):13–21.
Sevigny JJ, Peng Y, Liu L, Lines CR. Item analysis of ADAS-Cog: effect of baseline cognitive impairment in a clinical AD trial. Am J Alz Dis Dement. 2010;25:119–24.
JH is solely responsible for the content of this manuscript. The author read and approved the final manuscript.
Ethics approval and consent to participate
Consent for publication
In the past two years the author has received consultancy fees and honoraria from the following organisations: 23andMe, AbbVie, Access to Quality, Amgen, Anavex, Astellas Pharma, AstraZeneca, Avonex, Avraham, Axon, Axovant, Biogen Idec, Boehringer Ingelheim, Bracket, Cambridge Brain Sciences, Catenion, Cognitive Therapeutics, Compass Pathways Ltd., CRF Health, DeNDRoN, Enzymotec, ePharmaSolutions, Eisai, Eli Lilly, Forum Pharma, Fresh Forward, G4X Discovery, GfHEu, Goetzpartners, Heptares, Janssen AI, Johnson & Johnson, Kaasa Health, Kyowa Hakko Kirin, LivaNova PLC, Lundbeck, MedAvante, Merck, MyCognition, Mind Agilis, Neurocog, Neurim, Neurotrack, Novartis, Nutricia, Orion Pharma, PharmaNet/i3, Pfizer, Prana Biotech, PriceSpective, Probiodrug, Prophase, ProStrakan, Regeneron, Reviva, Roche, Sanofi, Servier, Shire, Takeda, TCG and TransTech Pharma.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.