Importance Presently, the clinical standard for reporting the results of an amyloid positron emission tomography scan is to assign a dichotomous rating of positive or negative for the presence of amyloid. In a 4-year longitudinal study, we investigated whether using a continuous measure of the magnitude of baseline amyloid burden would provide valuable information about the rate of future cognitive decline over the subsequent 4 years compared with a dichotomous measure in middle-aged and older adults. Objective To examine whether a continuous, dose-response relationship between amyloid burden and cognitive decline was present among middle-aged and older adults. Design, Setting, and Participants This cohort study included 174 participants from the Dallas Lifespan Brain Study who were 40 to 89 years old at the beginning of the study, were cognitively normal at baseline (a Mini-Mental State Examination score of 26 or higher) with no history of neurological or psychiatric disorders, and had completed amyloid imaging ([ 18 F]-florbetapir) at baseline and cognitive assessments at baseline and a 4-year follow-up. Continuous amyloid burden was measured as the mean cortical standardized uptake value ratio (SUVR) at baseline. Main Outcomes and Measures Linear mixed models assessed the effect of increasing baseline amyloid over time (SUVR × time interaction) on episodic memory, reasoning, processing speed, vocabulary, and Mini-Mental State Examination performance. Age, sex, education, apolipoprotein e4, and the random effect of intercepts were included as covariates. Results The mean (SD) age for all participants (n = 174) was 66.44 (11.74) years, and 65 participants (37%) were men. The primary analyses yielded significant SUVR × time interactions in episodic memory, processing speed, vocabulary, and Mini-Mental State Examination performance, but not in reasoning performance. Higher baseline SUVR projected greater cognitive decline over 4 years. When controlling for variance related to a dichotomized positive/negative classification, most effects on cognition remained. Dichotomized amyloid status alone yielded fewer significant effects of amyloid on cognitive decline than continuous SUVR. Among amyloid-positive participants, increasing baseline SUVR predicted an increasing decline in episodic memory, but other effects on cognition were more limited. Finally, higher baseline amyloid burden among middle-aged adults was related to changes in vocabulary, with the effect driven by 3 apolipoprotein e4 homozygotes. Conclusions and Relevance These results suggest that the magnitude of amyloid burden at baseline is associated with the rate of cognitive decline over 4 years and potentially provides important information about the rate of future cognitive decline that is not available from a dichotomous positive/negative categorization.