Increasing evidence supports an association between midlife cardiovascular risk factors (CVRFs) and risk of dementia, but less is known about whether CVRFs influence cognition in midlife. We examined the relationship between CVRFs and midlife cognitive decline. In 2,675 black and white middle-aged adults (mean age 50.2 ± 3.6 years, 57% female, 45% black), we measured CVRFs at baseline: hypertension (31%), diabetes mellitus (11%), obesity (43%), high cholesterol (9%), and current cigarette smoking (15%). We administered cognitive tests of memory, executive function, and processing speed at baseline and 5 years later. Using logistic regression, we estimated the association of CVRFs with accelerated cognitive decline (race-specific decline ≥1.5 SD from the mean change) on a composite cognitive score. Five percent (n = 143) of participants had accelerated cognitive decline over 5 years. Smoking, hypertension, and diabetes mellitus were associated with an increased likelihood of accelerated decline after multivariable adjustment (adjusted odds ratio [AOR] 1.65, 95% confidence interval [CI] 1.00-2.71; AOR 1.87, 95% CI 1.26-2.75; AOR 2.45, 95% CI 1.54-3.88, respectively), while obesity and high cholesterol were not associated with risk of decline. These results were similar when stratified by race. The likelihood of accelerated decline also increased with greater number of CVRFs (1-2 CVRFs: AOR 1.77, 95% CI 1.02-3.05; ≥3 CVRFs: AOR 2.94, 95% CI 1.64-5.28) and with Framingham Coronary Heart Disease Risk Score ≥10 (AOR 2.29, 95% CI 1.21-4.34). Midlife CVRFs, especially hypertension, diabetes mellitus, and smoking, are common and associated with accelerated cognitive decline at midlife. These results identify potential modifiable targets to prevent midlife cognitive decline and highlight the need for a life course approach to cognitive function and aging.