Cardiovascular risk factors in midlife have been linked to late life risk for Alzheimer’s disease and related dementias (ADRD). The relation of vascular risk factors on cognitive decline within midlife has been less studied. Using data from the Study of Women’s Health Across the Nation, we examined associations of midlife hypertension, elevated lipid levels, diabetes, fasting glucose, central adiposity, and Framingham heart age with rates of cognitive decline in women who completed multiple cognitive assessments of processing speed, and working and verbal memory during midlife. Diabetes, elevated fasting glucose, central obesity, and heart age greater than chronological age were associated with rate of decline in processing speed during midlife. Vascular risk factors were not related to rate of decline in working or verbal memory. Midlife may be a critical period for intervening on cardiovascular risk factors to prevent or delay later life cognitive impairment and ADRD.