The association between hypertension developed before midlife and late-life brain health is understudied and, because of the cardioprotective benefits of estrogen before menopause, may differ by sex. To assess the association of early adulthood hypertension and blood pressure (BP) change with late-life neuroimaging biomarkers and examine potential sex differences. This cohort study used data from the Study of Healthy Aging in African Americans (STAR) and Kaiser Healthy Aging and Diverse Life Experiences (KHANDLE) study, which were harmonized longitudinal cohorts of racially and ethnically diverse adults aged 50 years and older from the San Francisco Bay area and Sacramento Valley in California. The STAR was conducted from November 6, 2017, to November 5, 2021, and the KHANDLE study was conducted from April 27, 2017, to June 15, 2021. The current study included 427 participants from the KHANDLE and STAR studies who received health assessments between June 1, 1964, and March 31, 1985. Regional brain volumes and white matter (WM) integrity were measured via magnetic resonance imaging between June 1, 2017, and March 1, 2022. Hypertension status (normotension, transition to hypertension, and hypertension) and BP change (last measure minus first measure) were assessed at 2 multiphasic health checkups (MHCs; 1964-1985) in early adulthood (ages 30-40 years). Regional brain volumes and WM integrity were measured using 3T magnetic resonance imaging and z standardized. General linear models adjusted for potential confounders (demographic characteristics and study [KHANDLE or STAR]) were used to assess the association of hypertension and BP change with neuroimaging biomarkers. Sex interactions were tested. Among 427 participants, median (SD) ages were 28.9 (7.3) years at the first MHC, 40.3 (9.4) years at the last MHC, and 74.8 (8.0) years at neuroimaging. A total of 263 participants (61.6%) were female and 231 (54.1%) were Black. Overall, 191 participants (44.7%) had normotension, 68 (15.9%) transitioned to hypertension, and 168 (39.3%) had hypertension. Compared with participants who had normotension, those who had hypertension and those who transitioned to hypertension had smaller cerebral volumes (hypertension: β = -0.26 [95% CI, -0.41 to -0.10]; transition to hypertension: β = -0.23 [95% CI, -0.44 to -0.23]), with similar differences in cerebral gray matter volume (hypertension: β = -0.32 [95% CI, -0.52 to -0.13]; transition to hypertension: β = -0.30 [95% CI, -0.56 to -0.05]), frontal cortex volume (hypertension: β = -0.43 [95% CI, -0.63 to -0.23]; transition to hypertension: β = -0.27 [95% CI, -0.53 to 0]), and parietal cortex volume (hypertension: β = -0.22 [95% CI, -0.42 to -0.02]; transition to hypertension: β = -0.29 [95% CI, -0.56 to -0.02]). Participants with hypertension also had smaller hippocampal volume (β = -0.22; 95% CI, -0.42 to -0.02), greater ventricular volumes (lateral ventricle: β = 0.44 [95% CI, 0.25-0.63]; third ventricle: β = 0.20 [95% CI, 0.01-0.39]), larger free water volume (β = 0.35; 95% CI, 0.18-0.52), and lower fractional anisotropy (β = -0.26; 95% CI, -0.45 to -0.08) than those who had normotension. Holding hypertension status constant, a 5-mm Hg increase in systolic BP was associated with smaller temporal cortex volume (β = -0.03; 95% CI, -0.06 to -0.01), while a 5-mm Hg increase in diastolic BP was associated with smaller parietal cortex volume (β = -0.06; 95% CI, -0.10 to -0.02). The negative association of hypertension and BP change with regional brain volumes appeared stronger in men than women for some regions. In this cohort study, early adulthood hypertension and BP change were associated with late-life volumetric and WM differences implicated in neurodegeneration and dementia. Sex differences were observed for some brain regions whereby hypertension and increasing BP appeared more detrimental for men. These findings suggest that prevention and treatment of hypertension in early adulthood is important for late-life brain health, particularly among men.