Patients with cancer experience acute declines in physical function, hypothesized to reflect accelerated aging driven by cancer-related symptoms and effects of cancer therapies. No study has examined long-term trajectories of physical function by cancer site, stage, or treatment compared with cancer-free controls. Examine trajectories of physical function a decade before and after cancer diagnosis among older survivors and cancer-free controls. This prospective cohort study enrolled patients from 1993 to 1998 and followed up until December 2020. The Women’s Health Initiative, a diverse cohort of postmenopausal women, included 9203 incident cancers (5989 breast, 1352 colorectal, 960 endometrial, and 902 lung) matched to up to 5 controls (n = 45 358) on age/year of enrollment and study arm. Cancer diagnosis (site, stage, and treatment) via Medicare and medical records. Trajectories of self-reported physical function (RAND Short Form 36 [RAND-36] scale; range: 0-100, higher scores indicate superior physical function) estimated from linear mixed effects models with slope changes at diagnosis and 1-year after diagnosis. This study included 9203 women with cancer and 45 358 matched controls. For the women with cancer, the mean (SD) age at diagnosis was 73.0 (7.6) years. Prediagnosis, physical function declines of survivors with local cancers were similar to controls; after diagnosis, survivors experienced accelerated declines relative to controls, whose scores declined 1 to 2 points per year. Short-term declines in the year following diagnosis were most severe in women with regional disease (eg, -5.3 [95% CI, -6.4 to -4.3] points per year in regional vs -2.8 [95% CI, -3.4 to -2.3] for local breast cancer) or who received systemic therapy (eg, for local endometrial cancer, -7.9 [95% CI, -12.2 to -3.6] points per year with any chemotherapy; -3.1 [95% CI, -6.0 to -0.3] with radiation therapy alone; and -2.6 [95% CI, -4.2 to -1.0] with neither, respectively). While rates of physical function decline slowed in the later postdiagnosis period (eg, women with regional colorectal cancer declined -4.3 [95% CI, -5.9 to -2.6] points per year in the year following diagnosis vs -1.4 [95% CI, -1.7 to -1.0] points per year in the decade thereafter), survivors had estimated physical function significantly below that of age-matched controls 5 years after diagnosis. In this prospective cohort study, survivors of cancer experienced accelerated declines in physical function after diagnosis, and physical function remained below that of age-matched controls even years later. Patients with cancer may benefit from supportive interventions to preserve physical functioning.