Background: Adults with chronic medical conditions complicated by food insecurity or physical limitations may have higher barriers to accessing telehealth implemented during the COVID-19 pandemic. Objective: To examine the relationships of self-reported food insecurity and physical limitations with changes in health care utilization and medication adherence comparing the year before (March 2019-February 2020) and the first year of the COVID-19 pandemic (April 2020-March 2021) among patients with chronic conditions insured by Medicaid or Medicare Advantage. Methods: A prospective cohort study of 10,452 Kaiser Permanente Northern California members insured by Medicaid and 52,890 Kaiser Permanente Colorado members insured by Medicare Advantage was conducted. Difference-in-differences (DID) between the pre-COVID and COVID years in telehealth versus in-person health care utilization and adherence to chronic disease medicines by food insecurity and by physical limitation status were measured. Results: Food insecurity and physical limitations were each associated with small but significantly greater shifts from in-person to telehealth. Medicare Advantage members with physical limitations also had significantly greater decline in adherence to chronic medications from year to year compared with those without physical limitations (DID from pre-COVID year to COVID year ranged from 0.7% to 3.6% greater decline by medication class, p < 0.01). Conclusions: Food insecurity and physical limitations did not present significant barriers to the transition to telehealth during the COVID pandemic. The greater decrease in medication adherence among older patients with physical limitations suggests that care systems must further address the needs of this high-risk population.