Quality of life (QoL) is relevant to people living with HIV (PLWH) with increased life expectancy because of antiretroviral therapy. Our cross-sectional study examined associations between sociodemographic, HIV-related and psychological variables, and QoL, overall and by age. PLWH (n = 614) completed questionnaires at enrollment in an alcohol treatment study. QoL was assessed by the 12-item Short Form Survey, which includes physical and mental domains. Linear regression models evaluated the association of age and other factors with mental and physical QoL. PLWH younger than 50 years (n = 310) reported poorer mental QoL but better physical QoL compared to older PLWH (n = 304). Poorer mental QoL was associated with substance use, depression, and anxiety. Poorer physical QoL was associated with depression and history of injection drug use. We identified age-group differences in QoL for this primary care-based sample. Health care providers can use our findings to guide patient-centered care.