Sexual dysfunction is a treatment sequela in survivors of rectal cancer (RC). Differences in health-related quality of life (HRQOL) can occur based on ostomy status (permanent ostomy vs anastomosis). To describe alterations in sexual function and HRQOL based on ostomy status in long-term (≥5 years) survivors of RC. Survivors of RC with an ostomy (n = 181) or anastomosis (n = 394) were surveyed using validated HRQOL and functional status tools. We compared sexuality outcomes between the ostomy and anastomosis groups and reported differences adjusted for clinical and demographic characteristics. Qualitative data from one open-ended question on survivors’ greatest challenges since their surgery were analyzed to explore sexuality, symptoms, and relationships. Whether sexually active, satisfaction with sexual activity, and select sexual dysfunction items from the Modified City of Hope Quality of Life-Colorectal. Survivors with a permanent ostomy were more likely to have been sexually inactive after surgery if it occurred before 2000 and experience dissatisfaction with appearance, interference with personal relationships and intimacy, and lower overall HRQOL. Female survivors of RC with an ostomy were more likely to have problems with vaginal strictures and vaginal pain after surgery that persisted at the time of the survey (≥5 years later). Radiation treatment, tumor stage, soilage of garments in bed, and higher Charlson-Deyo comorbidity scores were negatively associated with outcomes. Six qualitative themes emerged: loss of and decreased sexual activity, psychological issues with sexual activity, physical issues with sexual activity, partner rejection, ostomy interference with sexual activity, and positive experiences with sexuality. Sexual dysfunction is a common long-term sequela of RC treatment, with more problems observed in survivors with a permanent ostomy. This warrants widespread implementation of targeted interventions to manage sexual dysfunction and improve HRQOL for these survivors.