Nonsteroidal anti-inflammatory drugs (NSAIDs) use has consistently been associated with lower risk of colorectal cancer (CRC); however, studies showed inconsistent results on which cohort of individuals may benefit most. We performed multivariable logistic regression analysis to systematically test for the interaction between regular use of NSAIDs and other lifestyle and dietary factors on CRC risk among 11,894 cases and 15,999 controls. Fixed-effects meta-analyses were used for stratified analyses across studies for each risk factor and to summarize the estimates from interactions. Regular use of any NSAID, aspirin, or non-aspirin NSAIDs was significantly associated with a lower risk of CRC within almost all subgroups. However, smoking status and BMI were found to modify the NSAID-CRC association. Aspirin use was associated with a 29% lower CRC risk among never-smokers (OR = 0.71; 95% CI: 0.64, 0.79), compared to 19% and 17% lower CRC risk among smokers of pack-years below median (OR = 0.81; 95% CI: 0.71, 0.92) and above median (OR = 0.83; 95% CI: 0.74, 0.94), respectively (p-interaction = 0.048). The association between any NSAID use and CRC risk was also attenuated with increasing BMI (p-interaction = 0.075). Collectively, these results suggest that obese individuals and heavy smokers are unlikely to benefit as much as other groups from the prophylactic effect of aspirin against CRC.