Gestational weight gain is known to influence fetal growth. However, it is unclear whether the associations between gestational weight gain and fetal growth vary by trimester. In a diverse cohort of 8,977 women who delivered a singleton between 2011 and 2013, we evaluated the associations between trimester-specific gestational weight gain and infant size for gestational age. Gestational weight gain was categorized per the 2009 Institute of Medicine (IOM) recommendations; meeting the recommendations was the referent. Large for gestational age and small for gestational age were defined as birthweight > 90th percentile or <10th percentile, respectively, based on a national reference standard birthweight distribution. Logistic regression models estimated the odds of having a large or small for gestational age versus an appropriate for gestational age infant. Only gestational weight gain exceeding the IOM recommendations in the 2nd and 3rd trimesters independently increased the odds of delivering a large for gestational age infant (Odds Ratio (95% Confidence Interval): 1st: 1.17 [0.94, 1.44], 2nd: 1.47 [1.13, 1.92], 3rd: 1.70 [1.30, 2.22]). Gestational weight gain below the IOM recommendations increased the likelihood of having a small for gestational age infant in the 2nd trimester only (1.76 [1.23, 2.52]). There was effect modification, and gestational weight gain below the IOM recommendations increased the likelihood of having a small for gestational age infant in the 2nd trimester and only among women with a pre-pregnancy body mass index from 18.5-24.9 kg/m2 (2.06 [1.35, 3.15]). These findings indicate that gestational weight gain during the 2nd and 3rd trimesters is more strongly associated with infant growth. Interventions to achieve appropriate gestational weight gain may optimize infant size at birth.