We show that differences in childhood environments play an important role in shaping gender gaps in adulthood by documenting three facts using population tax records for children born in the 1980s. First, gender gaps in employment rates, earnings, and college attendance vary substantially across the parental income distribution. Notably, the traditional gender gap in employment rates is reversed for children growing up in poor families: boys in families in the bottom quintile of the income distribution are less likely to work than girls. Second, these gender gaps vary substantially across counties and commuting zones in which children grow up. The degree of variation in outcomes across places is largest for boys growing up in poor, single-parent families. Third, the spatial variation in gender gaps is highly correlated with proxies for neighborhood disadvantage. Low-income boys who grow up in high-poverty, high-minority areas work significantly less than girls. These areas also have higher rates of crime, suggesting that boys growing up in concentrated poverty substitute from formal employment to crime. Together, these findings demonstrate that gender gaps in adulthood have roots in childhood, perhaps because childhood disadvantage is especially harmful for boys.