Often, studies will aggregate all participants identified as Hispanic/Latino, despite genetic and environmental substructures, preventing the meaningful interrogation of the roles of genetics and environment in human health. Using the Hispanic Community Health Study/Study of Latinos (HCHS/SOL), we examined how self-identified background group and genetic ancestry influence gene-environment interactions between body mass index (BMI) and a polygenic score for BMI (PGSBMI). Participants (n = 7,075) identified with six background groups: Central American, Cuban, Dominican, Mexican, Puerto Rican, and South American. Generalized linear models incorporating complex survey weighting were used to model BMI through joint and stratified (background group, estimated Amerindigenous [AME] ancestry) analyses including PGSBMI and other health-related variables. Interaction effects were modeled between PGSBMI and diet and age at immigration. Comparing pooled to background group-stratified analyses, we observe heterogeneous distributions of environmental and sociocultural variables, as well as differing associations with AME ancestry. Within the multivariate model, PGSBMI performance decreased with increasing AME ancestry. After stratification, PGS-age-at-immigration interactions remained statistically significant in some strata: Mexican background individuals born in the US (50 states/DC) (β = 1.33, p < 0.01), Dominican background individuals 6-12 years old (β = 4.38, p < 0.001), and Cuban background individuals 0-5 years old (β = 2.20, p = 0.015) relative to those ≥ 21 years old at migration. It is vital to understand populations of interest to model them appropriately and prevent possible confounding or misinterpretation. While this work focuses specifically on Hispanic/Latino groups, these lessons are relevant to other groups as we diversify work to better understand gene-environment interactions.