Polygenic scores have potential for predicting health outcomes based on genomic data. However, their portability across diverse populations is hindered by the overrepresentation of European ancestries in discovery genome-wide association (GWAS) studies. The current study evaluated the transferability of substance use polygenic scores across European, African, American, and East Asian populations. European-based scores demonstrated reduced predictive accuracy in non-European populations with mean relative accuracies of 36% in American, 22.5% in East Asian, and 3.8% in African ancestries. Beyond discrete genetic ancestry groups, we found that European-based score prediction declined as a function of genetic distance from the European-stratified GWAS. That is, predictive accuracy declines as genetic distance, derived from continuous measures of ancestry, from the discovery GWAS increases. We evaluated the extent to which differences across populations in linkage disequilibrium (LD), allele frequency, and heritability explained the reduction in accuracy. Across phenotypes we found that approximately 56.0% of accuracy decline in American ancestries was explained by LD and allele frequency differences, with differences in LD appearing to be the primary factor. For East Asian ancestries this was 67.1% and was 84.0% in African ancestries. Differences across populations in heritability accounted for only small portions of reduced polygenic score transferability. This suggests that LD and frequency differences explain significant portions of accuracy loss, but substantial reductions persist, suggesting contributions from other factors like causal effect size differences. The current study provides important information on the causes and extent of the polygenic score transferability problem and has important implications for refinement of polygenic scores. Expanding the diversity of GWAS and target samples remains essential for improving accuracy and equity in prediction.