Population genetics is the discipline that turns a genotype matrix into a story about history. Two samples of the same species, genotyped on the same chip, will show different allele frequencies because of drift, migration, admixture, and selection — and every downstream analysis, GWAS included, inherits those differences. This post covers the tools I actually use and the ways they lie.
Population structure: PCA first, ADMIXTURE second
Principal components analysis (PCA) on a mean-centred, sample-normalised genotype matrix is the cheapest honest look at structure you can take. The EIGENSOFT lineage (Patterson, Price & Reich 2006) is still the reference implementation, and its smartpca builds the projection that GWAS pipelines reuse as covariates. PCA gives continuous axes; it does not name populations, and it is dominated by the largest, most differentiated splits. A third PC is not automatically "the third ancestry".
ADMIXTURE (Alexander, Novembre & Lange 2009) fits a maximum-likelihood model of K ancestral populations and per-individual ancestry proportions, which is genuinely useful when you want discrete admixture fractions rather than axes. The catch is K: it must be chosen, cross-validation error helps but does not settle it, and every K from 2 to 12 will produce a plausible-looking bar plot. Treat ADMIXTURE output as a model, never as ground truth.
- Prune variants in LD before PCA and ADMIXTURE; correlated markers over-weight a few regions.
- Remove close relatives first; structure estimates explode with duplicate samples.
- Report both plots. They answer different questions and disagree informatively.
Linkage disequilibrium is a time-dependent quantity
Linkage disequilibrium (LD) is the non-random association of alleles at different loci. It is generated by drift, admixture, and selection, and broken down by recombination and recurrent mutation. Two consequences dominate practice. First, LD is the reason GWAS works at all: a genotyped marker tags untyped causal variants, so imputation panels and fine-mapping both depend on the local LD structure. Second, LD decays with physical distance at a rate set by effective population size and recombination, so r² over 100 kb says as much about history as about biology.
Compute r² and D′ per population, never globally. An LD block defined in one ancestry is the wrong block in another, and a fine-mapping credible set transferred across ancestries without re-estimating LD is a common and avoidable error.
Fst: useful, bounded, and often over-read
The fixation index FST measures the proportion of genetic variance partitioned between populations. The Weir & Cockerham (1984) estimator remains the standard because it handles small, unequal sample sizes without the bias of naive moment estimators. FST ranges 0 to 1, but most human genome-wide averages sit below 0.15, and a per-site value is noisy. Use genome-wide FST to describe differentiation; use per-site FST only as one input to a sweep scan, with windowed smoothing and a null distribution.
Selective sweeps: multiple signals, one process
A selective sweep leaves several correlated footprints: reduced diversity, extended haplotype homozygosity, derived-allele frequency skew, and elevated FST between selected and unselected populations. No single statistic detects all of them, which is why serious scans combine complementary tests.
| Test | Signal detected |
|---|---|
| iHS | Long haplotypes within one population |
| XP-EHH | Haplotype length contrast between populations |
| CLR / SweepFinder | Site-frequency-spectrum distortion |
| FST outlier | Between-population differentiation |
Voight et al. (2006) introduced iHS and mapped the within-population sweep landscape; Sabeti et al. (2007) formalised XP-EHH as the cross-population complement. Both are haplotype-based and both require phased data, so phasing error — not selection — drives many apparent outliers.
The intersection with GWAS
Structure and association are inseparable. Uncontrolled ancestry inflates test statistics, which is why PCA covariates or a mixed model are mandatory. But the relationship runs deeper: LD determines the resolution of every hit, allele-frequency differences across ancestries determine which loci are even testable in a given cohort, and trans-ancestry meta-analysis is the strongest current lever on fine-mapping. Sweep scans also intersect GWAS directly — an FST outlier overlapping a disease locus is either biology or confounding, and distinguishing them requires phenotype geometry, not more statistics.
Practical order of operations
Variant QC and LD pruning, relatedness removal, PCA, ADMIXTURE at several K, per-population LD estimation, then either association with structure covariates or a sweep scan against a properly simulated null. Skip a step and you will not know which one produced your signal. The unglamorous truth is that most population-genetic errors are QC errors wearing a statistical costume.
References
- Patterson N, Price AL, Reich D. Population structure and eigenanalysis. PLoS Genet. 2006;2(12):e190.
- Price AL, Patterson NJ, Plenge RM, et al. Principal components analysis corrects for stratification in genome-wide association studies. Nat Genet. 2006;38(8):904–909.
- Alexander DH, Novembre J, Lange K. Fast model-based estimation of ancestry in unrelated individuals. Genome Res. 2009;19(9):1655–1664.
- Weir BS, Cockerham CC. Estimating F-statistics for the analysis of population structure. Evolution. 1984;38(6):1358–1370.
- Voight BF, Kudaravalli S, Wen X, Pritchard JK. A map of recent positive selection in the human genome. PLoS Biol. 2006;4(3):e72.
- Sabeti PC, Varilly P, Fry B, et al. Genome-wide detection and characterization of positive selection in human populations. Nature. 2007;449(7164):913–918.
- Purcell S, Neale B, Todd-Brown K, et al. PLINK: a tool set for whole-genome association and population-based linkage analyses. Am J Hum Genet. 2007;81(3):559–575.
- Auton A, Brooks LD, Durbin RM, et al. A global reference for human genetic variation. Nature. 2015;526(7571):68–74.
- Visscher PM, Wray NR, Zhang Q, et al. 10 years of GWAS discovery: biology, function, and translation. Am J Hum Genet. 2017;101(1):5–22.