- THIS ARTICLE
-
Abstract
- Full Text (PDF)
- Alert me when this article is cited
- Alert me if a correction is posted
- SERVICES
- Similar articles in this journal
- Similar articles in PubMed
- Alert me to new issues of the journal
- Download to citation manager
- Reprints & Permissions
- CITING ARTICLES
- Citing Articles via HighWire
- Citing Articles via Google Scholar
- GOOGLE SCHOLAR
- Articles by Cherry, J. L.
- Search for Related Content
- PUBMED
- PubMed Citation
- Articles by Cherry, J. L.
Selection in a Subdivided Population With Dominance or Local Frequency Dependence
Joshua L. Cherryaa Department of Organismic and Evolutionary Biology, Harvard University, Cambridge, Massachusetts 02138
Corresponding author: Joshua L. Cherry, Cambridge, MA 02140., cherry{at}oeb.harvard.edu (E-mail)
Communicating editor: N. TAKAHATA
| ABSTRACT |
|---|
The interplay between population structure and natural selection is an area of great interest. It is known that certain types of population subdivision do not alter fixation probabilities of selected alleles under genic, frequency-independent selection. In the presence of dominance for fitness or frequency-dependent selection these same types of subdivision can have large effects on fixation probabilities. For example, the barrier to fixation of a fitter allele due to underdominance is reduced by subdivision. Analytic results presented here relate a subdivided population that conforms to a finite island model to an approximately equivalent panmictic population. The size of this equivalent population is different from (larger than) the actual size of the subdivided population. Selection parameters are also different in the hypothetical equivalent population. As expected, the degree of dominance is lower in the equivalent population. The results are not limited to dominance but cover any form of polynomial frequency dependence.
NATURAL populations are likely to be characterized by some kind of population structure. The population-genetic and evolutionary consequences of such structure have been investigated since the beginnings of population genetics (![]()
![]()
![]()
![]()
![]()
![]()
![]()
![]()
![]()
![]()
![]()
![]()
![]()
![]()
![]()
![]()
![]()
![]()
One area of interest is the interaction of population structure with selection. ![]()
![]()
![]()
If the assumption of genic selection is relaxed, the problem becomes more difficult. Subdivision affects fixation probabilities when there is dominance for fitness or when the relative fitnesses of genotypes depend on their frequencies. An area of particular interest has been the case of underdominance (heterozygote disadvantage), which can serve as a barrier to the fixation of a fitter genotype. Subdivision reduces this barrier, and this effect has been interpreted as a simple case of Wright's "shifting balance" theory (![]()
Analytic results, including expressions for fixation probabilities, have been obtained for selection with dominance in the low-migration limit (![]()
![]()
![]()
![]()
| MODELS AND RESULTS |
|---|
Consider a finite island model of population structure, in which a finite number of demes ("islands") exchange migrants with one another. The population consists of D demes, each containing N haploid or N/2 diploid individuals. Each generation involves migration, selection, and genetic drift. The order in which these occur makes little difference when selection coefficients and migration rates are small compared to unity, so this order need not be specified. The migration rate (the expected fraction of genes that come from outside the deme in any generation) is given by m, and selection operates in a manner to be specified below.
Three processes, selection, drift, and migration, alter the frequency of an allele in a deme each generation. Migration is symmetric in the island model, so it does not affect the overall allele frequency
(it does not affect the mean, and its effect on the variance is negligible). The effects of selection and drift are approximately additive. The mean change in
results from selection, whereas the variance is the result of genetic drift.
Dominance:
![]()
Suppose that the fitnesses of genotypes aa, Aa, and AA are 1, 1 + 2hs, and 1 + 2s. The fitness difference between the two alleles is 2hs when paired with an a allele and 2s - 2hs when paired with an A allele. Thus the mean selective difference between the two alleles
, which might be called the marginal selection coefficient (by analogy to the marginal fitness), depends on the allele frequency and is given by
![]() |
(1) |
in an unstructured population with nonassortative mating. This can be rewritten as
(x) = 2hs + (2s - 4hs)x, which makes it clearer that
(x) has the form k0 + k1x, with k0 = 2hs and k1 = 2s - 4hs.
Now consider a subdivided population. We are interested in the mean and variance of the change in overall allele frequency from one generation to the next, M
and V
. Expressions for these as functions of
will allow use of diffusion approximations. To obtain these we need to consider the probability distribution of the allele frequency in the ith deme, xi. If drift within a deme is strong compared to selection, the population as a whole in effect serves in the short term as a source population for any subpopulation, with constant allele frequency
. Under these conditions the distribution of within-deme allele frequency is a beta distribution whose probability density function is

where
and the beta function B is defined by
(![]()
![]()
The beta-distribution approximation is valid when selection is weak compared to drift in a subpopulation, in the sense that |
| << 1/N. This condition must hold for all allele frequencies. In the present case,
(xi) = 2hs (1 - xi) + (2s - 2hs)xi takes on its most extreme values at allele frequencies of zero or one. Thus the constraint on the strength of selection becomes
![]() |
(2) |
and
![]() |
(3) |
This condition allows selection to be strong compared to drift in the population as a whole and hence allows selection to have a large effect on the fate of an allele.
The stochastic change in allele frequency in a deme comes from the binomial sampling of alleles. Thus the variance of the change in allele frequency in the ith deme is
(1/N)xi(1 - xi). Using expressions for the first two moments of the beta distribution we can show that the variance of the change in population-wide frequency
is given by
![]() |
(4) |
The variance definition of effective population size Ne is given by
Thus we have

This is a well-known expression for the effective size of a subdivided population conforming to an island model (![]()
![]()
![]()
The mean change, on the other hand, is affected by the more complex selection scheme. For the ith deme, the change in allele frequency xi due to selection is approximately
(xi)xi(1 - xi) = (k0 + k1xi)xi(1 - xi). We are interested in the mean of this quantity. Using the approximation that the xi are beta distributed with
along with expressions for the first three moments of a beta distribution, we can show that the mean change in
is given by
![]() |
(5) |
This expression has the form (k0e + k1e
)
(1 -
). Together with the form of V
(Equation 4), this shows that the diffusion for the subdivided population is equivalent to that for a panmictic population with altered parameters. The constants k0e and k1e are to be interpreted as the effective values of k0 and k1, i.e., the values that, in a hypothetical Wright-Fisher population with size Ne, yield roughly the same trajectory of allele frequency as does the actual subdivided population. From Equation 5 it follows that
![]() |
(6) |
Converting back to notation involving h and s and their analogs in the equivalent panmictic population he and se, and recalling our expression for Ne, we obtain
![]() |
(7) |
As in the case of genic selection, subdivision increases Ne by a certain factor and decreases se by this same factor, so that Nese is unaffected by subdivision. The effect of subdivision on he is to move it toward 1/2 and therefore to decrease the effective strength of dominance or frequency dependence, as expected from previous work (![]()
![]()
![]()

Alternative parameterizations:
There are several natural parameterizations of the selection model used above. The parameterization involving s and h cannot be applied to symmetric over- or underdominance: s would have to be zero for symmetry, while hs would have to be nonzero for over- or underdominance. One parameterization that can represent the symmetric cases is that involving k0 and k1. A more commonly used notation for dominance gives the fitnesses of aa, Aa, and AA as 1, 1 + s1, and 1 + s2. Let s1e and s2e be the effective values of s1 and s2. Using the fact that s1 = 2hs and s2 = 2s, we obtain

Note that s1e depends on s2 as well as s1.
In the symmetric case s2 = 0. If we let s' = s1 then both homozygotes have fitness 1 and the heterozygote has fitness 1 + s'. If s'e is the effective value of s' under subdivision, we have

More complicated forms of selection:
The analysis given above applies when the frequency dependence of the selective difference between the alleles has the form
(x) = k0 + k1x. This case includes an arbitrary degree of dominance when the relative fitnesses of the diploid genotypes do not depend on their frequencies. This is not the only possible form of frequency dependence;
(x) can in principle have any form whatsoever. Can we obtain diffusion approximations for more complicated forms of
(x) when the population is subdivided, again assuming that |N
(x)| is always small compared to 1?
The results obtained above are consequences of the forms of the first three moments of the beta distribution. The useful properties of these moments extend to the higher-order moments. These properties permit the analysis of cases where the frequency dependence is described by any polynomial, i.e., where
(x) = P(x) = k0 + k1x + k2x2 + ... + knxn. We see that the diffusion for a subdivided population with this form of selection is equivalent to that for a panmictic population with a different size and with
e(x) = Pe(x) = k0e + k1ex + k2ex2 + ... + knexn. Simple dominance is a special case of this, with the degree of the polynomials P and Pe equal to 1.
Recall that we are interested in the expected value of
(x)x(1 - x) when x has a beta distribution with parameters
and
For polynomial
(x),
(x)x(1 - x) is the sum of terms of the form kixi+1(1 - x). For any beta distribution

This follows from the recursion properties of the beta function mentioned earlier. In the present case, a + b = 2Nm, so the denominator is independent of
. The numerator expands to an expression of the form Q(
)
(1 -
), where Q(
) is a polynomial of degree i in
. It follows that E{P(x)x(1 - x)} has the form Pe(
)
(1 -
) for some polynomial Pe of the same degree as P. The coefficients of this polynomial (k0e, k1e, k2e, ... kne), which depend on the coefficients of P and on N and m, may be obtained by some tedious but straightforward algebra. Consider, for example, the case of quadratic frequency dependence, where
(x) = P(x) = k0 + k1x + k2x2. We have

Thus the quadratic term of P contributes quadratic, linear, and constant terms to Pe, so the value of k2 affects not only the value of k2e, but also the values of k1e and k0e. Combining this result with those given for first-degree frequency dependence we obtain
![]() |
(8) |
(compare with Equation 6). The same principle could be applied to a polynomial
(x) of any degree or to a Taylor series of an analytic function.
| COMPUTER SIMULATIONS |
|---|
To test the approximations used above, I have run computer simulations and compared the results to theoretical predictions. In these simulations frequency-dependent selection acts on a haploid population. Most of the simulations are for cases of linear frequency dependence and can also be interpreted in terms of diploidy with dominance. I utilize the parameterization involving s and h for these cases.
In these simulations the state of the population is represented by an array of D integers, each corresponding to a deme. Each integer indicates the number of copies of allele A in the deme and hence ranges from 0 to N. Each generation the new value for each deme is drawn from a binomial distribution. The index parameter n of this binomial (number of "trials") is equal to N. The probability parameter p (probability of "success") is determined by the current allele frequency in the deme xi, the population-wide mean allele frequency
, the migration rate m, and the selection parameters s and h. Let
This would be the expected allele frequency in the ith deme in the next generation if there were no selection. The (marginal) selection coefficient is given by
(
). Therefore p = (1 +
)
/(1 + 
).
Predictions of fixation probabilities follow from combination of the theory presented here with classical diffusion results. ![]()
![]()
Fig 1 compares theoretical predictions of fixation probabilities to the results of simulations for N = 100, D = 100, s = 10-4, and various values of m and h. The predictions are all very close to the observed values: all predictions are within 3.6% of the simulation results, and a majority are within 1%. The plot illustrates that the theory captures the effects of both the degree of dominance and the migration rate on the probability of fixation. Furthermore, all the observed mean fixation times are within a few percent of the theoretical predictions (maximum deviation 2.5%; data not shown), indicating that the diffusion is a good description of the trajectory of allele frequency as well as the probability of ultimate fixation.
|
Table 1 presents results for larger values of s. For s = 3 x 10-4, the predictions are still close to the observed values, differing by at most 7.4%. For s = 10-3, some of the predictions differ significantly from the simulation results for the smaller migration rate, especially at the extremes of over- and underdominance. This is to be expected because the weak selection assumption (Equation 2 and Equation 3) is not met: for h = -4, 2N(1 - h)s = -1, and for h = 3, 2Nhs = 0.6, neither of which is very small in magnitude compared to 1. Even at these extremes, however, the predictions come within 16% of the observed values. For the less extreme values of h the theoretical predictions are quite good.
|
Table 2 shows some results for different numbers of demes (D). With 30 demes, the predictions are again quite good: all of them are within 10% of the simulation results. With as few as 10 demes the assumption of a large number of demes is seriously violated. Although many of the predictions are close to the observations, some differ from them by as much as 32%.
|
In the simulation results presented so far the allele was initially present in a single copy. Fig 2 shows results for a range of initial allele frequencies. Fig 2A shows results for overdominance (h = 3) and underdominance (h = -4) with different migration rates. All of the points (simulation results) fall along the theoretical curves. All of the predicted fixation probabilities are within 3% of the simulation estimates, and the probabilities of loss also agree within 3%. Fig 2B shows results for a case of strong underdominance (h = -15). Results for such extreme underdominance were not given for an allele starting at a single copy because some fixation probabilities would be so low that they would be difficult to estimate by simulation. For higher initial allele frequencies the fixation probability is much larger and can be measured more easily. Fig 2B illustrates that the theory correctly predicts the reduced probability of fixation of a rare allele at high migration rates and the increased fixation probabilities at lower m. The predicted fixation probabilities are all within 6% of the estimates from simulation. Furthermore, all of the predicted probabilities of loss are within 5% of the simulation results.
|
The theory presented here covers any frequency dependence described by a polynomial. In the simulations discussed above this was a first-degree polynomial. Fig 3 compares predictions and results for cases of quadratic frequency dependence. The agreement of the predictions with the results is excellent: all of the predictions are within a few percent of the simulation results (the largest difference is 3.5%).
|
| DISCUSSION |
|---|
The theory presented here relates a subdivided population with frequency-dependent selection (including the case of dominance) to an equivalent panmictic population characterized by different parameters. Subdivision alters both the size of the equivalent panmictic population (the effective population size Ne) and the effective values of all of the parameters describing selection. In the case of dominance, the dominance parameter h is in effect moved toward 1/2 by subdivision; i.e., fitness is made effectively closer to additive. For frequency dependence described by any polynomial in allele frequency, the effective values of all of the polynomial coefficients are altered by subdivision.
The theoretical treatment assumed that selection was weak in one sense, but allowed that it was strong in another sense. The requirement is that selection is weak compared to drift in a subpopulation, i.e., that the product of deme size (N) and (marginal) selection coefficient is always small in magnitude compared to unity. This allows selection to be quite strong with respect to drift in the population as a whole, so that selection may have a large effect on the fate of an allele without the assumptions being violated.
Computer simulations confirm that the theory closely predicts fixation probabilities so long as the parameters meet the stated conditions, namely that D is large and |N
(x)| << 1 for all x between zero and one. As expected, when the parameter values violate these conditions the predictions are less reliable. Nonetheless, when the conditions are moderately violated, for example, when selection and within-deme drift are of comparable strength, the predictions are still quite good.
A special case covered by the theoretical results is that of dominance for fitness in the absence of other sources of frequency dependence. It was shown that subdivision in effect decreases the degree of dominance, as measured by the deviation of the dominance parameter h from 1/2, by a factor that depends only on the deme size N and the migration rate m. Specifically, subdivision changes the effective deviation of h from 1/2, he - 1/2, by a factor of Nm/(Nm + 1). The direction of this effect is in accord with the well-established fact that subdivision decreases the effect of dominance on fixation probabilities (![]()
![]()
![]()
The diffusion approximation derived here for a finite island model completely describes the trajectory of allele frequency over time in the presence of dominance or frequency-dependent selection. The results followed from the moments of the distribution of within-deme allele frequencies. This is a beta distribution for the island model. Under other models of subdivision this distribution may have a different form, but the moments of the distribution nonetheless characterize the population. These moments could be derived theoretically or measured empirically. So long as Fst is independent of allele frequency and ordinary genetic drift is the only stochastic force operating, these moments can be used to relate the subdivided population to an equivalent panmictic population even in the presence of frequency-dependent selection.
| ACKNOWLEDGMENTS |
|---|
I thank Christina Muirhead and Jon Wilkins for comments on the manuscript. John Wakeley encouraged me to work on this topic. This work was supported by National Science Foundation grant DEB-9815367 to John Wakeley.
Manuscript received October 21, 2002; Accepted for publication January 2, 2003.
| LITERATURE CITED |
|---|
CHERRY, J. L. and J. WAKELEY, 2003 A diffusion approximation for selection and drift in a subdivided population. Genetics 163:421-428.
DOBZHANSKY, T. and S. WRIGHT, 1941 Genetics of natural populations. V. Relations between mutation rate and accumulation of lethals in populations of Drosophila pseudoobscura. Genetics 26:23-51.
KIMURA, M., 1957 Some problems of stochastic processes in genetics. Ann. Math. Stat. 28:882-901.
KIMURA, M. and T. OHTA, 1969 The average number of generations until fixation of a mutant gene in a finite population. Genetics 61:763-771.
LANDE, R., 1985 The fixation of chromosomal rearrangements in a subdivided population with local extinction and colonization. Heredity 54(3):323-332.
MARUYAMA, T., 1970a Effective number of alleles in a subdivided population. Theor. Popul. Biol. 1:273-306.[Medline]
MARUYAMA, T., 1970b On the fixation probability of mutant genes in a subdivided population. Genet. Res. 15:221-225.[Medline]
MARUYAMA, T., 1972a Distribution of gene frequencies in a geographically structured finite population. I. Distribution of neutral genes and of genes with small effect. Ann. Hum. Genet. 35:411-423.[Medline]
MARUYAMA, T., 1972b Distribution of gene frequencies in a geographically structured population. III. Distribution of deleterious genes and genetic correlation between different localities. Ann. Hum. Genet. 36:99-108.[Medline]
MARUYAMA, T., 1972c Distribution of gene frequencies in a geographically structured population. II. Distribution of deleterious genes and of lethal genes. Ann. Hum. Genet. 35:425-432.[Medline]
MARUYAMA, T., 1974 A simple proof that certain quantities are independent of the geographical structure of population. Theor. Popul. Biol. 5:148-154.[Medline]
MARUYAMA, T. and M. KIMURA, 1980 Genetic variability and effective population size when local extinction and recolonization of subpopulations are frequent. Proc. Natl. Acad. Sci. USA 77:6710-6714.
NAGYLAKI, T., 1998 The expected number of heterozygous sites in a subdivided population. Genetics 149:1599-1604.
NEI, M. and N. TAKAHATA, 1993 Effective population size, genetic diversity, and coalescence time in subdivided populations. J. Mol. Evol. 37:240-244.[Medline]
SANTIAGO, E. and A. CABALLERO, 1995 Effective size of populations under selection. Genetics 139:1013-1030.[Abstract]
SLATKIN, M., 1977 Gene flow and genetic drift in a species subject to frequent local extinctions. Theor. Popul. Biol. 12:253-262.[Medline]
SLATKIN, M., 1981 Fixation probabilities and fixation times in a subdivided population. Evolution 35:477-488.
SLATKIN, M., 1991 Inbreeding coefficients and coalescence times. Genet. Res. 58:167-175.[Medline]
SPIRITO, F., M. RIZZONI, and C. ROSSI, 1993 The establishment of underdominant chromosomal rearrangements in multi-deme systems with local extinction and colonization. Theor. Popul. Biol. 44:80-94.[Medline]
TAKAHATA, N., 1991 Genealogy of neutral genes and spreading of selected mutations in a geographically structured population. Genetics 129:585-595.[Abstract]
WANG, J. and A. CABALLERO, 1999 Developments in predicting the effective size of subdivided populations. Heredity 82:212-226.
WHITLOCK, M. C. and N. H. BARTON, 1997 The effective size of a subdivided population. Genetics 146:427-441.[Abstract]
WRIGHT, S., 1931 Evolution in Mendelian populations. Genetics 16:97-159.
WRIGHT, S., 1939 Actualites Scientifiques et Industrielles 802: Exposes de Biometrie et de la Statistique Biologique XIII. Hermann et Cie, Paris.
WRIGHT, S., 1940 Breeding structure of populations in relation to speciation. Am. Nat. 74:232-248.
WRIGHT, S., 1941 On the probability of fixation of reciprocal translocations. Am. Nat. 75:513-522.
WRIGHT, S., 1943 Isolation by distance. Genetics 28:114-138.
This article has been cited by other articles:
![]() |
K. R. Takahasi Evolution of Coadaptation in a Subdivided Population Genetics, May 1, 2007; 176(1): 501 - 511. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. Fearnhead Perfect Simulation From Nonneutral Population Genetic Models: Variable Population Size and Population Subdivision Genetics, November 1, 2006; 174(3): 1397 - 1406. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. Williamson, A. Fledel-Alon, and C. D. Bustamante Population Genetics of Polymorphism and Divergence for Diploid Selection Models With Arbitrary Dominance Genetics, September 1, 2004; 168(1): 463 - 475. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. L. Cherry Selection, Subdivision and Extinction and Recolonization Genetics, February 1, 2004; 166(2): 1105 - 1114. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. Roze and F. Rousset Selection and Drift in Subdivided Populations: A Straightforward Method for Deriving Diffusion Approximations and Applications Involving Dominance, Selfing and Local Extinctions Genetics, December 1, 2003; 165(4): 2153 - 2166. [Abstract] [Full Text] [PDF] |
||||
- THIS ARTICLE
-
Abstract
- Full Text (PDF)
- Alert me when this article is cited
- Alert me if a correction is posted
- SERVICES
- Similar articles in this journal
- Similar articles in PubMed
- Alert me to new issues of the journal
- Download to citation manager
- Reprints & Permissions
- CITING ARTICLES
- Citing Articles via HighWire
- Citing Articles via Google Scholar
- GOOGLE SCHOLAR
- Articles by Cherry, J. L.
- Search for Related Content
- PUBMED
- PubMed Citation
- Articles by Cherry, J. L.











