Evolution under Stochastic Transmission: Mutation-Rate Modifiers

In evolutionary models of large populations, it is common to analyze the effects of cyclic or random variation in the parameters that describe selection. It is less common, however, to study how stochasticity in the genetic transmission process itself affects evolutionary outcomes. Suppose that a gene locus has alleles A and a under constant selection. This locus is linked to a modifier locus with alleles M₁ and M₂, which control the mutation rate from A to a. The Reduction Principle states that, near a mutation--selection balance where M₁ is fixed with mutation rate u₁, a rare allele M₂ can invade if its associated rate u₂ is lower than u₁. This result, valid for both haploids and diploids, assumes constant mutation rates through time. We extend this framework by allowing the mutation rate associated with M₂ to fluctuate randomly across generations, denoted as u2,t. In this stochastic setting, the condition for invasion by a new modifier allele depends not only on the resident mutation rate u₁ and the mean mutation rate u₂ associated with the invading allele, but also on the temporal distribution of u2,t, the strength of selection at the $A/a$ locus, and the recombination rate between M₁/M₂ and $A/a$. The analysis shows how stochasticity and recombination in transmission do not simply modify the magnitude of evolutionary change predicted under deterministic assumptions. Instead, through their interaction with selection and linkage, they can generate conditions under which the direction of modifier evolution is qualitatively reversed relative to the deterministic Reduction Principle.