Selection, structure and the heritability of behaviour

Abstract Characters which are closely linked to fitness often have low heritabilities (VA/VP). Low heritabilities could be because of low additive genetic variation (VA), that had been depleted by directional selection. Alternatively, low heritabilities may be caused by large residual variation (VR=VP – VA) compounded at a disproportionately higher rate than VA across integrated characters. Both hypotheses assume that each component of quantitative variation has an independent effect on heritability. However, VA and VR may also covary, in which case differences in heritability cannot be fully explained by the independent effects of elimination-selection or compounded residual variation. We compared the central tendency of published behavioural heritabilities (mean=0.31, median=0.23) with morphological and life history data collected by Mousseau & Roff (1987). Average behavioural heritability was not significantly different from average life history heritability, but both were smaller than average morphological heritability. We cross-classified behavioural traits to test whether variation in heritability was related to selection (dominance, domestic/wild) or variance compounding (integration level). There was a significant three-way interaction between indices of selection and variance compounding, related to the absence of either effect at the highest integration level. At lower integration levels, high dominance variance indicated effects of selection. It was also indicated by the low CVA of domestic species. At the same time CVR increased disproportionately faster than CVA across integration levels, demonstrating variance compounding. However, neither CVR nor CVA had a predominant effect on heritability. The partial regression coefficients of CVR and CVA on heritability were similar and a path analysis indicated that their (positive) correlation was also necessary to explain variation in heritability. These results suggest that relationships between additive genetic and residual components of quantitative genetic variation can constrain their independent direct effects on behavioural heritability.