Estimating diversity and entropy profiles via discovery rates of new species

Summary The compositional complexity of an assemblage is not expressible as a single number; standard measures such as diversities (Hill numbers) and entropies (Rényi entropies and Tsallis entropies) vary in their order q which determines the measures' emphasis on rare or common species. Ranking and comparing assemblages depend on the choice of q . Rather than selecting one or a few measures to describe an assemblage, it is preferable to convey the complete story by presenting a continuous profile, a plot of diversity or entropy as a function of q ≥ 0. This makes it easy to visually compare the compositional complexities of multiple assemblages and to judge the evenness of the relative abundance distributions of the assemblages. In practice, the profile is plotted for all values of q from 0 to q = 3 or 4 (beyond which it generally changes little). These profiles are usually generated by substituting species sample proportions into the complexity measures. However, this empirical approach typically underestimates the true profile for low values of q , because samples usually miss some of the assemblage's species due to under‐sampling. Although bias‐reduction methods exist for individual measures of order q = 0, 1 and 2, there has been no analytic method that unifies these bias‐corrected estimates into a continuous profile. For incomplete sampling data, this work proposes a novel analytic method to obtain accurate, continuous, low‐bias diversity and entropy profiles with focus on low orders of q (0 ≤ q ≤ 3). Our approach is based on reformulating the diversity and entropy of any order q in terms of the successive discovery rates of new species with respect to sample size, that is the successive slopes of the species accumulation curve. A bootstrap method is applied to obtain approximate variances of our proposed profiles and to construct the associated confidence intervals. Extensive simulations from theoretical models and real surveys show that the proposed profiles greatly reduce under‐sampling bias and have substantially lower bias and mean‐squared error than the empirical profile, especially for q ≤ 1. Our method is also extended to deal with incidence data.