Subgroup analysis is an important tool for studying treatment effect moderation. However, when a subgroup has a relatively small proportion (referred to as "focal subgroup"), standard subgroup analysis could encounter practical difficulties (e.g., low estimation precision). In this study, we propose an incremental subgroup analysis approach, which considers how the treatment effect would change as the proportion of focal subgroup gradually increases. The proposed approach provides estimates and confidence intervals for incremental subgroup effects, allowing visualization of the effect moderation trend with a continuous curve along with the corresponding confidence band. For estimation with baseline covariates, we extend a doubly robust method that can incorporate machine learning approaches for relaxing modeling assumptions, while allowing quantification of uncertainty for the effect estimate (e.g., via confidence intervals). Simulations are conducted to evaluate the performance of the estimation method. We illustrate the application of the proposed approach in an empirical example, assessing the moderation in the effect of a preventive intervention based on a relatively small subgroup. We hope that the proposed subgroup analysis approach provides an alternative or complementary method for studying effect moderation by subgroups.
Liu et al. (Mon,) studied this question.