Action-incrementer: Action-incremental learning for temporal action segmentation

We address temporal action segmentation (TAS) that can continuously adapt to new actions beyond a fixed set of predefined action categories, mirroring the human ability to learn and accumulate knowledge over time. Prior incremental learning for TAS introduces new action classes through high-level activities, requiring annotations for full activity-associated action sets rather than focusing supervision on newly introduced atomic actions. In this paper, we define action-incremental learning for TAS and propose a novel framework for this setting, named Act ion- Inc rementer ( Act-Inc ). To support continual adaptation to emerging actions, in contrast to existing frame-wise TAS methods, we reformulate TAS as a mask-based segmentation task that predicts class-agnostic temporal masks and their corresponding action labels. This decoupled formulation encourages temporally coherent segment representations under action-incremental learning. Furthermore, we employ adaptive knowledge distillation and mask-based pseudo-labeling to preserve knowledge of previously learned actions while learning new classes. These components alleviate catastrophic forgetting and background semantic shift during action-incremental learning. We evaluate the proposed framework under various incremental learning scenarios on popular TAS benchmarks. Experimental results demonstrate that Act-Inc consistently outperforms representative continual learning baselines adapted to TAS.