Force-Aware Multimodal Imitation Learning: Transformer-driven CVAE for long-horizon contact-rich manipulation

Accurately and safely performing long-horizon, contact-rich manipulation tasks remains a fundamental challenge in robotic imitation learning. We propose Force-Aware Multimodal Imitation Learning (FAMIL), a novel framework based on a transformer-enabled Conditional Variational Autoencoder (CVAE) that leverages visual, force, and proprioceptive inputs to produce robust, force-sensitive action policies. To address compounding errors and the lack of force awareness in conventional vision-only methods, FAMIL introduces a cross-attention mechanism for effective multimodal fusion and adopts action chunking to shorten horizon length and mitigate demonstration noise. At inference time, the framework enables real-time, deterministic action generation by discarding the encoder and utilizing zero-style latent variables, while a temporally weighted aggregation of predicted actions ensures smooth adaptation to dynamic environments without retraining.In the contact-rich whiteboard erasing task, FAMIL achieved a success rate of 96%, representing a 58% improvement compared to the vision-only baseline, and generalizes effectively to unseen positions and objectives. The long-horizon manipulation and dual-arm manipulation experiments achieved success rates of 92% and 95%, respectively, demonstrating that the proposed framework can effectively suppress compounding errors and is applicable to robots with various configurations.