Experience Filter: Translating Heterogeneous Signals into Motion

Human experience is inherently rich and multisensory, composed of intertwined visual, auditory, and kinesthetic feedback. Digital media capture only its audiovisual dimensions, omitting its mechanosensory dimensions. While full-body (kinesthetic) haptic feedback can enrich media, no general method exists for deriving structured motion from heterogeneous signals and combining it with the audiovisual content for rendering. The Experience Filter maps multimedia signals into structured motion trajectories through a common pipeline of modality-specific feature extraction followed by a modality-agnostic shared encoding, recombining them into a synchronized multisensory experience: each signal is reduced to a shared descriptor space - signal intensity, dominant spectral feature, and distance from neutral axis - then mapped to a trajectory via a fixed, deterministic, training-free encoding. Because this encoding space is hardware-independent, the motion can be retargeted to robotic platforms of arbitrary degrees of freedom and morphology. A feature-sensitivity analysis shows that the descriptor axes are independent and the encoding-space topology follows from the chosen descriptors rather than being fixed. The framework is demonstrated on real-world music and images, producing continuous, synchronized image-audio-motion experiences without manual authoring. The Experience Filter thus provides an analytical mapping from multimedia signals to combined image-audio-motion rendering, enabling expressive, user-adjustable experience across robotic platforms.