Volumetric audio production and its binaural rendering method

NHK is currently conducting research and development on “Meta Studio,” volumetric video capture studio, which aims to produce a photo-realistic volumetric video by capturing 3-D shape and properties of reflection of a subject using multi-viewpoint cameras. Alongside volumetric videos, research is being conducted on volumetric audio production that captures the sound radiation characteristics of the subject, the sound source, and reproduces the three-dimensionality and volume of the sound image as well as how it is perceived from various listening positions and directions. A microphone array consisting of 82 directional microphones has been constructed in Meta Studio. Using this microphone array, sound waves propagating in various directions along with direct sound emitted from the sound source are recorded as audio signals. The recorded signals are then expanded by a spherical wave function that serves as the basis functions for expressing the radiation characteristics in each direction. The exterior sound field near the listener is obtained by translating the expansion coefficients based on the addition theorem of spherical Bessel functions. From the sound pressure distribution in the sound filed and head-related transfer functions, binaural signals to be presented to both ears are calculated based on Kirchhoff-Helmholtz integral.