Audio production and reproduction of six-degree of freedom content using audio-related metadata

Six-degree of freedom (6DoF) content for immersive media is composed by superimposing virtual spaces for audio and video. When producing content, it is necessary to assign appropriate positions to audio objects using audio-related metadata to avoid conflicts in the perception of localization between hearing and seeing. However, in the case of live-action content, there are several hurdles to define the positions of audio objects: the units used to describe the virtual space differ between the audio-related metadata and the scene descriptions in video, the locations where audios are produced are not explicitly marked on video objects, and audio-related metadata cannot be edited in real-time. To address the issue, a content production environment that allows creators to freely set the positions of audio objects while referencing video objects was developed by using a game engine. Additionally, in 6DoF content, events and user interactions occur, making it essential for the time axes of the virtual spaces of audio and video to align. To the end, an audio renderer capable of interoperating with a video renderer was developed, by implementing mutual communication functionality with the video renderer, random access functionality for content, and functionality to separately record position data for audio objects.