Vidgen: Long-Form Text-to-Video Generation with Temporal, Narrative and Visual Consistency for High Quality Story-Visualisation Tasks

Generating long-form videos conditioned on large story based text input is a new and relatively unexplored task. Current text-to-video models are designed to generate short video clips conditioned on small input texts, which while temporally consistent, are severely lacking in narrative and visual consistency which are key elements of good story visualisation. This paper presents Vidgen, a pipeline that ensures Long-Form Text-to-Video Generation with Temporal, Narrative and Visual Consistency. To address the above issues Vidgen employs a new approach which leverages a Large-Language- Model (LLM) for the pre-processing of input story/script, which extracts key actions and story elements from the given script, and converts it to a standardised format which will be read and understood by the text-to-video model. The paper also proposes fixing "embeddings" (spatial-information about the appearance of the character) using a new, faster and improved textual- inversion approach with pre-trained weights, to ensure consistent looking characters. The work done in this paper will allow any traditional text-to-video model to accept large input text in the form of a story and generate high quality, temporally consistent arbitrary-length videos that have consistent looking characters for the entire duration of the video.