Text-to-video generation with multi-loss sequential generative adversarial network

Generating videos from text remains challenging due to issues like structural complexity, semantic alignment, temporal coherence, and frame discontinuity. It is still highly difficult to generate realistic videos where the frames must comply to both spatial and temporal coherence, despite the fact that Generative Adversarial Networks (GANs) have been successfully deployed to generate videos conditioned on a text description. This study proposes a Sequential Generative Adversarial Network (Seq-GAN) that employs multiple generators and discriminators to generate realistic and high-quality videos using written descriptions, while maintaining semantic alignment and relevance between frames. The approach incorporates a modified Variational Autoencoder (VAE) with reduced and adjusted 2D layers to extract static “gist” features for improved performance. For dynamic frame generation, the model uses three generators (Gₚ, Gₙ, Gy) and three discriminators (Dₚ, Dₙ, Dy) trained with different optimizers: RMSprop, Adam, and SGD. Gₚ and Dₚ generate basic frame structures, while Gₙ, Gy, Dₙ, and Dy refine video quality and ensure relevance to the text. Experimental results on SBMG, KTH, and UCF-101 datasets show that Seq-GAN achieves notable improvements in Inception Score (IS), Fréchet Video Distance (FVD), and CLIP-similarity confirming the method’s effectiveness in generating realistic, semantically accurate video sequences.