MM-instruct: Generated visual instructions for large multimodal model alignment

This is the authors' abstract. We don't add key points for this paper.

This paper presents MM-Instruct, an automated pipeline for generating diverse and high-quality visual instruction data to better align large multimodal models (LMMs) with real-world use cases. While previous works have focused on question-answering data, their generated instruction datasets pose challenges for broader application scenarios. Additionally, manually collecting diverse instruction data at scale from users is prohibitively costly. To mitigate these issues, MM-Instruct leverages ChatGPT to automatically generate diverse instructions from a limited set of seed instructions through augmentation and summarization. It then uses an open-sourced large language model (LLM) to construct instruction-following answers and builds a large-scale visual instruction dataset. Evaluating the LLaVA-Instruct models trained with the generated data shows significant improvements in instruction-following capabilities compared to LLaVA-1.5 models. MM-Instruct releases its synthetic dataset containing diverse instructions and high-quality instruction-answer pairs to support training LMMs for real-world applications.