Key points are not available for this paper at this time.
Multimodal conversational AI systems have gained significant attention due to their potential to enhance user experience and enable more interactive and engaging interactions. This vital and complex research field seeks to integrate diverse modalities, including text, images, and speech, to develop conversational AI systems capable of comprehending, perceiving, and generating responses within a multimodal framework. By seamlessly incorporating various modalities, these systems can provide a more comprehensive and immersive conversational experience, enabling users to communicate in a more natural and intuitively. This research presents a novel multimodal architecture empowered by Deep Neural Networks (DNNs) for simultaneous integration and processing of diverse modalities. Multimodal data encompasses various sources like text, images, audio, video, or sensor data. The objective is to merge and harness information from these modalities to amplify learning and enhance performance across a spectrum of tasks. This research explores the extension of ChatGPT, a state-of-the-art conversational AI model, to handle multimodal inputs, including text and images or text and speech. We present a comprehensive analysis of the benefits and challenges of integrating various options into ChatGPT, examining their impact on understanding, interaction, and overall system performance. Through extensive experimentation and evaluation, we demonstrate the potential of multimodal ChatGPT to provide richer, more context-aware conversations, while also highlighting the existing limitations and open research questions in this evolving field. Multimodal ChatGPT outperform the current GPT-3.5 by 16.51% and it is clear that multimodal ChatGPTis capable of better performance and offer a pathway for further progress in the field of language models.
Pai et al. (Wed,) studied this question.