What question did this study set out to answer?

The aim is to develop a chatbot that utilizes multimodal data for more accurate skin disease diagnosis.

May 16, 2026Open Access

Advancing Skin Disease Diagnosis Chatbot

Key Points

The aim is to develop a chatbot that utilizes multimodal data for more accurate skin disease diagnosis.
Developed a multimodal chatbot combining image analysis and natural language processing.
Used DenseNet169 and ResNet50 transfer learning models for feature extraction from skin lesion images.
Evaluated the system on a dataset of 11,747 images, with 7,930 for training and 3,817 for testing.
Achieved an accuracy of 77.07% and an AUC score of 96.72% in image analysis.
NLP module demonstrated an accuracy of 93.62% in processing user inputs.
System significantly improved diagnostic accuracy and user engagement compared to traditional methods.

Abstract

— Skin diseases are often early warning signs of underlying health problems and therefore timely and precise diagnosis is vital to successful treatment. Traditional diagnostic methods such as clinical examination and single-modality deep learning systems are unable to incorporate patient-specific contextual information, resulting in less personalisation and accuracy. We present a new multimodal skin disease diagnosis system that combines deep learning-based image analysis and natural language processing (NLP) in an interactive Telegram chatbot environment. This system uses an ensemble of DenseNet169 and ResNet50 transfer learning models for the extraction of fine-grained as well as global features from skin lesion images. It tackles the problem of vanishing gradients and limited feature representation.The proposed framework also considers user-provided contextual information such as skin type, history of chemicals and previous treatments along with image data to make personalised and context-aware predictions. The NLP-enabled chatbot allows for real-time interaction, gathers pertinent user information, and dynamically adjusts via self-learning capabilities to enhance response quality and diagnostic relevance over time. This amalgamation of multimodal data increases the robustness, generalisation, and overall performance of the system compared to conventional approaches.The experimental evaluation was performed on a dataset of 11,747 images, where 7930 images were used for training and validation and 3817 images for testing. The image-based ensemble model achieved an accuracy of 77.07% and an AUC score of 96.72%, while the NLP module achieved an accuracy of 93.62% in understanding and processing user inputs. The results show that the proposed system greatly enhances diagnostic accuracy, user engagement, and accessibility. Moreover, the use of the Telegram platform for deployment makes it scalable and user-friendly, providing the opportunity for preliminary diagnosis with the help of AI in real-time, and thus allowing for early medical intervention

Mark Helpful

Bookmark

Relay

View Full Paper