Robust dermatological image analysis via multitask learning on heterogeneous datasets

Deep learning has substantially advanced dermatological image analysis, but most methods remain tied to single datasets, annotation types, or narrowly defined tasks, limiting their ability to generalize across real clinical variability. This work proposes a unified cross-dataset multitask learning (MTL) framework for skin lesion analysis that jointly integrates classification and segmentation tasks from heterogeneous dermatoscopic and clinical image datasets. The model uses a shared Swin Transformer backbone with task-specific heads, treating each dataset-task pair as a distinct but related objective, thereby allowing datasets with different label structures and annotation types to be trained within the same architecture. To stabilize joint optimization, a dynamic loss orchestration (DLO) mechanism is introduced to adaptively regulate task contributions according to training behavior, reducing task-specific overfitting and encouraging more transferable feature representations. Experiments across multiple public dermatology datasets show that multi-dataset training improves performance across the evaluated benchmarks over single-task configurations. The proposed framework achieves 82.03% accuracy on Derm7pt and 96.95% accuracy on HAM10000 for classification, while reaching IoU scores of 94.03% on PH2 and 92.63% on HAM10000 for segmentation. Qualitative analyses using segmentation outputs, Grad-CAM visualizations, and t-SNE embeddings further indicate that the shared representation captures lesion-focused patterns that are consistent with clinical intuition. These results support cross-dataset MTL as an effective strategy for building versatile and interpretable dermatological image analysis models.