Eyes don’t lie: a multitask and multimodal classification for ASD screening on Indian data

“Autism spectrum disorder” (ASD) is a developmental disorder that encompasses impairments in social communication as well as restrictive/repetitive behaviors. Most of the traditional ASD screenings rely on standardized questionnaires like MCHAT. Although these approaches perform well in practice, they have inherent subjectivity in their approach, culture-specific biases, and parent-report biases that may lead to delayed or wrong diagnoses. This paper presents an alternate multimodal approach to ASD screening through deep learning-based analysis, examining eye positioning using images and measuring attention using videos. To achieve that, a primary dataset of Indian children was produced, consisting of 14 autistic and 14 non-autistic children. This paper proposes a multitask classification approach, the first task of which proposes a hybrid ConvNextBase and LightGBM model that uses eye positioning patterns of typically developing and autistic children and classifies them according to their positioning towards the region of interest. The results of the model on the primary Indian dataset achieved an accuracy of 97.65%, a precision of 99%, a recall of 94.29%, an F1-score of 97.06%, 99% specificity, and up to 98% ROC-AUC. The reliability of the proposed model was further validated using Cohen’s Kappa coefficient, with values of 0.95 for the 90–10 split, 0.92 for the 80–20 split, and 0.95 for the 70–30 split, indicating an almost perfect level of agreement and confirming the robustness of the primary dataset for the study. Furthermore, paired t-test analysis (t = 14.4, p < 0.001) confirms that the observed performance improvements are statistically significant. The second task proposes a system based on the “histogram of oriented gradients”(HOG) and “Linear Support Vector Machine” (Linear SVM) that quantifies participants’ attention rate and engagement ratios during live activities. Adding “max-margin object detection” (MMOD) to the proposed model helps detect the face from multiple angles. This is a novel feature used in this work for ASD trait classification. In experimentation, it is evaluated that Autistic children exhibit 36.46% of their attention and 72.98% of their engagement, whereas typically developing children demonstrated 72.53% of their attention and 96.26% of their engagement. All performance metrics are validated using paired t-tests. These findings suggest that eye positioning and attention patterns are promising indicators for ASD screening. This study confirms that deep learning-based approaches offer automatic, efficient, and culturally unbiased alternatives to traditional ASD screening methods. The results of the experiment reveal possibilities for computer vision integration for eye gaze analysis to help identify ASD traits at early stages and to mitigate overreliance on human judgments.