Integrating vision and language: a novel approach for translation of low-resource Indic languages

Cross-lingual learning provides an excellent chance for knowledge transfer across multiple languages. However, the substantial resource disparity between high- and low-resource languages creates considerable issues. This study focuses on two Indic language families, Indo-Aryan, and Dravidian, as well as a definitely endangered low-resource language that often lacks the extensive training data available in high-resource languages, such as English. We present a unique approach termed Resource-Aware Multimodal Translation (RAMT), which combines large language models with vision-based character recognition to improve translation efficacy across a range of resource levels. RAMT uses the Continuous Wavelet Transform to translate low-resource text into a spatial representation, enabling a plug-and-play training process. This method streamlines the training process across multiple languages, reducing reliance on large datasets and enhancing model portability. In addition, our method captures sequential dependencies and spatial properties in the text, which improves stroke extraction and inter-character interactions. Empirical assessments of seven languages demonstrate considerable gains in both performance and processing speed, demonstrating RAMT’s usefulness in bridging the resource gap in cross-lingual applications. Our findings demonstrate that this integrated technique promotes more equitable language processing solutions, paving the way for improved access and comprehension in low-resource linguistic environments.