Engineering visual–textual fusion for automatic item categorization in Indonesian e-commerce platforms

In the modern digital era, e-commerce remains a fundamental pillar of the global economy. The rapid growth of online ventures has fundamentally reshaped how products are bought and sold. However, as the number of products on e-commerce platforms in Indonesia increases, the main challenge is accurately matching products to consumer preferences. While previous studies have primarily used unimodal approaches to product matching, relying on a single type of information, little research has examined how to recognize new products on e-commerce platforms. Utilizing a multimodal approach that integrates information from diverse data types, such as text and images, is increasingly appealing for improving product-matching quality. This study's main contribution is to create vector representations of image and text features using deep learning techniques, including Convolutional Neural Networks and Doc2Vec. It also involves merging features through a cross-modality approach using FeedForward Neural Networks and determining the appropriate parameters for clustering new products into existing clusters using Hierarchical Clustering. The research demonstrates that employing a multimodal approach that leverages text and image information can enhance product matching quality on e-commerce platforms in Indonesia, achieving a Normalized Mutual Information of 0.96.