Learning Dietary Effects on Personalized Weight via Cross-Gated Temporal Modeling

Weight management plays a crucial role in personal health management. However, most existing approaches rely on calorie-based estimation, which require users to manually record diet and exercise information. This process is inconvenient for daily life and not suitable for long-term use. Considering that taking photos of meals is easier than keeping detailed logs, weight prediction can be performed by effectively using visual and textual dietary data. Therefore, we propose a Cross-feature extraction Time-series model with a Gating mechanism (CTG) that predicts daily weight variations from multi-day dietary and weight data. Specifically, CTG integrates dietary images, textual descriptions, and historical weight records as inputs. CTG employs CLIP to obtain visual-textual representations. And we introduce a cross-feature extraction module to model bidirectional dependencies between diet and weight through time-aligned feature selection and linear layer. Besides, we use a GRU-based temporal predictor to accomplish the goal of weight prediction. Extensive experiments demonstrate that CTG can achieve promising performance, effectively reducing prediction errors The code will be available at: https://github.com/pingpighsy/CTG-main.