Multimodal Document Parsing Algorithm for Power Grid Simulation Based on Vision-Transformer and Cross-Attention

Multimodal documents for power grid simulation face challenges such as complex layout structure, difficulties in cross-modal semantic alignment, and unstable parameter extraction due to easy misinterpretation of symbols and values. This paper proposes a parsing algorithm based on Vision Transformer (ViT) and Cross-Attention: First, the PDF/image is denoised and skewed, and the layout is detected, dividing the page into text blocks, chart blocks, formula blocks, and table blocks. Then, ViT is used to extract patch-level features from image-class regions, and a text encoder is used to perform contextual representation of the OCR sequence, embedding coordinates/layout levels as position. Second, a cross-modal cross-attention module is constructed to query image features using text and vice versa, obtaining an aligned unified token sequence. Third, structured decoding is performed on the unified sequence, jointly extracting title levels, table cells, legends-curves, formula symbols-meanings, and "parameter-unit-value" triples. Finally, consistency constraints were used to verify unit conversion and numerical range, and JSON and simulation parameter tables that could be stored in the database were output. On the power grid simulation document dataset, the F1 score reached 94.8%, the TEDS score reached 97.4%, the mAP score for chart element recognition reached 92.6%, and the end-to-end page parsing accuracy reached 92.4%.