Multimodal spatiotemporal knowledge graph construction for conflict-event analysis: an integrated framework with geographic entity-map association

Heterogeneous conflict-event information is distributed across textual reports, web images, and scanned maps, making it difficult to represent entities, events, geographic locations, temporal attributes, and multimedia evidence within a unified knowledge graph. This study proposes an integrated framework for constructing multimodal spatiotemporal knowledge graphs from event-oriented heterogeneous data. The framework combines textual entity, relation, and event extraction; visual object detection; scanned-map coordinate recognition; geographic entity grounding; cross-modal entity fusion; provenance-aware storage; and graph visualization. A cascaded YOLOv5-PaddleOCR procedure first detects and crops coordinate annotation regions from scanned maps and then reconstructs the map extent from the recognized coordinates. Textual place entities are geocoded using GeoNames and linked to relevant maps through point-polygon spatial intersection. Quantitative experiments showed F1-scores of 0.76-0.84 for the evaluated text extraction tasks. For aircraft, ship, and general web-image detection, YOLOv8 obtained mAP@0.5 values of 0.935, 0.890, and 0.922, respectively. On the 287-map scanned-map dataset, YOLOv5 achieved a region-detection F1-score of 0.977. In the 278-map paired out-of-fold OCR evaluation, complete map-coordinate recognition increased from 64.03% with direct PaddleOCR to 87.05% with the cascaded procedure, and the paired difference was statistically significant (exact McNemar test, p < 0.001). Under an event- and source-document-grouped evaluation protocol, the complete constrained cross-modal entity-fusion method achieved an F1-score of 0.884. A conflict-event case study further demonstrates the practical applicability of the framework.