A Multimodal AI and Visual-Programming Framework for Sketch-Based BIM: Formalization and Empirical Validation

Early-stage architectural design is still dominated by hand-drawn 2D floor-plan sketches, and converting these sketches into object-based Building Information Modeling (BIM) models remains a manual bottleneck that delays downstream simulation, costing, and coordination. Recent advances in multimodal artificial intelligence (vision-language models, code-generating large language models) and visual programming for BIM (Dynamo, Grasshopper, Refinery) have not yet been unified. Objective: We present and empirically evaluate a semi-automatic framework in which a vision-language model (VLM) emits executable visual-programming code (Dynamo-compatible DesignScript) that is wrapped into a node graph and executed in Dynamo to drive Revit, producing native BIM elements; the IFC export is independently validated in BlenderBIM as a visual sanity check. This is a proof-of-concept evaluation (n = 5 case studies), not a claim of production-scale deployment. Methods: The seven-module pipeline (sketch input, preprocessing, VLM interpretation, code-to-graph translation, Dynamo + Revit execution, IFC export with a BlenderBIM sanity check, and human-in-the-loop review with a feedback loop) is formalized mathematically—a line-sharpness function, a probabilistic AI-success model, and a closed-form architect-effort expression—and evaluated on five case studies (a Medipol student sketch, a public RPLAN sample, a tablet-drawn two-story house, a CubiCasa5K sample, and a real Istanbul apartment) against three baselines: a CNN recognizer (MuraNet), a JSON-emitting LLM pipeline (Sketch2BIM), and a hand-coded visual-programming framework. The primary VLM is Qwen-VL 9B (open-weight, Apache 2.0); GPT-4V and Claude 3.5 Sonnet are ablated on Case A only as closed-weight sensitivity checks. Results: Across the five cases, the framework achieves wall/opening F1 of 0.84–0.91 (mean 0.88), IFC schema validity of 100% in every case, a 56–61% reduction in time-to-editable-BIM, and 58–74% fewer architect edits than fully manual modeling, outperforming all three baselines on the architect-edits-required metric. The closed-form model predicts a 72% effort reduction under nominal parameters but overestimates the observed 56–61% by 11–16 percentage points; once the empirically observed architect-edit time is substituted, the calibrated model matches the observed range. Numbers are means over three runs per case. Conclusions: Unifying multimodal AI code generation with visual programming for BIM produces a more robust, editable sketch-to-BIM pipeline than the JSON-schema approaches in the current literature. Caveat: the evidence is a proof-of-concept scale (n = 5); broader validation (≥20 cases, cross-practice) is a key objective for future research.