Answers are grounded in this paper and cite their sources.
Background: Reliable structured-output generation is a prerequisite for using large language models (LLMs) in automated clinical documentation workflows, but many evaluations focus on clinical quality before testing whether outputs are parseable, schema-compliant, and stable. Methods: We evaluated three locally deployed open-weight LLMs in the 7- to 8-billion-parameter range (Llama3-Med42-8B, Meta-Llama-3-8B-Instruct, and Mistral-7B-Instruct-v0.3) for structured admission-note editing. Seventy de-identified English-language admission notes (35 internal medicine and 35 surgical) were processed by each model in three independent runs under two output-control conditions: a free-text JSON prompt and a schema-enforced structured-output condition. A total of 1260 local inferences were performed in LM Studio on consumer-grade hardware. Automated proxy metrics assessed JSON/schema validity, run-to-run stability, instruction compliance, verbosity, numeric-token preservation, and uncertainty-marker change without clinician adjudication of clinical correctness. Results: Under the free-text JSON prompt, the tested Mistral-7B-Instruct-v0.3/embedded-prompt configuration had the weakest structural reliability (74.3–78.6% first-pass validity per run; 18.6–21.4% persistent parse/schema failures after retry), with at least one final failure for 17 of 70 notes. In a message-format sensitivity analysis using Meta-Llama-3-8B-Instruct, embedding system instructions in the user message increased first-attempt invalid outputs compared with separate system/user roles (55/700, 7.9% vs. 12/700, 1.7%). Under schema enforcement, all models produced 70 of 70 first-pass valid, schema-compliant outputs in every run. Documentation behavior nevertheless differed by model, including differences in verbosity and numeric-token preservation. Conclusions: Schema enforcement removed parsing failures in this sample but did not eliminate model-specific editing behavior. Proxy-based screening can identify structurally unstable model-prompt or model-format configurations before clinician review.
No takes yet. Share an insight, caveat, or question.
Yang et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: