Scaffold or Crutch? Immediate Benefits and an Unresolved Delayed Cost of LLM-Assisted Practice in Vocational Education

Large language models (LLMs) are entering vocational classrooms faster than behavioral evidence on durable skill. In a class-randomized 2 × 2 factorial experiment embedded in a vocational data-analysis course, 272 students in eight classes practiced with either a course-customized LLM assistant or resource-matched conventional support and then took an immediate posttest with the assistant available or unavailable; three weeks later, all cells completed an unassisted parallel-form retest (236 retained; 13.2% attrition). Primary inference is design-aware—randomization tests over the eight-class assignment space, cluster-robust t(7) intervals, and wild cluster bootstrap—with individual-level ANCOVA as a precision-weighted description. Test-phase availability improved immediate performance by 7.0 points (d = 0.50, p < 0.001; randomization p = 0.063, wild bootstrap p = 0.008), and assisted practice raised it by 5.1 points (p = 0.001; randomization p = 0.17). Three weeks later, the estimates indicated a retention cost of assisted practice (−4.1 points, d = −0.31, p = 0.006; cluster-robust p = 0.030), with lower far transfer and error detection (Holm p = 0.049, not design-confirmed); however, randomization tests (p = 0.208–0.333) and the wild bootstrap (p = 0.11) did not confirm the delayed cost, which therefore remains a model-based, unresolved estimate. No moderator buffered it (metacognitive awareness b = 1.10, p = 0.814); self-reported willingness to reuse the assistant rose in the trained arm, and copy-adoption of assistant output tracked it within the trained arm (r = −0.27, unadjusted). The findings are evidence about when an LLM is present during learning, not a verdict on the technology.