Calibrated Intervention in Human–AI Collaborative Forecasting: How Modification Intensity and Direction Interact to Determine Performance

Human oversight of AI predictions can either correct errors or introduce new ones. This study examines whether users can calibrate interventions in continuous AI-assisted forecasting, detecting errors, judging correction direction, and adjusting appropriately. An Agency (modification intensity) × Effectiveness (direction correctness) framework is introduced, producing a 2×2 strategy space. In an eye-tracking study (N = 16), users demonstrated magnitude sensitivity, adjusting more where AI errors were larger, but poor direction calibration, with accuracy near chance when errors lacked clear patterns. A significant Agency × Effectiveness interaction confirmed that modifying more does not guarantee modifying correctly; high-intensity, low-accuracy interventions degraded forecast accuracy below the unmodified AI baseline. Strategy states were situation-dependent: 77% of participants shifted quadrants across tasks. Gaze-based process markers were associated with intervention intensity but not accuracy. These findings indicate that appropriate reliance in continuous correction tasks requires directional calibration, not merely engagement.