Hierarchical reinforcement learning using a world model for multi-objective optimization of micro-milling

Existing micro-milling optimization methods often rely on fixed objective weights or direct parameter control, limiting their ability to handle state-dependent trade-offs among surface quality, energy consumption, and machining time. This study proposes a preference-constrained hierarchical actor-critic framework with a learned world model for sequential parameter optimization. Unlike direct-parameter reinforcement learning, the proposed method separates objective-level preference learning from feasible command generation. A high-level Dirichlet policy selects an interpretable preference vector over surface quality, energy, and time, while a constrained inverse control module maps it to spindle speed, feed rate, and depth of cut within prescribed bounds and rate limits. A multimodal temporal convolutional gated recurrent unit (MTC-GRU) world model trained on 900 physical Ti-6Al-4V micro-milling trials serves as the rollout environment for policy learning. Controller performance is evaluated through closed-loop rollouts within the frozen learned environment, while three predetermined variable-parameter trajectories, each physically replayed nine times, are used to assess recursive world-model fidelity under low, medium, and high transition intensities. Within the learned environment, preference-constrained hierarchical actor-critic (PC-HAC) improves surface quality, energy use, and machining time under the stationary objective, and achieves larger gains in the cumulative energy-budget task. At κ = 0.80, PC-HAC reduced the budget-penalized score by 53.9% relative to the greedy controller. The recursive physical replay evaluation yields overall R 2 values of 0.810 for surface roughness and 0.975 for energy, with prediction error increasing as transition magnitude increases. These findings support the proposed preference-space control strategy within the investigated micro-milling setting and provide a foundation for future physical closed-loop validation.