Selective Component Ablation for Refusal Removal

Safety alignment in large language models (LLMs) is predominantly achieved through training models to refuse harmful requests. However, prior work has demonstrated that refusal behavior is mediated by a single direction in the residual stream space, implying that ablating this direction can remove refusal behavior. Existing ablation methods—whether activation-space directional ablation or weight-space abliteration—apply a uniform rank-one modification to entire weight matrices, ignoring the substantial heterogeneity in how different components (attention heads and MLP neurons) contribute to refusal behavior. We present PICK (PrecIse Component-piCKing), a method for selectively ablating components under a KL divergence budget constraint. PICK first computes a specificity score for each attention head and MLP neuron with respect to the refusal direction, then greedily selects components by estimating KL cost purely from weight mathematics, and finally applies selective rank-one LoRA adapters that precisely modify only the weight columns of selected components. Experiments on SmolLM3-3B and MiniCPM5-1B demonstrate that PICK removes 81.8% and 93.1% of refusal behavior respectively, while incurring KL divergences of only 0.000558 and 0.000001—over an order of magnitude lower than typical ablation approaches. Importantly, PICK requires no fine-tuning data or optimization: it operates entirely through forward inference and weight-space computation, completing in 3–10 minutes on a single GPU. Code is open-sourced at https://github.com/XuehangCang/pick.