RCF-Face: Risk-Calibrated Factorization for Lightweight Face Verification Under Composite Disturbances

Compact face verifiers at access-control and surveillance terminals must operate under specular reflection, off-frontal pose, and mask occlusion, disturbances that frequently co-occur in practice. Existing objectives optimize identity discrimination and score-threshold reliability separately. As a result, disturbance cues leak into the identity embedding and inflate the impostor-score tail near the operating false-acceptance rate (FAR)—a structural weakness that backbone scaling alone does not resolve within a fixed parameter budget. We propose RCF-Face, a 1.12 M-parameter verifier for the sub-2 M-parameter, sub-0.2 GFLOP budget of Ampere-class edge GPUs, trained under a unified objective that combines representation-level leakage suppression (HSIC-based identity–disturbance factorization with augmentation-based counterfactual consistency) and decision-level score shaping via a softplus tail-risk penalty anchored to the target FAR. Across four disturbance-focused protocols (CFP-FP, IJB-C, RMFRD, and a reflection-augmented subset), RCF-Face reaches a Composite-Disturbance Score (CDS, the mean verification accuracy over the four protocols) of 89.05 at 0.166 GFLOPs, improving on the capacity-matched AdaFace-Lite baseline by 4.67 percentage points, with both per-protocol gains over this baseline significant under Bonferroni correction. Zero-shot evaluation on two open-source reflection-related datasets (SoF and MeGlass) shows that the robustness gains transfer to real-world images, although these datasets isolate single disturbance axes and do not constitute a composite-disturbance benchmark. On a Jetson Orin NX 16 GB, batch-1 latency is 2.41 ms (FP16) and 1.66 ms (INT8), supporting Ampere-class edge-GPU deployment.