A Robust Regression Algorithm with Fast Convergence

We present Supervised Gauss-Newton (abbreviated to "Sup-GN"), an extension of the well-known Iterative Reweighted Least Squares (IRLS) algorithm.Sup-GN dramatically increases convergence speed over IRLS once the solution is approached.It employs damping of Gauss-Newton iterations in a manner analagous to Levenberg-Marquardt (L-M) iterations.L-M dynamically mixes Gauss-Newton iterations with gradient descent through a variable damping factor.Sup-GN mixes Gauss-Newton iterations with standard IRLS iterations using a similar damping factor.Gauss-Newton iterations provide quadratic convergence but may diverge, whereas IRLS iterations have been proved to converge, through a majorization strategy.These two extremes are mixed in Sup-GN via a similar damping factor to Levenberg-Marquardt. Results on simulated and real data demonstrate the advantage in convergence speed of Sup-GN over standard IRLS, for practical linear and non-linear estimation problems.We integrate Sup-GN with Graduated Non-Convexity (GNC) which has been shown to provide beneficial global convergence properties for IRLS.