Optimization of Noise Shaping Filter for Quantizer With Error Feedback

Noise shaping filters for quantizers with error feedback are designed to mitigate the effects of quantization errors. In this paper, we prove that if the transfer function from the quantization error to the signal-of-interest has minimum phase and there is no constraint on the feedback signal, then the scaled inverse of the transfer function is the optimal noise shaping filter. Next, we design a noise shaping filter to minimize the variance or the l2norm of the error in the signal-of-interest under a constraint on the variance or the l2norm of the feedback signal, which can be expressed as bilinear matrix inequalities (BMIs). Although the BMIs are not convex, we prove that the minimization of the error variance under the constraint on the variance of the feedback signal can be cast into a convex optimization problem. This enables us to design optimal noise shaping filters using numerical methods. We formulate our design problem as a convex optimization problem using extended linear matrix inequalities to obtain noise shaping filters, except for the special case. Examples are provided to demonstrate the effectiveness of the designed noise shaping filters.