Least‐squares Filters Without Transient Errors: An Examination of the Errors in Least‐squares Filter Design*

A bstract A new approach has been developed for the design of cross‐equalization filters by the least‐squares method. The filters estimated by this new exact method are subject to only two types of error: bias and random error. Cross‐equalization filters estimated by a more conventional least‐squares method are further subject to “transient error”. This type of error becomes important when designing filters from a data gate of a length comparable with the length of the filter, i.e., less than four times the length of the filter. The effect of altering various design parameters has been investigated for the new method. It has been found that the proportion of bias in the filter decreases as the effective filter length increases, whereas the random error in the filter decreases with increase in either the signal‐to‐noise ratio of the data or the ratio of the data duration to the filter length. The level of whitening applied to the auto‐correlation matrix before inversion was not found to be a critical design parameter. Also, two techniques have been tested for reducing any anomalous d.c. component in the calculated filter.