Nyquist's problem—The joint optimization of transmitter and receiver in pulse amplitude modulation

Using a criterion of minimum mean square interference, the optimum, linear, generally unrealizable receiver for a noisy pulse amplitude modulation (PAM) system is specified for the case in which a finite number of identically shaped pulses with random, possibly correlated amplitudes are transmitted. The interference is composed of both noise and intersymbol interference. The optimum receiver, using the same criterion, is also obtained for the case in which the intersymbol interference is constrained to be zero. The average error probabilities associated with these receivers are compared with each other and with a matched filter receiver in one example. In addition, formulas for optimum transmitted pulse shapes are derived, and the joint optimization of transmitter and receiver is carried out in detail for the case of signaling through a noisy RC filter. Other formulas enable one to explore the dependence of the minimum mean square interference on the data rate, the noise spectrum, the impulse response of the transmission medium, and the autocorrelation of the message sequence.