Parameters setting based on bit error probability formula for an amplitude modulated, baseband, communication system affected by timing jitter and frequency selective channel.

A high data rate wired communication serializer/deserializer system (SerDes) bit error rate performance is subject to two main perturbations, additive thermal noise and timing jitter. Considering only M order Pulse Amplitude Modulation (M-PAM), the modulation order selection results from a trade-off between additive thermal noise and timing jitter. To determine the best modulation order, an analytical expression of the Bit Error Probability (BEP) is derived considering both disturbances. This expression assumes no waveform distortion at sampling, an uncorrelated gaussian timing jitter and a bandlimited Additive White Gaussian Noise (AWGN). Hence, it stands as a benchmark for SerDes performance. Its accuracy is discussed by comparing it to Binary Error Rate (BER) obtained via simulation for various channels. This analysis is extended to a more advanced scenario with a channel taken from the IEEE 802.3df working group for which the strong distortion introduced by the channel is reduced by using different kinds of equalizers.