Digital Modulation and Receiver Design

Introduction and Context

Wireless communication systems convey information by varying a carrier waveform according to a sequence of digital symbols. The choice of modulation determines spectral efficiency, power efficiency, implementation complexity, resistance to interference, and the requirements imposed on the receiver. Receiver design, in turn, determines how reliably those symbols can be recovered in the presence of thermal noise, fading, interference, oscillator uncertainty, and bandwidth limitations.

Digital modulation is therefore not merely a waveform-generation technique. It is a system-level design decision linking source data rate, channel bandwidth, transmitter power, antenna characteristics, synchronization, signal processing, and quality-of-service requirements. This chapter develops the principles of amplitude-shift keying (ASK), phase-shift keying (PSK), quadrature-amplitude modulation (QAM), and frequency-shift keying (FSK), followed by pulse shaping, matched filtering, coherent and noncoherent detection, maximum-likelihood reception, and error-rate analysis.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Describe signal-space representations of ASK, PSK, QAM, and FSK.

2. Explain the function of pulse-shaping filters and Nyquist criteria.

3. Derive the role of the matched filter in maximizing decision-point signal-to-noise ratio.

4. Distinguish coherent, differentially coherent, and noncoherent detection.

5. Formulate maximum-likelihood decisions for common modulation schemes.

6. Evaluate symbol-error rate (SER) and bit-error rate (BER) under AWGN assumptions.

7. Select a modulation scheme under bandwidth, power, fading, and implementation constraints.

1. Digital Communication Signal Model

A transmitted complex baseband signal can be represented as

\[

s(t)=\sum_{k} a_k p(t-kT_s),

\]

where \(a_k\) is the complex symbol selected from an alphabet of size \(M\), \(p(t)\) is the unit-energy pulse shape, and \(T_s\) is the symbol duration. The gross bit rate is

\[

R_b=R_s\log_2 M,

\]

where \(R_s=1/T_s\) is the symbol rate. This relation assumes that all symbols are equally likely and that no coding or framing overhead is included.

For a passband carrier, the transmitted waveform may be written as

\[

x(t)=\Re\{s(t)e^{j2\pi f_c t}\},

\]

where \(f_c\) is the carrier frequency. In an additive white Gaussian noise (AWGN) channel, the received signal is \(r(t)=s(t)+n(t)\), with noise modeled as a zero-mean Gaussian random process having two-sided power spectral density \(N_0/2\). In wireless channels, this model is often extended to

\[

r(t)=h s(t)+n(t),

\]

where \(h\) is a complex channel coefficient. If \(h\) is known at the receiver, coherent detection can compensate its amplitude and phase; otherwise, channel estimation or noncoherent methods may be required.

A modulation constellation is a set of signal points in signal space. Its minimum Euclidean distance, average energy, peak energy, and geometric symmetry strongly influence error probability and amplifier requirements.

2. Amplitude, Phase, and Frequency Modulation

2.1 Amplitude-shift keying

In binary ASK, the carrier amplitude represents the binary symbol. On-off keying (OOK), a special case, uses

\[

s_1(t)=\sqrt{\frac{2E_b}{T_b}}\cos(2\pi f_c t),\qquad s_0(t)=0,

\]

for \(0\leq t<T_b\), assuming equal-energy normalization for the nonzero signal. ASK has simple transmitters and receivers, but its amplitude information is directly corrupted by fading, shadowing, and amplitude interference. OOK is widely used in low-cost optical links, infrared controls, and low-power short-range systems.

2.2 Phase-shift keying

In binary phase-shift keying (BPSK), the two symbols are antipodal:

\[

s_1(t)=\sqrt{\frac{2E_b}{T_b}}\cos(2\pi f_c t),

\quad

s_0(t)=-\sqrt{\frac{2E_b}{T_b}}\cos(2\pi f_c t).

\]

Their signal-space points are \(+\sqrt{E_b}\) and \(-\sqrt{E_b}\). Antipodal signaling maximizes distance for a given average energy in a binary constellation. In coherent AWGN detection,

\[

P_b=Q\left(\sqrt{\frac{2E_b}{N_0}}\right),

\]

where \(Q(x)=\frac{1}{\sqrt{2\pi}}\int_x^\infty e^{-u^2/2}du\). This result assumes ideal carrier and timing synchronization, equally likely bits, and an AWGN channel.

Quadrature phase-shift keying (QPSK) uses four phases, commonly represented by two independent BPSK components. With Gray labeling and coherent detection, its BER is

\[

P_b=Q\left(\sqrt{\frac{2E_b}{N_0}}\right),

\]

while its symbol error probability is

\[

P_s=1-\left[1-Q\left(\sqrt{\frac{E_s}{N_0}}\right)\right]^2,

\]

where \(E_s=2E_b\) for QPSK. The equality of QPSK and BPSK BER at equal \(E_b/N_0\) illustrates the bandwidth advantage of transmitting two bits per symbol without sacrificing coherent AWGN power efficiency.

Differential PSK encodes information in phase changes rather than absolute phase. It avoids resolving the carrier phase ambiguity but generally incurs an error-performance penalty. Differential detection is particularly useful when rapid phase acquisition is difficult.

2.3 Quadrature-amplitude modulation

QAM varies both in-phase and quadrature amplitudes:

\[

s(t)=I(t)\cos(2\pi f_c t)-Q(t)\sin(2\pi f_c t).

\]

For square \(M\)-QAM, the constellation contains \(\sqrt M\) amplitude levels on each axis. It transmits \(\log_2M\) bits per symbol and provides high spectral efficiency. However, closely spaced points reduce the minimum distance for a fixed average energy, increasing sensitivity to noise, nonlinear amplification, phase noise, and inaccurate channel estimation.

For square Gray-coded QAM in AWGN, a commonly used exact expression for symbol error probability is

\[

P_s=1-\left[1-2\left(1-\frac{1}{\sqrt M}\right)Q\left(\sqrt{\frac{3E_s}{(M-1)N_0}}\right)\right]^2.

\]

At moderate or high signal-to-noise ratio, the approximate BER is

\[

P_b\approx \frac{4}{\log_2M}\left(1-\frac{1}{\sqrt M}\right)Q\left(\sqrt{\frac{3\log_2M}{M-1}\frac{E_b}{N_0}}\right).

\]

The approximation assumes square QAM, Gray coding, equally likely symbols, coherent detection, and AWGN. Practical systems use adaptive modulation: lower-order QPSK or robust QAM under poor channel conditions and higher-order 16-, 64-, or 256-QAM when the channel supports the required error rate.

2.4 Frequency-shift keying

FSK conveys information through carrier frequency. In binary FSK (BFSK), the two signals are

\[

s_i(t)=\sqrt{\frac{2E_b}{T_b}}\cos(2\pi f_i t),\qquad i\in\{0,1\}.

\]

For orthogonal coherent BFSK, the tones satisfy an appropriate integral orthogonality condition over the symbol interval, commonly achieved with frequency separation proportional to \(1/T_b\). The coherent AWGN BER is

\[

P_b=Q\left(\sqrt{\frac{E_b}{N_0}}\right).

\]

For orthogonal noncoherent BFSK,

\[

P_b=\frac{1}{2}\exp\left(-\frac{E_b}{2N_0}\right).

\]

FSK can operate without precise carrier-phase recovery and has a constant-envelope form that is attractive with nonlinear, efficient power amplifiers. Its principal disadvantage is bandwidth consumption, particularly when tone spacing and frequency deviation are large.

3. Pulse Shaping and Bandwidth Control

Ideal rectangular pulses produce a sinc-shaped spectrum with slowly decaying sidelobes. Such spectra cause substantial adjacent-channel interference. Pulse shaping controls occupied bandwidth and intersymbol interference (ISI).

The Nyquist zero-ISI criterion requires the overall impulse response \(g(t)\) to satisfy

\[

g(0)=1,\qquad g(nT_s)=0\quad\text{for nonzero integer }n.

\]

The ideal sinc pulse meets this criterion but is not time-limited and is impractical to realize exactly. The raised-cosine spectrum is widely used. If the roll-off factor is \(\alpha\), with \(0\leq\alpha\leq1\), its approximate one-sided baseband bandwidth is

\[

B=\frac{(1+\alpha)R_s}{2},

\]

and the corresponding passband null-to-null bandwidth is approximately \((1+\alpha)R_s\). A smaller \(\alpha\) improves spectral efficiency but increases sensitivity to timing errors and filter imperfections.

In practice, root-raised-cosine (RRC) filters are divided between transmitter and receiver. If both have the same RRC response, their cascade is raised cosine, satisfying the zero-ISI condition at ideal sampling instants. Excess bandwidth, timing recovery, filter length, and implementation complexity must be selected together.

4. Matched Filtering and Sufficient Statistics

Assume the received signal over one symbol interval is

\[

r(t)=s_m(t)+n(t),

\]

where one of the known signals \(s_m(t)\) was transmitted. The filter matched to \(s_m(t)\) has impulse response, apart from a decision delay \(T\),

\[

h_m(t)=K s_m^*(T-t),

\]

where \(^*\) denotes complex conjugation and \(K\) is an arbitrary scale factor. Sampling its output at \(T\) produces the correlation

\[

y_m=\int_0^T r(t)s_m^*(t)dt.

\]

For AWGN, the matched filter maximizes the output signal-to-noise ratio at the sampling instant. It also produces a sufficient statistic: no information relevant to optimum detection is lost by reducing the continuous-time waveform to the appropriate correlation outputs, provided the signal set and noise model are known.

A practical receiver often uses a matched filter or an equivalent correlator bank, followed by timing synchronization, carrier recovery, channel estimation, equalization, symbol decision, and channel decoding. The filter must be matched to the overall transmitted pulse and channel response, not necessarily to an isolated transmitter pulse.

5. Coherent, Noncoherent, and Maximum-Likelihood Detection

5.1 Coherent detection

Coherent detection uses a local oscillator synchronized in frequency and phase with the received carrier. The receiver projects the waveform onto orthonormal basis functions and selects the nearest constellation point. Coherent detection gives excellent AWGN performance but requires carrier recovery and is sensitive to phase noise and frequency offset.

5.2 Noncoherent detection

Noncoherent detection does not require an absolute carrier phase reference. Energy detectors compare the energy in frequency branches, while differential detectors compare the phase of successive symbols. These receivers are simpler and can be robust during acquisition, but their error probability is generally higher than that of coherent receivers at the same \(E_b/N_0\).

5.3 Maximum-likelihood reception

For equally likely transmitted signals in AWGN, maximum-likelihood (ML) detection selects

\[

\hat m=\arg\min_m \|\mathbf y-\mathbf s_m\|^2,

\]

where \(\mathbf y\) is the received signal-space vector and \(\mathbf s_m\) is the candidate vector. If symbols have unequal prior probabilities, maximum a posteriori detection is more appropriate and includes prior-probability terms.

For a known complex channel coefficient \(h\), ML detection becomes

\[

\hat m=\arg\min_m |y-hs_m|^2.

\]

If \(h\) is unknown, the receiver must estimate it, marginalize over its uncertainty, or use a noncoherent likelihood. In fading channels, ML detection alone cannot remove deep fades; diversity, coding, interleaving, and adaptive transmission are often required.

6. Error Probability and Performance Metrics

The energy-per-bit to noise-density ratio, \(E_b/N_0\), is the principal normalized power-efficiency metric for uncoded systems. Spectral efficiency is often expressed as

\[

\eta=\frac{R_b}{B}\quad\text{bits/s/Hz}.

\]

There is a fundamental trade-off: increasing constellation order generally raises spectral efficiency but decreases minimum point spacing for a fixed average energy.

For Gray-coded constellations, most symbol errors at moderate or high SNR are to neighboring points and usually change one bit. Thus

\[

P_b\approx \frac{P_s}{\log_2M}

\]

may be a useful approximation only under those conditions; it is not a universal identity. Exact BER depends on the labeling and the error-event distribution.

In fading channels, average BER is obtained by averaging the conditional AWGN error probability over the instantaneous SNR distribution:

\[

\bar P_b=\int_0^\infty P_b(\gamma) f_\gamma(\gamma)d\gamma.

\]

Here \(\gamma\) is instantaneous \(E_b/N_0\), and \(f_\gamma\) is its probability density. This averaging explains why fading systems can exhibit error floors or much poorer performance than AWGN predictions.

7. Worked Design Example

A link must transmit 12 Mbit/s using square 16-QAM. Ignoring coding and framing overhead, each symbol carries \(\log_2 16=4\) bits. Therefore,

\[

R_s=\frac{12\times10^6}{4}=3\text{ Mbaud}.

\]

With roll-off factor \(\alpha=0.25\), the approximate passband bandwidth is

\[

B=(1+\alpha)R_s=1.25(3\text{ MHz})=3.75\text{ MHz}.

\]

If the system instead used QPSK, the symbol rate would be 6 Mbaud and the corresponding bandwidth would be 7.5 MHz for the same roll-off. Thus 16-QAM approximately halves the bandwidth but requires a higher SNR and more linear RF hardware. The decision is justified only if the channel provides sufficient link margin and the power amplifier can operate with the required back-off.

8. Applied Case: Adaptive Modulation in a Cellular Downlink

Background

Consider an illustrative cellular downlink serving users at varying distances from a base station. The system supports QPSK, 16-QAM, and 64-QAM, together with forward-error correction and link adaptation.

Problem

A user near the base station has high received SNR and moderate delay spread. Another user at the cell edge experiences low SNR, shadowing, and occasional frequency-selective fading. A single fixed modulation would either waste capacity near the base station or produce excessive retransmissions at the cell edge.

Analysis and approach

The base station estimates channel quality using pilot signals and feedback. When channel quality is high, 64-QAM is selected to provide six uncoded bits per symbol. At intermediate quality, 16-QAM provides four bits per symbol with greater minimum distance. Under poor conditions, QPSK provides two bits per symbol and improved robustness. Coding rate is adapted simultaneously, and interleaving spreads burst errors.

The receiver performs timing recovery, channel estimation, frequency-domain equalization, matched filtering or equivalent correlation, ML or near-ML constellation detection, and decoding. Link adaptation includes a margin because channel estimates are imperfect and feedback is delayed. Hybrid automatic repeat request (HARQ) combines retransmissions when the first transmission is unreliable.

Outcome

The near user obtains high spectral efficiency, while the edge user maintains an acceptable block-error rate through robust modulation, coding, and retransmission. The system achieves greater aggregate throughput than a fixed-modulation design, although it requires feedback, scheduling, calibration, and careful control of latency.

Learning points

Modulation cannot be selected independently of coding, channel estimation, amplifier linearity, and scheduling. The highest-order constellation is beneficial only when its error-rate and implementation requirements are satisfied. Robust design optimizes expected system throughput and reliability rather than peak modulation order.

9. Design Trade-offs, Limitations, and Best Practices

ASK is simple but vulnerable to amplitude variations. PSK offers strong power efficiency, with BPSK especially robust, but requires phase synchronization. QAM provides high spectral efficiency but demands high SNR, linear amplification, and accurate channel estimation. FSK supports noncoherent operation and efficient nonlinear amplification but consumes more bandwidth.

A sound design process should: define required BER or block-error rate; determine available bandwidth and regulatory emission limits; estimate the channel distribution rather than relying only on nominal SNR; select pulse shape and roll-off jointly with timing-recovery capability; account for peak-to-average power ratio; test carrier-frequency offset and phase-noise tolerance; and validate performance using both analytical models and measured or simulated channels.

High-order QAM has a high peak-to-average power ratio, causing distortion when the amplifier is driven near saturation. Digital predistortion, crest-factor reduction, and amplifier back-off can mitigate this problem, but they increase complexity and reduce power efficiency. OFDM-based systems add sensitivity to frequency offset and peak power, while equalization and coding address multipath and fading.

Regulatory requirements include occupied bandwidth, adjacent-channel leakage, spectral masks, and permitted power levels. Security and governance considerations include protecting pilot signals and feedback channels, since falsified channel-quality information can force unsafe modulation choices or enable denial-of-service attacks. AI-assisted link adaptation can predict channel quality and optimize modulation and coding, but it must be evaluated for robustness, explainability, distribution shift, and compliance with reliability requirements. AI should complement, not replace, deterministic safety limits and conformance testing.

Contemporary Developments

Modern systems combine modulation with multiple antennas, beamforming, coding, and adaptive scheduling. Probabilistic constellation shaping adjusts symbol probabilities to approach favorable spectral-efficiency and power-efficiency trade-offs. Generalized frequency division multiplexing and filtered multicarrier methods seek improved spectral containment, while machine-learning methods assist equalization, channel estimation, and modulation classification. Research continues on joint waveform and hardware design, energy-efficient transmitters, integrated sensing and communication, and reliable communications at very high carrier frequencies.

Chapter Summary

Digital modulation maps information symbols onto amplitude, phase, frequency, or combinations of these dimensions. ASK is simple but amplitude-sensitive; PSK is power-efficient and phase-dependent; QAM provides high spectral efficiency at the cost of SNR and linearity requirements; and FSK supports constant-envelope, potentially noncoherent operation with greater bandwidth demand. Pulse shaping controls bandwidth and ISI, with raised-cosine and root-raised-cosine filters being central practical choices. A matched filter maximizes decision-point SNR in AWGN and supplies sufficient statistics for optimum detection. Coherent receivers generally outperform noncoherent receivers but require synchronization. ML detection selects the signal point most consistent with the received observation and channel model. SER and BER depend on constellation geometry, labeling, SNR, synchronization, and channel fading. Practical modulation selection must jointly consider bandwidth, power, fading, amplifier behavior, complexity, regulation, and required reliability.

Key Terms

ASK: Modulation in which carrier amplitude represents the symbol.

BFSK: Binary frequency-shift keying using two carrier frequencies.

BER: Probability that a detected bit differs from the transmitted bit.

Constellation: Geometric set of signal points representing possible symbols.

Coherent detection: Detection using a synchronized carrier reference.

E_b/N_0: Energy per bit divided by noise spectral density.

FSK: Modulation using different carrier frequencies for different symbols.

Gray coding: Labeling in which adjacent constellation points differ by one bit.

ISI: Distortion caused when neighboring pulses interfere at a sampling instant.

Matched filter: Filter matched to a signal waveform to maximize sampled SNR in AWGN.

ML detection: Selection of the signal hypothesis with greatest likelihood.

Noncoherent detection: Detection without an absolute carrier-phase reference.

PSK: Modulation in which carrier phase represents the symbol.

QAM: Modulation that independently varies in-phase and quadrature amplitudes.

Q-function: Gaussian tail-probability function used in error analysis.

Raised-cosine filter: Nyquist pulse-shaping filter characterized by roll-off factor.

SER: Probability that a transmitted symbol is detected incorrectly.

Spectral efficiency: Information rate per unit bandwidth.

Symbol rate: Number of transmitted symbols per second.

Review Questions

Short Answer Questions

1. What is the relationship between bit rate, symbol rate, and constellation size?

2. Why is QAM more sensitive to noise than BPSK at comparable average energy?

3. What function does a matched filter perform in an AWGN receiver?

4. Distinguish coherent and noncoherent detection.

5. What is the purpose of the raised-cosine roll-off factor?

Long Answer Questions

1. Explain the signal-space representations, advantages, and limitations of ASK, PSK, QAM, and FSK.

2. Derive the ML decision rule for equally likely signals in AWGN.

3. Discuss pulse shaping, the Nyquist criterion, and practical use of root-raised-cosine filters.

4. Compare coherent BFSK, noncoherent BFSK, BPSK, and QPSK in terms of receiver complexity and AWGN error performance.

5. Explain how fading, amplifier nonlinearity, and channel estimation affect modulation selection.

Analytical / Application Questions

1. A system must transmit 20 Mbit/s using 64-QAM and a roll-off factor of 0.2. Calculate the symbol rate and approximate passband bandwidth, ignoring overhead.

2. For a link requiring high spectral efficiency but experiencing variable SNR, design an adaptive modulation strategy using QPSK, 16-QAM, and 64-QAM. State the measurements and safeguards required.

3. Compare the suitability of coherent QPSK and noncoherent BFSK for a low-power sensor network with stringent bandwidth and oscillator-cost constraints.

Multiple Choice Questions with Answers

Question 1

For an M-ary modulation scheme carrying equally likely symbols, the uncoded bit rate is

A. R_s/M

B. R_s log2 M

C. M R_s

D. R_s + log2 M

Correct Answer: B. R_s log2 M

Question 2

Which modulation scheme varies both in-phase and quadrature amplitudes?

A. BFSK

B. BPSK

C. QAM

D. OOK

Correct Answer: C. QAM

Question 3

The principal purpose of a matched filter in AWGN is to

A. Eliminate all channel fading

B. Maximize sampled signal-to-noise ratio

C. Increase carrier frequency

D. Generate additional bandwidth

Correct Answer: B. Maximize sampled signal-to-noise ratio

Question 4

For coherent BPSK in AWGN, the BER is

A. Q(sqrt(2Eb/N0))

B. Q(sqrt(Eb/N0))

C. 1/2 exp(-Eb/(2N0))

D. 1/M

Correct Answer: A. Q(sqrt(2Eb/N0))

Question 5

With Gray coding, neighboring QAM constellation points generally differ by

A. Zero bits

B. One bit

C. log2(M) bits

D. M bits

Correct Answer: B. One bit

Question 6

A raised-cosine filter with roll-off factor alpha has approximate passband bandwidth

A. R_s/(1+alpha)

B. (1-alpha)R_s

C. (1+alpha)R_s

D. alpha/R_s

Correct Answer: C. (1+alpha)R_s

Question 7

For equally likely signals in AWGN, ML detection selects the candidate with the

A. Largest frequency

B. Smallest Euclidean distance from the received vector

C. Smallest symbol index

D. Largest peak-to-average ratio

Correct Answer: B. Smallest Euclidean distance from the received vector

Question 8

Which modulation is particularly compatible with nonlinear efficient power amplifiers because it can have a constant envelope?

A. FSK

B. High-order QAM

C. ASK

D. 16-QAM

Correct Answer: A. FSK

Question 9

A 16-QAM system carries how many bits per symbol?

A. 2

B. 4

C. 8

D. 16

Correct Answer: B. 4

Question 10

If a system transmits 12 Mbit/s using QPSK with no overhead, its symbol rate is

A. 3 Mbaud

B. 6 Mbaud

C. 12 Mbaud

D. 24 Mbaud

Correct Answer: B. 6 Mbaud