Sampling in Practice: Best Practices and Gotchas

Chris Lomont, with Claude Fable 5.2

What this article is. The checklist in a set of three articles on the sampling theorem: what to do in practice, and the misunderstandings that are easiest to fall into.

Related articles. Expert notes: Sampling Theory Notes . General-audience summary: Sampling Theory Notes Summary . Companion topics: image resampling, dithering and quality metrics (summary , checklist ); color spaces and HDR (summary , checklist ).

Contents

• Decision table
• Best practices by task
• Common errors and gotchas
• Three questions for any “beyond Nyquist” claim

Section and figure numbers refer to the expert notes.

Decision table

Situation	Do this	Why
Choosing a sampling rate	Set it from the error you can tolerate and how fast the spectrum decays, then add a guard band	Nothing real is band-limited; there is no rate at which the error switches off. §2.3
Signal content close to half the sampling rate	Raise the rate or accept a long reconstruction filter	The filter length grows as the guard band shrinks. §3.3
Reducing word length	Add triangular dither two steps wide before the quantizer	It makes the error’s mean and variance independent of the signal. §4.3
High input frequency	Budget clock jitter before bit depth	Jitter caps signal-to-noise ratio by input frequency, whatever the nominal resolution. §5.1
Any non-linear operation on samples	Oversample around it, then filter and decimate	Non-linearity makes new frequencies, and they alias. §5.4
Sampling a narrow band at a high carrier	Undersample at an allowed rate, away from the wedge tips, with a band-pass filter in front	The minimum rate is fragile, and noise from every folded zone lands in band. §6.3
Irregular sample times	Check conditioning, not only the average rate	The same density can be numerically useless. §7.3
A method that claims to beat the Nyquist rate	Ask the three audit questions at the end of this page	The gain comes from a signal model, and the model has a failure mode. §10.5

Best practices by task

Acquire

1. Treat “twice the highest frequency” as a lower bound that is never met exactly. Sampling a sine at exactly half the sampling rate can return all zeros. §3.1
2. Size the anti-alias filter as a system decision. A gentle analog filter with heavy oversampling and a sharp digital filter is the standard answer. §6.2
3. Remember that the anti-alias filter is also a noise filter. Every bit of front-end bandwidth beyond the signal folds noise into the band. §6.1
4. Compute three ceilings for any converter: quantization, jitter and thermal noise. The lowest is a first estimate of the result; the noise powers add, so the true figure is a little below it. §6.1
5. Specify clocks by the input frequency. Undersampling a 1 GHz carrier at 100 MS/s still needs a 1 GHz-quality clock. §5.1

Quantize

6. Dither whenever the signal can sit near a few quantization steps. Without it the error is distortion locked to the signal, not noise. §4.2
7. Use the full-scale-sine formula only for full-scale sines. $6.02N+1.76$ dB overstates the signal-to-noise ratio of real program material with headroom. §12
8. Count bits, not samples. Rate and resolution trade against each other, from half a bit per octave of oversampling for plain conversion to much more with noise shaping. §4.4

Reconstruct and display

9. Do not judge peak level or waveform shape from raw samples. Straight lines between samples are the wrong reconstruction; a tone near half the sampling rate looks amplitude-modulated and is not. §12
10. Leave headroom for intersample peaks, or measure true peak on an oversampled signal. Samples that never clip can reconstruct to a waveform that does. §3.4
11. Prefer a windowed or smooth-transition kernel with a guard band to a truncated sinc. A truncated sinc is accurate nowhere. §3.3
12. Expect ringing from any sharp filter. It is the price of the sharp cutoff, and it is worse the sharper the filter. §5.3

Images and video

13. Prefilter before reducing an image. Dropping pixels, or bilinear interpolation without a prefilter, produces moiré on fine regular texture. §2.4
14. View test patterns pixel for pixel. A viewer that rescales adds its own aliasing. §12
15. Expect false color from a color-filter sensor on fine gray detail. Red and blue are sampled at half the density of the full grid. §12
16. Treat frame rate and rolling shutter as sampling in time. Wagon wheels and bent propellers are aliases. §12

Beyond band-limiting

17. Match the model to the signal. Splines, wavelets, sparsity and parametric models each have their own sampling theorem and their own form of aliasing. §8 to §10
18. Check what happens when the model is slightly wrong. Compressed sensing that recovers eight on-grid tones fails when the tones move off the grid by 2.5% of a bin. §9.3
19. Ask for the condition number. Gap filling, zero-crossing reconstruction, super-resolution of close sources, high-order modulo unwrapping and phase retrieval are all exact in theory and unstable in practice. §12

Common errors and gotchas

The theorem

• “Sample at twice the highest frequency” is not enough. The theorem needs strictly more than two samples per cycle. §3.1
• Sampling is not quantization. The theorem makes time discrete, not amplitude. §12
• “Nyquist rate” and “Nyquist frequency” are used inconsistently. One is a property of the signal, the other of the system. Define both on first use. §12.1
• The textbooks disagree about the history and about whether the inequality is strict. §12.1

Audio and converters

• Digital audio is not a staircase. The steps are inside the converter, before its filter. More bits lower the noise floor; they do not make “smaller steps”. §5.2
• Adding noise removes distortion. A sine smaller than one step vanishes without dither and is recoverable with it. §4.2
• A 16-bit converter with 1 ps of clock jitter is a 10-bit converter at 100 MHz. §5.1
• Band-limited in is not band-limited out. Clipping, rectifying, multiplying and compressing all make new frequencies. §5.4
• Aliasing folds noise too. A converter with 1 GHz of input bandwidth clocked at 100 MS/s collects ten zones of noise. §6.1
• Leakage is not aliasing. A tone that does not fit the analysis window spreads across bins; a window fixes it, a higher sampling rate does not. §12

Reconstruction

• The sinc is not the practical ideal. It decays too slowly, and windowed sincs do not even reproduce a constant exactly. §12.1
• “Cubic B-splines blur” is an implementation error: the samples were used as coefficients without the prefilter. §12.1
• Kernels that pass through the samples are not optimal. Dropping that requirement and adding a digital filter gains an order of accuracy at the same cost. §8.2

Irregular and exotic sampling

• Irregular sampling has no simple Nyquist frequency, and random sampling turns aliases into noise. §7.4
• “We have enough samples” is a statement about counting, not conditioning. §7.3
• Band-limited functions can wiggle arbitrarily fast, at an exponential cost in amplitude. Any scheme that promises resolution beyond the band limit from noisy data pays that cost somewhere. §7.6
• Sub-Nyquist is not sub-information. The measurement count tracks the degrees of freedom of the assumed model. §10.5
• Aliasing is sometimes the feature: multi-frame super-resolution needs aliased frames, and a slowly sampled gyroscope can pick up speech through it. §12.1

Three questions for any “beyond Nyquist” claim

1. How many degrees of freedom does the assumed model have? The number of measurements cannot be smaller.
2. What is the condition number of the recovery? Very small eigenvalues are behind every result that looks too good.
3. What happens when the model is slightly wrong? Band-limited sampling fails gently. Most of its competitors do not.