Image Processing Notes
Published
How pictures should be resized, rotated and dithered, and how to judge the result. Three articles: expert notes with the mathematics, measurements and references; a plain-language summary ; and a checklist of best practices and gotchas .
Image Sampling and Resampling: Kernels, Brightness Scales, Dither, and the Problem of Judging the Result
Chris Lomont, with Claude Fable 5.2
What this article is. The expert-level article in a pair on image resampling. It surveys reconstruction kernels and generalized sampling, the question of linear light against perceptual encoding, quantization and dithering, and how results are judged, with worked numerical examples and full references. It assumes Fourier analysis and basic signal processing.
Related articles. General-audience version: Resizing, Rotating and Dithering Pictures . Checklist: Best Practices and Gotchas . Companion topics: sampling theory and where it stops (summary ); color spaces and HDR (summary ).
Introduction
Here are notes I co-developed with Claude Fable 5.2 in October 2026 on how images should be resized, rotated, warped and dithered, and how to judge the result. They draw on about 250 papers and books, including very recent results. Public PDFs are linked where possible; other references point to their sources.
I also wanted to test the claims, so every figure is a measured example. I address some common points of contention, such as “is a pixel a point or a square or a X”, with a decent way to think about it: a pixel is a coefficient, not a little square, and pairing a kernel with a small digital filter (generalized sampling) beats the classic kernels at the same cost. I also address something I have wanted to chase down for some time: the comment in a paper by Nehab and Hoppe on doing certain operations in linear light and others on gamma-compressed values, a statement for which other sources and details are hard to find.
Sections on dithering, on quality metrics and what they miss, and on designing for perception close the notes, somewhat biased by my personal notes, paper collection, and interests.
I also made a simpler summary of these notes and a list of best practices and gotchas .
Contents
- Introduction
- 0. The short version
- 1. The classical picture and where it fails for images
- 2. Generalized sampling
- 3. What is a pixel?
- 4. The kernel catalog
- 5. Operations
- 6. Linear light versus perceptual encoding
- 6.1 The passage
- 6.2 The terms
- 6.3 The prefilter half: averaging is something light does
- 6.4 The reconstruction half: interpolation is a guess, judged by eye
- 6.5 Coverage and text: the prefilter rule in rasterization
- 6.6 Both stages together
- 6.7 Other practical consequences
- 6.8 Where the advice is debatable or incomplete
- 6.9 A working rule
- 7. Quantization, dithering and noise shaping
- 7.1 What quantization error is
- 7.2 Banding and bit depth, in linear and encoded values
- 7.3 Dither for multi-bit quantization
- 7.4 Thresholding against a pattern: ordered dither, noise functions, blue-noise masks
- 7.5 What “blue noise” means
- 7.6 Error diffusion
- 7.7 Error diffusion as two-dimensional noise shaping
- 7.8 Model-based and optimization-based halftoning
- 7.9 Dithering in linear versus encoded values
- 7.10 Color, palettes and e-paper
- 7.11 Dithering over time
- 8. Judging results: metrics and perception
- 8.1 PSNR
- 8.2 SSIM
- 8.3 Measured failure cases
- 8.4 Relatives of SSIM and other hand-designed metrics
- 8.5 Deep-feature metrics
- 8.6 Vision-model metrics, and why viewing conditions are not optional
- 8.7 A comparison table
- 8.8 What the datasets establish, and the perception–distortion trade-off
- 8.9 Known, unknown, and commonly misstated
- 9. Designing for perception instead of signal error
- 10. Recent research, 2018–2026
- 11. Lessons from implementing and testing these methods
- 12. Conclusions and open questions
- References
Conventions. Unit sample spacing unless a period $T$ is written. $\mathrm{sinc}(x)=\sin(\pi x)/(\pi x)$. $\hat f(\omega)=\int f(x)e^{-i\omega x}dx$, $\omega$ in radians per sample, $f=\omega/2\pi$ in cycles per sample, Nyquist frequency
$f=\tfrac12$. $\beta^n$ is the centered B-spline
of degree $n$. Following Nehab and Hoppe, $[\![f]\!]$ is the sequence of samples $f(k)$, $\mathbf c{\ast}\varphi=\sum_k c_k\varphi(\cdot-k)$ is a mixed (discrete with continuous) convolution
, and $f^\vee(x)=f(-x)$. A kernel has support $W$ (width in samples), approximation order $L$ (error falls as $T^L$) and degree $N$. “Linear” or “linear light” means values proportional to luminance; “encoded” or “code value” means values after the sRGB
transfer function (often loosely called gamma); $L^{\ast}$ is CIELAB
lightness. “ppd” is pixels per degree of visual angle. Citations are [Author Year]; the reference list is at the end.
Status of claims. Statements are of three kinds, and the text keeps them apart.
- Cited results, with the source and, where it matters, the page. Items that rest on an abstract, a secondary source or memory carry a dagger † here and in the reference list. Where two sources disagree, both values are given.
- Simulations made for this article (all 26 figures [Code]). They are evidence about specific test images, not general laws. Several came out differently from the textbook expectation and are reported as measured.
- Reasoning made for this article, marked “reading”, “derivation” or “synthesis”. Section 9 is mostly of this kind.
0. The short version
| Question | Conclusion | Evidence | Section |
|---|---|---|---|
| Is there a best kernel? | No. There is a best kernel for a stated criterion, signal class and cost. For repeated resampling of well-sampled images under squared error, prefiltered (“cardinal”) cubic O-MOMS and B-splines win clearly over Keys, Mitchell and bilinear at the same four taps. | [Thévenaz 2000a] Table IV; [Nehab 2014] Table 10.1; Figure 8 | 4, 5 |
| Why do four taps beat four taps? | A digital correction filter (the “prefilter” of spline interpolation) costs almost nothing and removes the requirement that the kernel itself interpolate. That requirement costs one approximation order or a large constant. | Figure 2; [Blu 2001] | 2 |
| Does the theory rank kernels correctly? | For smooth, alias-free content and small errors, yes. For a single enlargement the kernels are nearly indistinguishable by any metric; on sources that already contain aliasing , bilinear can score a higher PSNR than Lanczos or O-MOMS while looking worse. | Figures 8, 9, 25; [Getreuer 2011] | 4.7 |
| How should an image be reduced? | Widen the prefilter with the scale factor, and apply it to linear-light values. Many library defaults do neither. | Figures 5, 11; [Parmar 2022] | 5.1, 6 |
| What did Nehab and Hoppe mean about color spaces? | Prefiltering is a physical average of light and must be done on linear intensities. Reconstruction is a guess at an unknown function, to be judged by eye; done on perceptually encoded values, the guess between two samples looks midway and ringing is the same visual size on dark and light sides. | [Nehab 2014] §8.3, pp. 50–51; Figures 11–15 | 6 |
| Is that advice a law? | The prefilter half is close to a theorem. The reconstruction half is a heuristic that holds when the reconstructed transitions are resolved by the eye, and it conflicts with the common “always linear” rule. | Section 6.8 | 6 |
| What is a pixel? | A coefficient of a basis function. “Point sample”, “little square” and “spot” are three choices of basis; each is right for one device or one operation. | Figure 3 | 3 |
| What does dither do? | It makes the quantization error independent of the signal (triangular dither of two steps fixes mean and variance); shaping then moves the error to frequencies where the eye is less sensitive. Shaping cannot reduce the total; it raises it. | Figures 17, 20, 21; [Christou 2008] | 7 |
| In which space should one dither? | Sub-step dither for bit-depth reduction: in the space where the quantizer acts. Halftoning to few levels: the error must be carried in linear light, because the eye averages light. | Figures 22, 23; [Kolpatzik 1992] | 7.9 |
| Are PSNR and SSIM perceptual? | No. Neither knows the viewing distance, the display or the encoding. On the BAPPS two-alternative test SSIM agrees with people 63.1% of the time and plain $L_2$ 63.2%. | [Zhang 2018]; Figures 24, 25 | 8 |
| What is better? | Vision-model metrics that take pixels per degree and luminance (ColorVideoVDP, FovVideoVDP, HDR-VDP-3, FLIP) for visibility questions; DISTS or LPIPS for texture-tolerant similarity, with known weaknesses. No metric is safe to optimize blindly. | Section 8 | 8 |
| Is there a sampling theory built for perception? | Pieces exist (display-and-eye-referred prefilters, model-based halftoning, perceptual quantization curves, the perception–distortion theorems). A unified theory does not, as far as the material gathered here shows. | Section 9 | 9 |
Two themes run through the report.
-
Every stage is a projection onto, or an estimate within, some space, and the space matters more than the kernel’s name. Downscaling projects onto a coarser space; upscaling estimates a function from coefficients; the digital filter in between changes basis. Errors come from using the coefficients of one basis as if they belonged to another.
-
Physical operations and perceptual judgments live on different brightness scales. Light adds linearly, so anything that models light being averaged (prefilters, blur, coverage, compositing, halftone dots fusing) belongs in linear light. Anything that models a viewer’s judgment of difference (interpolation overshoot, quantization step size, error metrics) is closer to uniform on an encoded scale. Most gotchas in Section 6 and many in Sections 7 and 8 are cases of doing one on the other’s scale.
1. The classical picture and where it fails for images
1.1 Fourier and Nyquist in two dimensions
Sampling $f(x,y)$ on the unit square lattice replicates its spectrum on the reciprocal lattice: $\hat f_s(\boldsymbol\omega)=\sum_{\mathbf k\in\mathbb Z^2}\hat f(\boldsymbol\omega+2\pi\mathbf k)$. If $\hat f$ vanishes outside the square $|f_x|,|f_y|<\tfrac12$ cycles per sample, the replicas do not overlap and $f$ is recovered by the separable kernel $\mathrm{sinc}(x)\,\mathrm{sinc}(y)$ [Heckbert 1989, p. 47]. Three two-dimensional facts have no one-dimensional counterpart.
- The alias-free region of a square lattice is a square, so the highest representable radial frequency depends on direction: $\tfrac12$ along the axes and $\tfrac{\sqrt2}{2}$ along the diagonal [Lau 2008, pp. 49–50]. The eye’s sensitivity runs the other way (lower on diagonals; the angular model of Daly quoted in [Lau 2008, pp. 68–69]), which is the usual explanation for setting halftone screens at 45 degrees (standard lore, not checked for this article †).
- For an isotropically band-limited signal a hexagonal lattice needs fewer samples than a square one (Petersen and Middleton 1962 †, not obtained; hex-spline versions in [Van De Ville 2004; Condat 2007], not summarized here).
- Separable kernels are not isotropic. Of the common ones only the Gaussian is both separable and circularly symmetric [Heckbert 1989, p. 47]. Anisotropy shows as diagonal edges treated differently from axis-aligned ones.
The classical pipeline is prefilter, sample, reconstruct:
$$f\;\longrightarrow\;f{\ast}\psi^\vee\;\longrightarrow\;[\![f{\ast}\psi^\vee]\!]\;\longrightarrow\;\tilde f=[\![f{\ast}\psi^\vee]\!]{\ast}\varphi ,$$with $\psi$ the analysis filter (prefilter, antialiasing filter, sampling kernel) and $\varphi$ the reconstruction kernel [Nehab 2014, pp. 2–3]. In the Shannon model both are the sinc. Mitchell and Netravali named the two kinds of failure: prealiasing, when the prefilter leaves energy above the Nyquist frequency and sampling folds it down, and postaliasing, when the reconstruction filter passes the spectral replicas [Mitchell 1988].
1.2 Why images are not band-limited
An image of a scene with occluding edges contains step discontinuities, whose amplitude spectrum falls only as $1/\omega$. Measured power spectra of natural images fall as $\omega^{-p}$ with $p$ between 1.6 and 3.0 [Sacht 2014, eq. 22]: a power law has no cutoff. An image is also of finite extent, and no function of bounded support is band-limited (the companion article on sampling theory , Section 2 , gives the degrees-of-freedom version of this statement and the rates at which aliasing error decays with smoothness).
Nor does the capture chain make images band-limited. Figure 3 (Section 3 ) computes the modulation transfer of a diffraction-limited f/5.6 lens and a 4 µm pixel with 100% fill: the product still passes 0.332 of the contrast at the Nyquist frequency and 0.092 at one and a half times Nyquist. Only an optical low-pass filter brings the response at Nyquist to zero, and then at the cost of 0.482 instead of 0.681 at half Nyquist. Synthetic images are worse: a renderer that takes one sample per pixel has no prefilter at all, and a box prefilter (area coverage) is a poor low-pass filter. Getreuer makes the point with a medical example: “Antialias filtering is not possible on physical matter, and any attempt to do so would probably be harmful to the patient” [Getreuer 2011].
So every practical image is a set of samples of a function that was not band-limited, taken through a prefilter that was not ideal. The useful question is no longer “was the Nyquist condition met?” but “what is the error, and which error do we mind?” PBRT states the shift directly: “the goal of research in reconstruction theory has shifted from perfect reconstruction to developing reconstruction techniques that can be shown to minimize error between the reconstructed function and the original function, regardless of whether the original was band-limited” [PBRT, §7.6, citing Unser 2000].
1.3 The trade among blur, ringing, aliasing and anisotropy
Mitchell and Netravali’s 1988 paper remains the clearest statement that filter design for images is a compromise among visible defects, settled by observers. Their two-parameter family of cubics (formula in Section 4.2 ) was rated by nine expert observers over more than 500 samples, and the $(B,C)$ plane was divided into regions of blur, ringing, anisotropy and “satisfactory”; the recommendation $B=C=\tfrac13$ lies in the last [Mitchell 1988]. The defects are:
- Blur: attenuation inside the pass band. A kernel with no negative lobes must blur.
- Ringing: oscillation near edges, caused by negative lobes. They quote Brown’s finding that “a single transient lobe of ringing was effective at sharpening, but multiple transients … always degrade image quality”, and they note an asymmetry that returns in Section 6 : where an undershoot is clamped at black it “results in pronounced black spots”, while “similar clamping occurs to white, but is less noticeable because of the eye’s nonlinear response to contrast” [Mitchell 1988, p. 223].
- Aliasing: pre- and postaliasing as above. Postaliasing at the sample frequency itself appears as a ripple on flat areas unless the kernel satisfies the partition of unity, $\sum_n k(x-n)=1$ (their constraint 7).
- Anisotropy: a separable kernel responding differently to different orientations.
Nehab and Hoppe add positivity to the list and observe that “the perceptual quality of an approximation involves a subjective balance between aliasing, blurring, ringing, and positivity, to which the $L_2$ norm is oblivious” [Nehab 2014, p. 11, crediting Kajiya and Ullner 1981].
Figure 1. Ten reconstruction kernels (top) and their frequency responses in dB (bottom) [Code]. Top right: the sinc and the equivalent interpolating kernels of the prefiltered cubic B-spline and cubic O-MOMS, with the bare bases dotted. Measured gain at the Nyquist frequency and largest stop-band response beyond 0.75 cycles per sample: box −3.92 and −10.5 dB; tent −7.84 and −20.9; Keys −6.15 and −24.1; Mitchell −8.33 and −27.1; Lanczos-3 −5.99 and −38.9; cardinal cubic B-spline −6.15 and −38.3; cardinal cubic O-MOMS −5.55 and −40.4; the bare cubic B-spline without its prefilter −15.69 and −41.8. At a quarter cycle per sample, in the middle of the pass band, the tent has already lost 1.82 dB, Mitchell 1.46 dB and the bare B-spline 3.65 dB, against 0.13 dB for the cardinal B-spline and 0.01 dB for the cardinal O-MOMS. The deepest negative lobe is −0.036 for Mitchell, −0.074 for Keys, −0.147 for Lanczos-3, −0.137 and −0.167 for the two cardinal splines, and −0.217 for the sinc. Lanczos is drawn unnormalized here; in every resampling figure its weights are renormalized to unit sum.
Figure 1 puts numbers on the trade. Pass-band flatness and stop-band rejection improve together only when negative lobes deepen: the kernels that hold the pass band to within a fraction of a decibel are the ones whose first negative lobe is 14 to 17% of the peak, and they overshoot a step by 11 to 12% (Figure 6).
1.4 Why the sinc is not the practical ideal
The sinc is optimal for one criterion (squared error) on one signal class (band-limited). Away from that class it fails in ways that are specific and visible.
- Slow decay. Its envelope falls as $1/x$, so every sample influences every output point, and “windowing causes significant degradation” [Heckbert 1989, p. 41].
- Gibbs oscillation on real data. “The perfect reconstruction filter causes artifacts when applied to aliased samples: ringing artifacts appear in the reconstructed function” [PBRT, pp. 293–294]. Getreuer measures the consequence for a fourfold enlargement: on a smooth chirp the sinc has the lowest RMSE of fifteen methods (27.2 against 47.2 for bilinear), and on a piecewise-smooth image it has the highest of all but nearest neighbor (26.3 against 24.5 for bicubic ) [Getreuer 2011, p. 256].
- Negative light. Its lobes are deepest of all (−21.7% of the peak). Intensities cannot be negative, so the output must be clamped, and the clamp is not symmetric to the eye (Section 6 ).
- Truncation breaks the partition of unity. “The use of windowed or truncated sinc functions is not recommended because these fail to satisfy the partition of unity” [Unser 2000, pp. 575–576]. A windowed sinc does not reproduce a constant exactly, so a flat field acquires ripple, and under repeated application the mean drifts. In the repeated-rotation test of Thévenaz, Blu and Unser a truncated sinc of support 4 ends at −4.04 dB SNR on a CT image where Keys reaches 38.74 dB [Thévenaz 2000a, Table IV]. Renormalizing the weights at every output sample repairs the constant and makes Lanczos competitive (Section 4.7 ), which is why protocols that normalize and protocols that do not report opposite conclusions about it.
- No device reconstructs with it. A display emits light through its own pixel shape and the eye blurs the result. PBRT says of its own assumption of ideal reconstruction: “This assumption is patently at odds with how actual displays work” [PBRT, p. 280, footnote 1]. Rocha, Oliveira and Gastal add that the standard “low-pass then sample” recipe assumes a sinc reconstruction, while the real one is display plus eye, “most definitely not a sinc interpolation” [Rocha 2020, p. 3].
Unser’s 2000 survey lists the same three objections to Shannon’s theorem as a practical tool: it is an idealization, the ideal filter does not exist, and the sinc is rarely used because of its slow decay [Unser 2000, p. 569]. The response, developed through the 1990s, is the subject of the next section.
2. Generalized sampling
2.1 Shift-invariant spaces
Fix a generating function $\varphi$ and consider every function that can be written as a combination of its integer shifts:
$$V_\varphi=\Big\{\tilde f=\sum_k c_k\,\varphi(\cdot-k)\Big\}=\{\mathbf c{\ast}\varphi\}.$$With $\varphi=\mathrm{sinc}$ this is the band-limited space, and the classical theorem is the statement that ideal prefiltering is the orthogonal projection onto it, the sinc shifts being orthonormal [Unser 2000, p. 571]. With $\varphi=\beta^0$ it is the space of piecewise-constant functions, with $\varphi=\beta^3$ the cubic splines. The image array is the coefficient sequence $\mathbf c$, and three questions separate cleanly: which space, how to choose the coefficients, and how good the best element of the space can be.
Three equivalent generators of the same space are worth naming [Nehab 2014, Table 3.1 and p. 17]. With $a_\varphi=\varphi{\ast}\varphi^\vee$ the autocorrelation:
- the cardinal (interpolating) generator $\varphi_{\rm int}=[\![\varphi]\!]^{-1}{\ast}\varphi$, whose coefficients are the samples of $\tilde f$;
- the dual generator $\mathring\varphi=[\![a_\varphi]\!]^{-1}{\ast}\varphi$, against which inner products give least-squares coefficients;
- the orthogonal generator $[\![a_\varphi]\!]^{-1/2}{\ast}\varphi$.
Each is the original $\varphi$ convolved with a digital filter. They span the same space; only the meaning of the coefficients changes.
2.2 Approximation order and the Strang–Fix conditions
Scale the lattice to spacing $T$. The space has approximation order $L$ if the best approximation error falls as $T^L$ for smooth $f$:
$$\|f-P_Tf\|= C_\varphi\,T^L\,\|f^{(L)}\|\quad\text{as }T\to0 \qquad\text{[Unser 2000, eq. 45]}.$$The Strang–Fix conditions say this holds exactly when $V_\varphi$ contains all polynomials of degree less than $L$; equivalently $\hat\varphi$ has zeros of order $L$ at every nonzero multiple of $2\pi$ [Nehab 2014, §9]. Order 1 is the partition of unity. A piecewise polynomial of degree $N$ can reach at most $L=N+1$, and needs support at least $W=N+1$ to do it; the kernels that achieve both are the MOMS family (maximal order, minimal support), all of the form $\varphi=\sum_n\lambda_n D^n\beta^{L-1}$ [Blu 2001, Theorem 1]. The B-spline is the smoothest member, the O-MOMS the one with the smallest constant $C_\varphi$ (Section 4.5 ).
Order describes the limit $T\to0$, which for images means content far below the Nyquist frequency. Sacht and Nehab are explicit about the limits of that view: “In practice, this will be the dominant effect only when we are able to reduce $T$ arbitrarily, or when the input signal has a narrow band around zero” [Sacht 2014, p. 3]. Their optimized kernels have order 1 by design and beat order-6 splines on their tests (Section 4.6).
2.3 The error kernel
Blu and Unser’s error kernel gives the whole frequency picture. For a prefilter $\tilde\varphi$ (including any digital filter) and reconstruction kernel $\varphi$, the squared error averaged over all sub-sample shifts of the signal is
$$\overline{\|f-\tilde f\|^2}=\frac1{2\pi}\int|\hat f(\omega)|^2\,E(T\omega)\,d\omega,\qquad E(\omega)=\big|1-\hat{\tilde\varphi}(\omega)^{\ast}\hat\varphi(\omega)\big|^2+|\hat{\tilde\varphi}(\omega)|^2\sum_{k\ne0}|\hat\varphi(\omega+2\pi k)|^2$$[Unser 2000, eqs. 41–44]. The first term is in-band distortion (blur, or its opposite), the second is postaliasing. The actual error for a particular alignment differs from the average by a correction that vanishes when $f$ is band-limited to the Nyquist frequency [Unser 2000, Theorem 3]. Two special cases:
- Interpolation ($\tilde\varphi=\delta$ followed by the digital filter $[\![\varphi]\!]^{-1}$): $E_{\rm int}(\omega)=\dfrac{\big|\sum_{n\ne0}\hat\varphi(\omega+2\pi n)\big|^2+\sum_{n\ne0}|\hat\varphi(\omega+2\pi n)|^2}{\big|\sum_n\hat\varphi(\omega+2\pi n)\big|^2}$ [Getreuer 2011, p. 241, after Blu and Unser 1999].
- Orthogonal projection: $E_{\rm opt}(\omega)=1-\dfrac{|\hat\varphi(\omega)|^2}{\sum_k|\hat\varphi(\omega+2\pi k)|^2}$, the smallest possible for the space [Thévenaz 2000a, p. 746].
Sacht and Nehab write the general kernel as the optimal one plus a penalty for not using the dual: $E=E_{\rm opt}+\hat a_\varphi\,|\hat q\hat\psi-\hat\varphi/\hat a_\varphi|^2$ [Sacht 2014, Theorem 1]. Near $\omega=0$, $\sqrt{E(\omega)}\approx C\omega^L$, which ties the kernel to the order and constant of Section 2.2 .
Figure 2. The Blu–Unser interpolation error kernel $E(\omega)$ for ten kernels, log–log (left) and linear (right) [Code]. Measured low-frequency slope of $\sqrt E$ and constant $C$ in $\sqrt E\approx C\omega^L$ ($\omega$ in radians per sample): box, slope 1.00, $C=0.2887$; tent, 2.00, $0.09129$; Mitchell, 2.00, $0.05556$; cubic B-spline applied without its prefilter, 2.00, $0.1667$; Keys, 3.01, $0.01151$; cardinal cubic B-spline, 4.00, $1.660\times10^{-3}$; cardinal cubic O-MOMS, 4.00, $6.27\times10^{-4}$. At half the Nyquist frequency ($\omega/\pi=0.5$) the error is $4.6\times10^{-2}$ for the tent, $2.6\times10^{-2}$ for Mitchell, $7.7\times10^{-3}$ for Keys, $3.6\times10^{-4}$ for the cardinal B-spline and $4.1\times10^{-5}$ for the cardinal O-MOMS. Unnormalized Lanczos-2 and Lanczos-3 and the Gaussian have no order: $\sqrt E$ tends to a floor of $1.19\times10^{-2}$, $3.6\times10^{-3}$ and $1.02\times10^{-2}$ as $\omega\to0$ because they do not reproduce constants. At the Nyquist frequency every kernel has $E\approx0.5$ or worse.
The measured constants in Figure 2 agree with the published ones where those exist: $0.09129$ for the tent is Unser’s $C_{\rm int}=9.129\times10^{-2}=1/\sqrt{120}$ [Unser 2000, pp. 578–579], and $0.01151$ for Keys is $\sqrt{1/7560}=0.0115$ [Thévenaz 2000a, Table II]. Three further readings of the figure are this article’s own and were checked against the measured values. (i) The constant of an order-2 kernel that is not interpolating is set by its blur: $1/6$ for the bare cubic B-spline (from $\hat\beta^3\approx1-\omega^2/6$) and $B/6=1/18$ for Mitchell at $B=\tfrac13$. (ii) For the cardinal cubic B-spline, $C_{\rm int}^2=\big(2\zeta(4)/(2\pi)^4\big)^2+2\zeta(8)/(2\pi)^8$, which gives $1.660\times10^{-3}$; the second term alone is the projection constant $C_{\rm opt}=9.09\times10^{-4}$, the value of the formula $\sqrt{2\zeta(2L)}/(2\pi)^L$ in [Blu 2001]. Interpolating instead of projecting therefore costs a factor 1.83 in the constant, at order 4. (iii) The published advantage of cubic O-MOMS over the cubic B-spline, a factor $\sqrt{21}\approx4.6$ in the constant [Blu 2001, Table II], is for projection. For interpolation the measured ratio is 2.65, that is 8.5 dB instead of 13 dB in the asymptotic regime.
2.4 Three ways to choose the coefficients
Nehab and Hoppe reduce the choice to one table [Nehab 2014, Table 4.1, p. 19]:
| Approach | Prefilter $\psi$ | Reconstruction | Digital filter $q$ |
|---|---|---|---|
| Interpolation | $\delta$ (point samples) | $\varphi$ | $[\![\varphi]\!]^{-1}$ |
| Orthogonal projection | $\varphi$ | $\varphi$ | $[\![a_\varphi]\!]^{-1}$ |
| Oblique projection | any $\psi$ | $\varphi$ | $[\![\varphi{\ast}\psi^\vee]\!]^{-1}$ |
The generalized pipeline inserts $q$ between sampling and reconstruction: $\mathbf c=[\![f{\ast}\psi^\vee]\!]{\ast}q$, $\tilde f=\mathbf c{\ast}\varphi$. Splitting $q=p^\vee{\ast}r$ shows what it does: it turns the compact filters $\psi$ and $\varphi$ into the effective prefilter $p{\ast}\psi$ and effective reconstruction kernel $r{\ast}\varphi$, which have infinite support and a much better frequency response, at the price of a recursive filter on the sample grid [Nehab 2014, p. 5].
Interpolation asks that $\tilde f$ pass through the samples. If $\varphi$ is itself interpolating ($[\![\varphi]\!]=\boldsymbol\delta$: box, tent, Keys, Lanczos) no filter is needed. If it is not (B-splines of degree above 1, O-MOMS), $q=[\![\varphi]\!]^{-1}$ is the classical spline prefilter. For the cubic B-spline, $[\![\beta^3]\!]=\tfrac16[1,4,1]$ and the inverse is one causal and one anticausal first-order recursion with pole $z_1=\sqrt3-2$:
$$c^+_k=6f_k+z_1c^+_{k-1},\qquad c_k=z_1\,(c_{k+1}-c^+_k)$$[Getreuer 2011; Ruijters 2012]. Nehab and Hoppe’s equivalent LU form needs eight precomputed factors to converge in single precision, costs “3n products and 2n additions”, and ran at 800 megapixels per second on a 2010 desktop processor [Nehab 2014, pp. 20–22]. Section 4.3 gives the poles for other degrees.
Orthogonal projection minimizes $\|f-\tilde f\|_2$ over the space. It needs the prefilter to be the dual of the reconstruction kernel, so it applies when a finer or continuous version of $f$ is available: downscaling, rendering.
Oblique projection accepts whatever prefilter the device or the budget supplies and corrects with $q$ so that the result is consistent: resampling $\tilde f$ with the same $\psi$ would return the same measurements. Its error is bounded by that of the orthogonal projection divided by the cosine of the angle between the two spaces,
$$\|f-P_\varphi f\|\;\le\;\|f-P_{\varphi\perp\psi}f\|\;\le\;\frac1{\cos\theta_{\psi,\varphi}}\,\|f-P_\varphi f\|$$[Unser 2000, Theorem 2; Nehab 2014, eq. 4.49], and the cosine is close to 1 for sensible pairs: 0.866 for box analysis with linear-spline synthesis, 0.836 with cubic [Lee 1998, Table I]. Unser’s worked example for linear splines gives $C_{\rm int}=9.129\times10^{-2}$ for interpolation against $C_{\rm opt}=3.727\times10^{-2}$ for the projection, with the oblique projection through a box prefilter within a factor 1.155 of optimal, and the comment that “sampling without analog prefiltering (interpolation) is by far the least favorable approach” and “the prefiltering does not need to be optimal” [Unser 2000, pp. 578–579].
Quasi-interpolation relaxes interpolation without requiring a prefilter on the continuous signal: choose $q$ so that polynomials up to the order of the space are reproduced, without insisting that $\tilde f$ pass through the samples. This recovers the projection’s constant from point samples for low-frequency content. Condat, Blu and Unser’s filters and Sacht and Nehab’s optimized ones are in Section 4.6.
2.5 Why the interpolation constraint on the kernel costs an order
A four-tap piecewise cubic can be made to do one of two things. It can interpolate by itself, as Keys does; then the best achievable order is 3, and the constant is $0.0115$. Or it can have order 4 and leave interpolation to the digital filter; then the constant is $1.66\times10^{-3}$ for the B-spline and $6.3\times10^{-4}$ for O-MOMS (Figure 2). The cost of evaluation is identical: four taps per dimension. The digital filter is a separate pass over the image, measured at a few percent of total time in the original experiments (3% for cubic) [Blu 1999, Table 1].
The interpolating four-tap cubic of maximal order does exist (Schaum’s cubic, the cubic I-MOMS, order 4), but its constant is 7.8 times that of the B-spline [Blu 2001, Table IV], and in the rotation test it lands below Keys (Section 4.7 ). Blu, Thévenaz and Unser state the conclusion plainly: “we are generally not in favor of this condition, except for applications where memory is tight and the image array needs to be stored in byte format” [Blu 2001, p. 1075]. The last clause is a real constraint: spline coefficients leave the range of the data. Nehab and Hoppe treat it by solving for coefficients constrained to $[-0.5,1.5]$ with five Gauss–Seidel iterations, and report about 93% of pixel values reproduced exactly (the maximum error is printed as 12% in the monograph and 1.2% in the 2011 report; one of the two is presumably a typo) [Nehab 2014, pp. 51–52; Nehab 2011].
One warning from the same source belongs here because it is the error behind the reputation of B-splines as blurry: “unless the output device uses $\varphi$ for reconstruction, it makes little sense to directly use the coefficient array $\mathbf c$ for display” [Nehab 2014, p. 23]. Coefficients are not pixel values, and pixel values are not coefficients. Feeding pixel values to $\beta^3$ without the prefilter is an order-2 smoothing filter with constant $1/6$, worse than bilinear (Figures 2, 6 and 8).
3. What is a pixel?
With the vocabulary of Section 2 the old argument about pixels becomes a question of which basis function goes with the numbers.
Point sample. Alvy Ray Smith’s memo says a pixel is a point sample of a continuous image and has no area [Smith 1995a]; PBRT repeats it: “there is no ‘area’ associated with a pixel … thinking of pixels as small squares with finite area is an incorrect mental model that leads to a series of errors” [PBRT, p. 295]. In the present notation this is the statement that the array holds $[\![f{\ast}\psi^\vee]\!]$ and that the reconstruction kernel is not specified by the data. It is the right default for processing, because it forces the question “reconstructed with what?”.
Little square. A box reconstruction, $\varphi=\beta^0$. It is the right model in three places: the exact area-average prefilter (coverage); the piecewise-constant space, in which the box is its own dual, so that “the optimal prefilter for a piecewise constant signal approximation is a box function” [Unser 2000, p. 573]; and, approximately, an LCD, which Nehab and Hoppe take as box reconstruction in their display experiments [Nehab 2014, p. 60]. Smith concedes the geometric usage in compositing: “Geometers think of ‘pixels’ as geometrical areas … Imagers think of pixels as point samples of a continuum. … In the end, it is the imaging model that dominates” [Smith 1995b].
Sensor aperture integral. On the capture side a pixel is an inner product of the optical image with the pixel aperture, after the lens and any optical low-pass filter. This is the prefilter $\psi$, and it is neither a delta nor band-limiting (Figure 3, right).
Display spot. On the output side a pixel is whatever the device emits: a Gaussian spot on a CRT (the model of Kajiya and Ullner, cited in [Nehab 2014, p. 11]), a box with subpixel structure on an LCD, a dot with gain on paper. PBRT draws the consequence: pixel values should be computed “such that the reconstructed image on the display device is as close as possible to the original image”, which “is a subtly different goal than expecting the display’s pixels to take on the image function’s actual value at their positions” [PBRT, p. 280]. Rocha, Oliveira and Gastal carry it through to the retina by convolving the display box with a model of the eye’s point spread (Section 9 ).
The reconciling view. A pixel value is a coefficient $c_k$, and it means something only together with a generator $\varphi$. The “point sample” view is the cardinal generator; the “little square” is $\beta^0$; the spot is the device’s $\varphi$; the sensor is $\psi$. Converting between views is a digital filter (Section 2.1 ), and errors arise when an array produced under one view is consumed under another without the conversion: spline coefficients displayed as pixels, box-filtered renderings fed to a sinc-like interpolator, coverage values treated as point samples of a matte.
Figure 3. Left: one 12×12 array of samples of a synthetic scene shown at 20× as little squares, Gaussian spots ($\sigma=0.25$ pixel), a tent reconstruction and a cardinal cubic B-spline reconstruction; the same 144 numbers in four bases. The spline reconstruction ranges over [0.020, 0.994] for samples in [0.120, 0.920]: it overshoots. Right: modulation transfer of a diffraction-limited f/5.6 lens at 550 nm (cutoff 324.7 cycles/mm), a 4 µm square pixel aperture and a two-spot optical low-pass filter. Lens × pixel is 0.681 at half Nyquist, 0.332 at Nyquist (125 cycles/mm) and 0.092 at 1.5 × Nyquist; with the low-pass filter 0.482, 0.000 and 0.065. [Code] Idealized monochrome chain, one axis, no color mosaic or aberrations.
3.1 Grid conventions and the half-pixel trap
A separate question from the shape of a pixel is where the samples sit. The convention used throughout this article, and by Nehab and Hoppe for graphics (their “dual grid”), is that pixel $i$ covers $[i,i+1)$ with its center at $i+\tfrac12$, so that an image of $n$ pixels covers $[0,n)$ and output sample $j$ of a resize by $s$ sits at input coordinate $(j+\tfrac12)s$ [Nehab 2014, §8; PBRT, p. 295]. Getreuer notes the property that singles this choice out: interpolation with a symmetric kernel on the centered grid commutes with flipping the image [Getreuer 2011]. Two other conventions are in circulation: scaling the index directly ($u=j\,s$), and aligning the corner samples ($u=j\,(n_{\rm in}-1)/(n_{\rm out}-1)$).
Figure 4. Positions of the output samples of a 2× enlargement under half-pixel centers, index scaling and align-corners (diagram), and a 24×24 glyph image enlarged with bilinear weights under each (tiles at 6×, with differences from the half-pixel result). Measured centroid of the ink: index scaling shifts the picture by −0.500 output pixel in both axes; align-corners leaves the centroid in place (−0.015 pixel) but stretches the picture about its center, with displacement running from −0.5 to +0.5 output pixel across the image. Largest difference from the half-pixel result: 0.285 of full scale for index scaling, 0.172 for align-corners. [Code]
A half-pixel shift is invisible in a single image and ruinous in a pipeline: it misregisters pyramids, biases optical flow, and in a repeated-transform test it turns an interpolator into a smoothing filter. Exactly this fault turns up in rotation routines (Section 11 ): source coordinates computed with centers at $k+\tfrac12$ are passed to interpolators that take integers as centers, so that at angle zero the bilinear rotator averages each pixel with its neighbor. The related trap for multiresolution work is that nesting (each coarse sample lying on a fine sample, or exactly between two) requires even-degree splines on one grid type and odd-degree on the other, which is why Nehab and Hoppe use degrees 3 and 5 on the dual grid [Nehab 2014, §8].
Boundaries need a convention too. For the recursive spline prefilter the half-sample-symmetric (mirror) extension is the recommended one, in preference to zero padding or periodization [Thévenaz 2000b; Briand 2018]; all figures here use it. Kirshner and Porat attribute “almost 2 dB” of their gain on a rotation test to handling the boundary exactly instead of by mirror extension [Kirshner 2008], which is a warning to exclude borders when comparing kernels.
4. The kernel catalog
4.1 One table
$W$ is the support in samples, $L$ the approximation order. “Prefilter” means a digital filter on the sample grid is required for the stated order (it is not the antialiasing prefilter). Cost is per output sample per dimension, before separable expansion. Rows marked † rest on memory.
| Kernel | $W$ | $L$ | Regularity | Interpolating by itself | Negative lobes | Prefilter | Taps; remarks |
|---|---|---|---|---|---|---|---|
| Nearest / box $\beta^0$ | 1 | 1 | discontinuous | yes | no | none | 1 |
| Linear / tent $\beta^1$ | 2 | 2 | $C^0$ | yes | no | none | 2; convex, so no overshoot |
| Shifted linear ($\tau\approx0.21$) | 2 | 2 | $C^0$ | no | no | one-pole IIR | 2; phase distortion |
| Dodgson quadratic ($r=1$) | 3 | 2 | $C^0$ | yes | yes | none | 3 |
| Quadratic B-spline $\beta^2$ (“Magic Kernel” †) | 3 | 3 | $C^1$ | no | no | IIR, pole $\sqrt8-3$ | 3 |
| Quadratic O-MOMS | 3 | 3 | discontinuous | no | — | IIR | 3 |
| Schaum quadratic (I-MOMS 2) | 3 | 3 | not recorded | yes | yes | none | 3 |
| Keys cubic, $a=-\tfrac12$ (Catmull–Rom ) | 4 | 3 | $C^1$ | yes | yes (−0.074) | none | 4 |
| Keys, other $a$ | 4 | 1 | $C^1$ | yes | yes | none | 4 |
| Mitchell–Netravali $(\tfrac13,\tfrac13)$ | 4 | 2 | $C^1$ | no | yes (−0.036) | none (used as is) | 4; mild blur by design |
| Schaum cubic (I-MOMS 3) | 4 | 4 | not recorded | yes | yes | none | 4; large constant |
| Cubic B-spline $\beta^3$, with prefilter | 4 | 4 | $C^2$ | no | cardinal kernel: −0.137 | IIR, pole $\sqrt3-2$ | 4 |
| Cubic B-spline, no prefilter | 4 | 2 | $C^2$ | no | no | none | 4; a smoothing filter |
| Cubic O-MOMS, with prefilter | 4 | 4 | $C^0$ | no | cardinal kernel: −0.167 | IIR, pole $(\sqrt{105}-13)/8$ | 4 |
| Condat quasi-interpolators (degrees 1–3) | 2, 3, 4 | 2, 3, 4 | as B-spline | no | — | all-pole IIR | as B-spline |
| Sacht–Nehab optimized (degrees 1–3) | 2, 3, 4 | 1 by design | bounded only | no | — | width-5 inverse FIR | about 10% slower than cubic O-MOMS |
| Lanczos-$a$ | $2a$ | none unnormalized; 1 with normalized weights | $C^1$ at the cutoff (value and slope vanish; the second derivative jumps) | yes | yes (−0.086 for $a=2$, −0.147 for $a=3$) | none | $2a$, plus a normalization |
| Quintic B-spline / O-MOMS, with prefilter | 6 | 6 | $C^4$ / $C^0$ | no | yes | IIR, two poles | 6 |
| Meijering quintic / septic | 6 / 8 | 3 | — | yes | yes | none | 6 / 8 |
| Gaussian | infinite | none (ripple 1.4% at $\sigma=0.5$) | $C^\infty$ | no | no | none | truncated |
| Rocha–Oliveira–Gastal SBS3 | about 2.8 | not stated | piecewise cubic | no | — | IIR (dual) | display-referred prefilter |
| Sinc | infinite | band-limited functions exact | $C^\infty$ | yes | yes (−0.217) | none | FFT only |
Sources: [Keys 1981; Mitchell 1988; Dodgson 1997; Blu 2001; Thévenaz 2000a; Condat 2005; Sacht 2014; Rocha 2020; Getreuer 2011] and the measurements of Figures 1 and 2. On quadratic O-MOMS regularity, Blu, Thévenaz and Unser note that for even $L$ the O-MOMS is continuous with a discontinuous derivative and for odd $L$ it is discontinuous, and conclude “there is no link between regularity and quality!” [Blu 2001, p. 1073].
4.2 Keys and Mitchell–Netravali
Keys’ cubic convolution [Keys 1981, eq. 4] is the one-parameter family
$$u(s)=\begin{cases}(a+2)|s|^3-(a+3)|s|^2+1 & 0\le|s|<1\\ a|s|^3-5a|s|^2+8a|s|-4a & 1\le|s|<2\\ 0 & 2\le|s|.\end{cases}$$A Taylor expansion of the interpolation error has leading terms proportional to $(2a+1)$, so that “the constraint $A_2=-1/2$ is the only choice … that will achieve third-order precision; any other condition will result in at most a first-order approximation” [Keys 1981, p. 1155]. With $a=-\tfrac12$ the pieces are $\tfrac32|s|^3-\tfrac52|s|^2+1$ and $-\tfrac12|s|^3+\tfrac52|s|^2-4|s|+2$: the Catmull–Rom spline, which Meijering finds had been “suggested at least three times before” [Meijering 2002]. Other values in use are $a=-1$ (Rifman and Bernstein, matching the slope of the sinc at 1) and $a=-\tfrac34$ (continuous second derivative at 1); both are first order, and in the rotation test $a=-1$ ends at 3.60 dB on the circular pattern where $a=-\tfrac12$ reaches 15.00 dB [Thévenaz 2000b, Table 2]. Keys also gives the boundary rule $c_{-1}=3f_0-3f_1+f_2$ and a six-point fourth-order kernel. A program that offers “bicubic” without naming $a$ has not said what it does.
Mitchell and Netravali [Mitchell 1988] parametrize all $C^1$ four-tap cubics satisfying the partition of unity:
$$k(x)=\frac16\begin{cases}(12-9B-6C)|x|^3+(-18+12B+6C)|x|^2+(6-2B) & |x|<1\\ (-B-6C)|x|^3+(6B+30C)|x|^2+(-12B-48C)|x|+(8B+24C) & 1\le|x|<2\\ 0&\text{otherwise.}\end{cases}$$$(1,0)$ is the cubic B-spline, $(0,C)$ the Keys family with $a=-C$, $(0,\tfrac12)$ Catmull–Rom. The error is $f-f_r=(2C+B-1)\,h f'\,r(x)+O(h^2)$, so the line $B+2C=1$ gives second order, and only its endpoint $(0,\tfrac12)$ third. The recommended $(\tfrac13,\tfrac13)$ sits on that line. It is not interpolating (it needs $B=0$), and Figure 2 shows what that means: order 2 with constant $B/6$, a mild fixed blur. That is a deliberate choice for a reconstruction filter in rendering, where the samples are noisy or aliased and a little blur masks defects. It is the wrong choice for repeated resampling, where the blur compounds (Figure 8), and Nehab and Hoppe’s table shows it there in second-to-last place. ImageMagick’s Robidoux filters lie in the same $B+2C=1$ family, which its documentation orders from blurrier to sharper as Spline, Robidoux, Mitchell, RobidouxSharp, Catmull–Rom. The usual constants are $B\approx0.3782$, $C\approx0.3109$ for Robidoux and $B\approx0.6796$, $C\approx0.1602$ for RobidouxSoft (from memory and forum posts †; a search found the 0.6796 pair in a forum thread and no write-up by Robidoux himself).
4.3 B-splines and their prefilters
$\beta^0$ is the unit box and $\beta^{n}=\beta^{n-1}{\ast}\beta^0$; in closed form $\beta^n(x)=\frac1{n!}\sum_{i=0}^{n+1}(-1)^i\binom{n+1}{i}\big(x-i+\tfrac{n+1}2\big)_+^n$ [Briand 2018]. The cubic is $\tfrac23-x^2+\tfrac12|x|^3$ on $[0,1]$ and $\tfrac16(2-|x|)^3$ on $[1,2]$. Degree $n$ has support $n+1$, order $n+1$, regularity $C^{n-1}$, is positive, and for $n\ge2$ does not interpolate; the cardinal spline tends to the sinc as $n$ grows.
The prefilter inverts the symmetric FIR filter of the integer samples of $\beta^n$. Its transfer function factors into $\lfloor n/2\rfloor$ pole pairs $(z_i,1/z_i)$, each applied as one causal and one anticausal recursion (formulas in Section 2.4 ), with overall gain $n!$ for odd $n$ and $2^n n!$ for even $n$ [Briand 2018, Prop. 2]:
| $n$ | Samples of $\beta^n$ at the integers | Poles $z_i$ |
|---|---|---|
| 2 | $[1,6,1]/8$ | $\sqrt8-3=-0.171573$ |
| 3 | $[1,4,1]/6$ | $\sqrt3-2=-0.267949$ |
| 4 | $[1,76,230,76,1]/384$ | $-0.361341,\;-0.0137254$ |
| 5 | $[1,26,66,26,1]/120$ | $-0.430575,\;-0.0430963$ |
| 6 | $[1,722,10543,23548,10543,722,1]/46080$ | $-0.488295,\;-0.0816793,\;-0.00141415$ |
| 7 | $[1,120,1191,2416,1191,120,1]/5040$ | $-0.535280,\;-0.122555,\;-0.00914869$ |
Poles from [Lee 1998, Table III] and [Briand 2018, Table 2]; rows 4 and 5 of the latter are misprinted in the PDF and the symmetric forms above are corrected here. Closed forms for the poles of $\beta^4$ and $\beta^5$ exist in [Thévenaz 2000b] but the copy read was garbled, so only decimals are given.
Practical points:
- Initialization. For the mirror boundary the causal recursion starts from $c^+_0=6\,(f_0+\sum_{k\ge1}z_1^kf_k)$, truncated when $|z_1|^k$ is below the target precision (about 12 terms for 24 bits with the cubic), and the anticausal one from $c_{N-1}=\frac{z_1}{z_1-1}c^+_{N-1}$ [Ruijters 2012]. Briand and Monasse tabulate the extension length needed per degree and precision (for example 14 samples in total for $n=3$ at $10^{-2}$ in two dimensions, 28 at $10^{-6}$) and warn that their boundary-transmitting algorithm fails for the constant extension [Briand 2018].
- Precision. Single precision is enough to about $10^{-6}$ relative error for the cubic; degree 11 in single precision floors at $6\times10^{-6}$ [Briand 2019].
- Higher degree. Unser sees “not much benefit beyond quintic splines” [Unser 2000, p. 579]; Briand and Monasse find the cubic differs from degree 16 by “around one gray level” and recommend higher degree “except when efficiency is primordial” [Briand 2018].
- GPU. A cubic B-spline can be evaluated with four bilinear texture fetches in two dimensions (Sigg and Hadwiger, cited in [Nehab 2014, §8]), which is 3 to 6 times slower than bilinear and about twice as fast as a 16-fetch Catmull–Rom or Mitchell. The prefilter is the obstacle: a 15-tap FIR approximation $b(k)=\sqrt3(\sqrt3-2)^{|k|}$ is 4 to 10 times faster on a GPU than the recursion [Champagnat 2011]. Csébfalvi obtains Catmull–Rom itself from four bilinear fetches, with no prefilter [Csébfalvi 2018].
- The misconception. “Unprefiltered cubic blurs more than linear” is measured in [Ruijters 2012] over 36 rotations of 10 degrees, and here in Figures 6 and 8 and in a larger round-trip test (Section 11 ), where the unprefiltered B-spline is last in all 42 columns, about 5.6 dB below the prefiltered one.
4.4 O-MOMS
Among all kernels of order $L$ and minimal support $L$, the one with the smallest asymptotic constant is $\varphi(x)=\Lambda_L(D)\,\beta^{L-1}(x)$ with the polynomials given by the recursion $\Lambda_{L+1}(x)=\Lambda_L(x)+\frac{x^2}{4(4L^2-1)}\Lambda_{L-1}(x)$, $\Lambda_1=\Lambda_2=1$, and constant $C_L=\frac{L!}{(2L)!\sqrt{2L+1}}$ [Blu 2001, Theorem 2]:
| $L$ | O-MOMS | Constant relative to the B-spline of the same order |
|---|---|---|
| 3 | $\beta^2+\tfrac1{60}D^2\beta^2$ | 1/1.8 |
| 4 | $\beta^3+\tfrac1{42}D^2\beta^3$ | 1/4.6 |
| 5 | $\beta^4+\tfrac1{36}D^2\beta^4+\tfrac1{15120}D^4\beta^4$ | 1/14.5 |
| 6 | $\beta^5+\tfrac1{33}D^2\beta^5+\tfrac1{7920}D^4\beta^5$ | 1/55.1 |
The cubic O-MOMS in explicit form [Thévenaz 2000a, p. 746]:
$$\text{o-Moms}^3(x)=\begin{cases}\tfrac12|x|^3-x^2+\tfrac1{14}|x|+\tfrac{13}{21}&0\le|x|<1\\[2pt]-\tfrac16|x|^3+x^2-\tfrac{85}{42}|x|+\tfrac{29}{21}&1\le|x|<2\\[2pt]0&2\le|x|.\end{cases}$$Its integer samples are $\tfrac1{21}[4,13,4]$, so the prefilter has the single pole $(\sqrt{105}-13)/8\approx-0.344131$. The term $\tfrac1{14}|x|$ gives it a cusp at the origin: it is only $C^0$. (One later paper, Liang 2008, prints the second piece with $+\tfrac{85}{42}|x|$; that is a sign error.) Quintic O-MOMS has poles $-0.475813$ and $-0.0709257$ [Getreuer 2011, Table 1]. The authors expected “at least a 6 dB SNR gain for signals whose frequency content lays essentially in the first half of the frequency domain” over the B-spline of the same support [Blu 2001]. As computed in Section 2.3 , the ratio of constants that applies to interpolation is 2.65 for the cubic, not 4.6.
4.5 Lanczos, other windowed sincs, quadratics, Schaum
Lanczos-$a$ is $\mathrm{sinc}(x)\,\mathrm{sinc}(x/a)$ on $|x| Quadratics. Dodgson’s family is $-2rx^2+\tfrac12(r+1)$ for $|x|\le\tfrac12$ and $rx^2-(2r+\tfrac12)|x|+\tfrac34(r+1)$ for $\tfrac12<|x|\le\tfrac32$; $r=1$ interpolates, $r=\tfrac12$ is the quadratic B-spline $\tfrac34-x^2$, $\tfrac12(|x|-\tfrac32)^2$ [Dodgson 1997]. The prefiltered quadratic B-spline and quadratic O-MOMS are the strong three-tap choices: in Table II of [Thévenaz 2000a] the quadratic O-MOMS at $W=3$ has a smaller constant ($0.0031$) than Meijering’s septic at $W=8$ (about $0.0044$) or Keys at $W=4$ ($0.0115$). Schaum’s kernels are the interpolating kernels of maximal order (central Lagrange, I-MOMS). They illustrate Section 2.5
: maximal order with a constant 7 to 8 times the B-spline’s. The handbook chapter says “we discourage the use of Schaum” [Thévenaz 2000b]. Smaller published variations (Gotchev’s modified B-spline, blended kernels, rational kernels, exponential splines from Sobolev reproducing kernels) exist; the blended and rational kernels gain at most 0.03 dB over Keys [Liang 2008]. Condat, Blu and Unser keep the B-spline and replace the interpolation prefilter by an all-pole filter $1/Q(z)$ designed so that the combined response matches the dual’s to order $L+1$ or $L+2$ [Condat 2005, Table 1; the venue, ICIP 2005, is from memory †]: In their test (17 rotations) the linear spline with this filter beats Keys on all six images at under half the time: “Bilinear quasi-interpolation performs better than bicubic interpolation, and is twice faster”. Related cheap corrections for linear interpolation: the FIR quasi-interpolator $-\tfrac1{12}z^{-1}+\tfrac76-\tfrac1{12}z$ (Blu and Unser 1999); Dalai’s five-tap FIR approximation to the dual of the hat, $\tfrac{49}{40}-\tfrac{11}{90}(z+z^{-1})+\tfrac7{720}(z^2+z^{-2})$ [Dalai 2005]; and the shifted linear interpolation of Blu, Thévenaz and Unser, in which the tent is displaced by $\tau$ and the samples prefiltered by $c_n=-\frac{\tau}{1-\tau}c_{n-1}+\frac1{1-\tau}f_n$, with $\tau_{\rm opt}=\tfrac12(1-\tfrac{\sqrt3}{3})\approx0.21$ [Blu 2004]. Shifted linear matches Keys after 15 rotations of Lena (28.25 against 28.15 dB) but distorts phase; with random rotation angles, where the phase errors do not cancel, it falls below Keys [Dalai 2005], and Sadeghi, Yu and Wang recommend the smaller shifts $\tau=1/8$ for natural images and $1/11$ for arbitrary signals [Sadeghi 2018]. Sacht and Nehab give up order altogether. They minimize $\int_0^d\omega^{-2}E(\omega)\,d\omega$ (the error kernel weighted by a natural-image spectrum) over the generator and a width-5 inverse-FIR digital filter, requiring only $L=1$ and bounded pieces [Sacht 2014]. The linear generator is $0.79076352\,\beta^1(x)+0.10461824\,[\beta^0(x+\tfrac12)+\beta^0(x-\tfrac12)]$, a tent with a small box added, discontinuous; the digital filter is the inverse of $[e_2,e_1,e_0,e_1,e_2]$ with $e_0=0.77412669$, $e_1=0.11566267$, $e_2=-0.00272602$. Quadratic and cubic coefficients are in their Appendix A; the generators are defined through distributions from [Blu 2003] and cannot be written as explicit piecewise polynomials. Reported: after 40 cumulative translations, PSNR 36.4 dB for their quadratic against 34.1 for Condat’s quadratic and 32.9 for quadratic O-MOMS; “the quintic cardinal B-spline has approximation order 6, and our cubic has only approximation order 1. Nevertheless, our cubic performs better” (a visual claim: the PSNR values in that figure are 13.044 and 13.047). Stated limits: the objective “is somewhat brittle”, the optimum is verified only as a local minimum [Schmitt 2015], there is “additional mild ringing”, and the linear scheme over-magnifies high frequencies under repeated half-pixel translation. The journal version (IEEE TIP 2015) is not obtained. Rocha, Oliveira and Gastal’s SBS3 is a prefilter designed for the display and the eye instead of for a reconstruction space (Section 9.1
). The reconstruction model is the display’s box convolved with a quadratic B-spline approximating the eye’s point spread, $\varphi=\mathrm{rect}{\ast}s$; the prefilter is its dual. For a 100 ppi display at 40 cm the kernel is a piecewise cubic with breakpoints 0.202514, 0.392458, 0.797486 and 1.392460 and thirteen coefficients (their eq. 4); the digital filter is computed from its autocorrelation and its numerical values are not published in the paper [Rocha 2020]. Theory. Ordered by the interpolation error kernel (Figure 2, and the predicted SNR for a white spectrum in [Thévenaz 2000a, Table III]: nearest 5.94 dB, linear 9.23, Keys 11.02, quadratic spline 12.11, quadratic O-MOMS 12.46, cubic spline 13.14, cubic O-MOMS 14.03): cardinal O-MOMS, then the cardinal B-spline of the same degree, then lower-degree prefiltered splines, then Keys, then Mitchell and the tent, then nearest. Unnormalized windowed sincs are outside the ranking (no order). Published. The classic experiment is 15 successive rotations of 24 degrees [Thévenaz 2000a, Table IV; SNR in dB, time for one 512×512 rotation on a Power Macintosh 9600/350]: Nehab and Hoppe’s larger experiment (about 50 kernels; repeated translation and rotation of a circular pattern and four Kodak images, plus one upscaling; mean SSIM) gives the averages: quintic O-MOMS 0.979, quintic B-spline 0.973, cubic O-MOMS 0.964, Condat cubic 0.943, Hamming $W=6$ 0.941, cubic B-spline 0.935, Condat quadratic 0.927, Lanczos $W=6$ 0.910, Keys 0.857, shifted linear 0.822, Mitchell 0.715, linear 0.644, nearest 0.457 [Nehab 2014, Table 10.1]. Their verdict: “The top performing methods all include a digital filter $q$ … Traditional linear interpolation, Mitchell and Netravali [1988], and Keys [1981] lag far behind.” Measured here (Figure 8, Section 5.4
; one photograph): rotation PSNR 31.33 dB for cardinal O-MOMS, 29.86 Lanczos-3, 29.29 cardinal B-spline, 26.61 Lanczos-2, 26.53 Keys, 23.81 Mitchell, 23.16 bilinear, 21.90 unprefiltered B-spline; translation 28.57, 26.88, 27.83, 24.40, 24.67, 23.18, 22.31, 21.39. The three agree on the main structure: prefiltered splines above the classic interpolating cubics above the smoothing kernels, with O-MOMS ahead of the B-spline. They disagree, or need care, in six places. Numbers for the same nominal experiment also differ between papers (linear on Lena after repeated rotation: 21.98 dB in [Thévenaz 2000a], 22 in [Blu 2004], 29.40 after 17 rotations in [Condat 2005]; cubic spline 31.98, 32 and 31.60 in three sources). Protocol details (number of rotations, boundary handling, region scored, image version) are enough to move results by several decibels, so only rankings within one paper are comparable. Minification by a factor $s$ needs a prefilter whose pass band shrinks with $s$: in Nehab and Hoppe’s notation $\mathbf c_s=[\![\mathbf c{\ast}(s\,\psi_{1/s})^\vee]\!]_{1/s}$, the kernel stretched by $s$ and renormalized [Nehab 2014, eq. 5.22]. For non-integer $s$ the sampled weights no longer sum to exactly one and must be normalized per output sample. Heckbert’s general statement covers every warp: the resampling filter is the reconstruction filter convolved with the warped prefilter; under magnification its support “is independent of the mapping, typically 1–4 pixels”, under minification it “could have arbitrary support: one pixel or a million pixels” [Heckbert 1989, p. 51]. The most common error is to skip the stretch. A “bilinear” or “bicubic” resize that keeps a two- or four-tap footprint whatever the scale is point sampling with a slightly blurred point. Parmar, Zhang and Zhu document that the default resize functions of OpenCV, TensorFlow and PyTorch behave this way, that libraries “give drastically different results”, and that the choice shifts the FID
score of generative models [Parmar 2022]. The omission is also a security hole. An unscaled kernel reads only a few source pixels for each output pixel (four for bilinear), so a second picture can be hidden in exactly those pixels and appears only after the resize: the image-scaling or camouflage attack, introduced by Xiao and co-authors, who attacked the nearest-neighbor, bilinear and bicubic resizers of OpenCV, Pillow, TensorFlow and Torch, and fooled four commercial cloud vision services whose scaling method was hidden [Xiao 2019, pp. 1, 6 and 12]. A later analysis lists as vulnerable nearest-neighbor, bilinear and bicubic in OpenCV and in TensorFlow’s Figure 5. Downscaling by 3.3 (660×660 to 200×200) by eight methods; zone plate (top) and photograph (bottom, processed in linear light). Measured on the zone plate: alias energy in the stop region, relative to the input amplitude, and pass-band gain at 0.3–0.45 cycles per output pixel. Nearest −0.0 dB alias; bilinear without antialiasing −1.8 dB; Keys without antialiasing −0.4 dB; box (exact area average) −11.7 dB alias, −3.12 dB gain; antialiased tent −19.8 and −5.67; antialiased Mitchell −22.5 and −5.21; antialiased Lanczos-3 −23.5 and −1.50; generalized sampling (cardinal cubic B-spline prefilter) −23.9 and −1.65. Lanczos-3 leaves the range of the data (−0.012 to 1.011); the generalized result stayed within [0, 1] on this image. [Code] Figure 5 separates the candidates on two axes. The unscaled kernels alias within 2 dB of point sampling. The box, the “exact” area average, removes only 11.7 dB of alias and already droops 3.1 dB at the top of the pass band: it is the least-squares prefilter for a piecewise-constant reconstruction and a poor one for anything else, a point Rocha, Oliveira and Gastal call “a disheartening conclusion” for LCDs [Rocha 2020]. Tent and Mitchell reject alias well but lose more than 5 dB of the top octave. Lanczos-3 and the generalized prefilter reach about −24 dB alias while holding the top of the pass band within 1.7 dB. The generalized recipe is worth writing out because it is cheap. Sample the stretched B-spline on the output grid, then apply the inverse B-spline filter on the output grid; the net prefilter is the cardinal spline, infinitely long, obtained from a four-tap footprint (seven or eight taps for a factor of two). For mip-maps
Nehab and Hoppe give
calling the cardinal cubic B-spline “an excellent prefilter”, at 35 floating-point operations per output pixel against 30 for Catmull–Rom and 49 for Lanczos with $W=6$ [Nehab 2014, eq. 5.52, p. 35]. Their test (500×700 to 65×91) finds that “the cardinal cubic B-spline and O-MOMS kernels virtually eliminate aliasing, while maintaining sharpness indistinguishable from the wider Lanczos kernel”, while Mitchell and Catmull–Rom still alias [Nehab 2014, Fig. 10.7]. Least squares. If the downscaled image will be enlarged again with a known kernel, the optimal reduction is the orthogonal projection onto that kernel’s space. Unser’s group implemented it for splines: Lee, Eden and Unser by the oblique projection with a box analysis function (fit a spline, integrate, resample the integral, difference, post-filter; cost that of a spline interpolation of one degree higher) [Lee 1998], and Muñoz, Blu and Unser exactly, by $n_1+1$ running sums, one resampling with a spline of degree $n+n_1+1$, $n_1+1$ finite differences and two digital filters, at a cost independent of the scale factor [Muñoz 2001]. Nehab and Hoppe restate the same construction as “repeated integration” [Nehab 2014, eqs. 5.43–5.46]. Zhang and co-authors’ “interpolation-dependent image downsampling” is the same idea in matrix form, $X=(H^TH)^{-1}H^TY$ for a given interpolation operator $H$, and gains 1 to 1.5 dB over direct subsampling in a halve-then-double test [Zhang 2011]. Least squares is not automatically what one wants to look at. The 2011 report notes that “using the orthogonal projection can be perceptually unsatisfactory … contains strong undesirable ringing artifacts due to the Gibbs phenomenon
” [Nehab 2011, §5.1], and displaying least-squares coefficients for a cubic space directly on a box-like display is wrong in the sense of Section 3
[Nehab 2014, Fig. 10.4]. Beyond linear prefilters (abstracts only †). Trentacoste, Mantiuk and Heidrich measured how much blur a reduced image needs in order to look as sharp or as blurred as the original, and add it locally [Trentacoste 2011]. Liu, He and Lau pose downscaling with $L_0$ priors on gradients and on the downsampling matrix [Liu 2018, not obtained]. Learned adaptive-kernel downscalers continue the line (ADK-Net 2025, not read), and Liang and co-authors propose a quality measure specific to downscaling, built on a generative model’s rate–distortion behavior [Liang 2024]. For striped color subpixels the prefilter is designed per channel: Platt derives subpixel values by minimizing a psychophysical error measure, and Betrisey and co-authors give the displaced box filters used for text [Platt 2000; Betrisey 2000; neither obtained]. All of this concerns the shape of the prefilter. On which values to apply it, see Section 6
: the prefilter must act on linear light, and Figure 11 shows the error from not doing so is far larger than any difference between kernels in Figure 5. Magnification needs no prefilter: $\mathbf c_s=[\![\mathbf c{\ast}\varphi]\!]_{1/s}$ [Nehab 2014, eq. 5.16]. The kernel alone determines the result, and the trade is sharpness against overshoot. Figure 6. A 96×64 synthetic image (text, one-pixel lines, a ring, a slanted edge) enlarged 4× with eight interpolators, and the continuous reconstruction of an ideal step. Measured overshoot as a percentage of the step, and 10–90% rise width in source pixels: nearest 0 and 0; bilinear 0 and 0.812; Mitchell 3.63 and 0.844; Keys 7.41 and 0.719; cardinal cubic B-spline 10.78 and 0.750; Lanczos-3 11.78 and 0.719; cardinal cubic O-MOMS 12.40 and 0.719; cubic B-spline without prefilter 0 and 1.312. Interpolation on sRGB-encoded values. [Code] Three points from Figure 6. The unprefiltered B-spline is the blurriest by far (rise width 1.31 pixels against 0.81 for bilinear). The four sharpest interpolators have the same rise width, 0.72 to 0.75 pixel, so on a single enlargement of a clean edge they are distinguished only by the amount and extent of ringing. And overshoot is the price of that sharpness: 11 to 12% of the step for Lanczos-3 and the cardinal splines, against 7.4% for Keys and 3.6% for Mitchell. Nehab and Hoppe argue that the step is too pessimistic a test: “images with good signal-to-noise ratio
and proper antialiasing rarely contain impulses or even sharp step-discontinuities. Instead, we think that the ramp response is more representative … and it predicts significantly reduced ringing for all kernels” [Nehab 2014, p. 57]. That is true of photographs and false of text, line art and screenshots, where ringing from these kernels is plainly visible and a monotone kernel or an edge-directed method is preferable. No linear kernel adds information. Above factors of two or three, the limit is the missing detail, not the kernel; edge-directed and learned methods address that and are outside the scope of this article (Section 10
lists what was catalogued). Reducing and re-enlarging by the same factor measures how much of the image a pair of filters preserves, and it has a computable ceiling: the orthogonal projection onto the reconstruction space. Figure 7. PSNR of a round trip (reduce by $s$, enlarge by $s$) on a 480×480 photograph for ten method pairs, absolute (left) and relative to the least-squares cubic-spline projection (right). At $s=2$: nearest/nearest 25.39 dB; box/bilinear 28.88; Mitchell/Mitchell 28.27; Keys/Keys 30.56; generalized/cardinal B-spline 32.37; Lanczos-3/Lanczos-3 32.52; least squares, linear spline 32.67; cubic-spline interpolation both ways with no prefilter 32.80; least squares, cubic spline 33.77. At $s=6$ the same methods give 19.80, 22.48, 22.34, 22.77, 22.94, 22.95, 23.00, 20.85 and 23.10 dB. Values are processed as encoded numbers (the comparison is between linear operators); discrete least squares on the sample grid; one image. [Code] The ceiling is close. At a factor of 2 the best prefiltered pairs are 1.25 to 1.4 dB below the least-squares cubic spline, and at factors of 4 and 6 within 0.35 and 0.15 dB. Mitchell both ways is below box-then-bilinear at $s=2$, because both of its passes blur. The least-squares linear spline is ahead of, or within 0.15 dB of, every non-projection method at $s\ge2$ (only unprefiltered interpolation at $s=2$ and the Lanczos-3 pairs at $s=3$ edge past it): the choice of coefficients matters as much as the choice of space, which is Unser’s point in Section 2.4
. The curve for interpolation without any prefilter is instructive: it is the best non-projection method at $s=1.5$ and 2 (39.03 and 32.80 dB) and falls to 2.3 dB below the ceiling at $s=6$. At small factors little of the spectrum lies above the new Nyquist frequency, and an antialiasing filter costs more in pass-band droop than it saves in alias; at large factors aliasing dominates. Lee, Eden and Unser found the same: “the primary reason for the lesser performance of the standard interpolation methods is aliasing” [Lee 1998, p. 688]. Their published round trips show the size of the effect on other data: at a scale of 0.5 on one image, cubic interpolation 31.77 dB, least squares 33.03, oblique 33.34; for non-integer magnification the projection gains “often more than 10 dB … for linear splines, … as much as 20 dB” [Lee 1998, Tables V and p. 689]. A larger round-trip test (seven classic images, six scales, twenty filters) agrees with Figure 7 in ordering: cardinal O-MOMS first on average (31.84 dB), then the wide Lanczos filters and the cardinal B-spline (31.5 to 31.8), Catmull–Rom (30.8), box (29.3), Mitchell (28.9), tent (28.1), and the unprefiltered B-spline last (26.0). A sub-pixel translation is a convolution of the coefficients with the sampled, shifted kernel, with the digital filter applied once: $\mathbf c_h=\mathbf c{\ast}[\![\tau_h{\ast}\varphi]\!]{\ast}\mathbf r$ [Nehab 2014, eq. 5.2]. A result worth knowing: the least-squares translation within the space of $\beta^n$ equals a plain translation in the space of $\beta^{2n+1}$, “twice the approximation order” for a wider footprint [Nehab 2014, eq. 5.8]. Rotation resamples on a rotated grid; minification is absent, so again no antialiasing prefilter is involved and the test isolates the interpolator. Repeating a transform until the image returns to its starting position, and comparing with the original, is the stress test introduced by Unser’s group. Its merit is that small per-step losses compound into unmistakable differences; its limits are that it measures squared error, and that almost nobody rotates an image fifteen times. Figure 8. After 15 successive rotations of 24° (top) and 20 successive translations of (0.35, 0.2) pixel undone by an integer shift (bottom), on a 256×256 photograph, scored on the central 112×112 window. Rotation PSNR and SSIM: cardinal O-MOMS 31.33 dB and 0.9525; Lanczos-3 29.86 and 0.9369; cardinal B-spline 29.29 and 0.9202; Lanczos-2 26.61 and 0.8400; Keys 26.53 and 0.8369; Mitchell 23.81 and 0.6765; bilinear 23.16 and 0.6283; B-spline without prefilter 21.90 and 0.5329; nearest 19.52 and 0.4894. Translation PSNR: O-MOMS 28.57, cardinal B-spline 27.83, Lanczos-3 26.88, Keys 24.67, Lanczos-2 24.40, Mitchell 23.18, bilinear 22.31, unprefiltered B-spline 21.39, nearest 12.68. Lanczos weights are normalized; splines are re-prefiltered at every step; no clipping between steps. [Code] Two results in Figure 8 are not the textbook expectation and are reported as measured. Nearest neighbor scores 12.68 dB on the translation test, far below its rotation score, because each step of (0.35, 0.2) pixel rounds to no movement at all: the image never moves, and is then compared after the integer un-shift of (7, 4) pixels. The number measures a 7-pixel misregistration, not blur. And Lanczos-3 beats the cardinal cubic B-spline on rotation (by 0.57 dB) but loses on translation (by 0.95 dB). It has six taps to the spline’s four, so a win is not a surprise; the reversal on translation is consistent with Nehab and Hoppe’s observation that Lanczos amplifies certain frequencies under repetition, but one image does not settle it. Cardinal O-MOMS is best in both tests, at four taps. The absolute values are 3 dB or so below the published ones for Lena (34.29 dB for cubic O-MOMS). The test image here has more fine texture (fur), and the ranking, not the level, is what transfers. For a single rotation the picture changes. Figure 9. A 160×160 pattern with detail at the sampling limit (3-pixel checkerboard, grating of 4-pixel period, one-pixel lines, star) rotated once by 30°, against an analytically rotated, area-sampled reference. PSNR over the central disk, and output range: nearest 16.75 dB, [0.100, 0.900]; bilinear 29.17 dB, [0.100, 0.900], with contrast (standard deviation) reduced to 0.842 of the source; Lanczos-3 24.67 dB, [−0.227, 1.185], contrast 0.983; cardinal O-MOMS 24.56 dB, [−0.270, 1.184], contrast 0.979. [Code] Bilinear wins the PSNR by 4.5 dB while visibly losing the contrast of the finest detail and modulating it with position. The reason is in the source: it was area-sampled (a box prefilter), so it carries contrast right up to and beyond the Nyquist frequency, part of it aliased. The sharp interpolators reproduce those samples faithfully, overshoot included (to −0.27 and 1.18 on data in [0.1, 0.9]), and are charged for the alias; the tent blurs some of it away. This figure is evidence about PSNR on aliased sources, to be read with Section 8
, and not a recommendation of bilinear. It is also a reminder that a “reference” defines what counts as error: against a band-limited reference the order would differ. Rotation by three shears factors a rotation as an $x$-shear by $-\tan(\theta/2)$, a $y$-shear by $\sin\theta$, and the same $x$-shear again [Paeth 1986, p. 2; Unser 1995, eq. 2]. Each shear is a set of one-dimensional translations, which can use the high-quality one-dimensional machinery above, and unlike the older two-pass factorization it needs no intermediate contraction. Unser, Thévenaz and Yaroslavsky report that the three-pass method with higher-order one-dimensional interpolators “outperforms all standard high-accuracy methods of which we are aware, both in terms of speed and quality”. For a 37° rotation of a circular test pattern their RMS errors are 40.1 for nearest neighbor, 23.7 for bilinear, 13.4 for the Keys cubic, 9.2 for three-pass cubic B-spline and 4.3 for the three-pass septic spline [Unser 1995, Fig. 5]. Whether generalized-sampling kernels improve it is an open question. A general warp varies the scale and orientation of the footprint from pixel to pixel. Heckbert’s formulation: with reconstruction filter $r$, warp $m$ and prefilter $h$, each output sample is $g(x)=\sum_kf(k)\,\rho(x,k)$ with
and “when resampling we never actually construct a continuous signal” [Heckbert 1989, eq. 3.2]. Locally the warp is affine with Jacobian $J$, and with Gaussians for both filters the composite is again a Gaussian, with covariance $J^{-1T}V_hJ^{-1}+V_r$: the elliptical weighted average (EWA) filter. Using unit covariances for both, $J^{-1T}J^{-1}+I$, “guarantees that the resampling filter is always wider than both the reconstruction and prefilters”, which fixes the original EWA’s tendency to “fall between the pixels” under magnification [Heckbert 1989, pp. 53–61]. The ellipse $Q(u,v)=Au^2+Buv+Cv^2 Practical systems approximate this in two steps. A mip-map prefilters the texture at power-of-two scales, and trilinear
filtering interpolates between the two levels bracketing the footprint size; it is isotropic, so it must choose between blurring to the long axis of the footprint and aliasing along it. Anisotropic filtering takes several trilinear samples along the long axis at the level set by the short axis. The scheme descends from Feline, which approximates the EWA ellipse by several isotropic probes along its major axis [McCormack 1999 †]. Rutanen’s notes on EWA add two cautions: the texture-space derivatives are not the ellipse’s principal axes (taking them as such underestimates eccentricity), and eccentricity should be clamped (“30 seems to be a good value”) [Rutanen 2009]. Figure 10. A checkerboard ground plane in perspective (footprint major axis from 0.35 to 54.2 texels per pixel, anisotropy from 1.27 to 20.6): reference (256 samples per pixel), point sampling, bilinear on the base level, trilinear mip-mapping, and anisotropic filtering with up to 16 trilinear taps. RMS error against the reference in sRGB code units, whole image and far third: point sampling 51.49 and 82.10; bilinear 44.68 and 71.40; trilinear 30.59 and 40.88; anisotropic 20.34 and 10.68. Standard deviation of the far third: reference 99.8, anisotropic 96.1, trilinear 80.3. Filtering in linear light. [Code] The anisotropic filter is the multi-tap form used by graphics hardware, not EWA. Figure 10 shows the expected order with measured sizes: trilinear removes the moiré
but washes out the far field (contrast 80.3 against the reference’s 99.8) because it blurs to the long axis; sixteen-tap anisotropic filtering
recovers it (96.1) and cuts the far-field error fourfold. Three further points connect texture filtering
to the rest of the report. First, the mip chain is a downscaling chain and inherits everything in Sections 5.1 and 6: a box-filtered chain is the convention and a weak prefilter; the chain must be built in linear light (Figure 11 shows a chain built on encoded values darkening level by level). Second, the magnification side is an interpolation problem, where bilinear is the hardware default and prefiltered cubic B-splines are available at four bilinear fetches with offset weights (Section 4.3
; [Sigg 2005], excerpt); Lin, Seiler and Yuksel propose hardware support that skips the higher-order terms where the texture is locally smooth [Lin 2021]. Third, recent work replaces deterministic filtering by stochastic sampling of the filter footprint with the filtering moved after shading [Fajardo 2023; Pharr 2024; Wronski 2025; Akenine-Möller 2025]; these are catalogued, not summarized (Section 10
). Section 8.3 of Nehab and Hoppe’s monograph, “Prefiltering, reconstruction, and color spaces”, is two short paragraphs and a figure. In full (printed pages 50–51; PDF pages 53–54 ; the page break falls after “moved to a”): 8.3 Prefiltering, reconstruction, and color spaces In reconstruction, the average between two function values should be perceived to have the intermediate value between them. Luminance values do not have this property due to a nonlinear response in the human visual system. This nonlinear perceptual response to luminance is called lightness. The gamma correction in the sRGB standard approximates the conversion of luminance to lightness. Therefore, when processing visual data we recommend performing reconstruction directly in sRGB space to achieve better perceptual results. In stark contrast, prefiltering is an area integral of radiant energy. It must be performed over linear intensities and only then can results be moved to a perceptual color space [Blinn, 1998]. As figure 8.2 demonstrates, prefiltering is best done in luminance space and reconstruction is best done in lightness space. The figure caption (printed p. 51): Figure 8.2: Prefiltering, reconstruction, and color spaces. (a) A large image with two ramps (one continuous and one dithered) was downsampled by prefiltering linear intensities (top ramps). Downsampled ramps look the same. When prefiltering is performed in lightness space, the results do not match (bottom ramps). (b) An image upsampled with quintic B-spline interpolation in lightness space. (c) The same image upsampled in linear intensities appears incorrect. That is the whole of their treatment. A search of the 87-page PDF for gamma, sRGB, linear intensity, lightness, luminance and perceptual finds only four other relevant sentences, of which one bears on this topic, in the discussion of prefilter choice: “Some favor the sharpness offered by filters with negative lobes (the ’negative lobists’ [Blinn, 1989]), while others fear the accompanying ringing in dark regions (a problem accentuated by gamma correction
)” [Nehab 2014, p. 11]. The 2011 technical report that preceded the monograph contains that sentence and no recommendation about spaces. The code listing in their Appendix A is single-channel and does no color conversion. The only source they cite for the prefilter half, Blinn’s 1998 column “A ghost in a snowstorm”, is not obtained and its content was not verified †. So the recommendation rests on a two-sentence argument for each half and one figure. It is correct in its main claim and it deserves a fuller explanation than it received, including the places where it is incomplete. Luminance $Y$ is proportional to radiant power weighted by the eye’s spectral sensitivity; “linear light” or “linear intensity” means pixel values proportional to it. Light adds: two sources together give the sum of their luminances, and a region seen from far enough that its detail is not resolved appears with its average luminance. Lightness is the perceptual response to luminance relative to a reference white. It is compressive: equal ratios of luminance are closer to equal steps than equal differences are. The CIE’s formula is $L^{\ast}=116\,Y^{1/3}-16$ for $Y>0.008856$ (and $903.3\,Y$ below), on a scale of 0 to 100. The sRGB encoding maps linear values to code values by $V=1.055\,Y^{1/2.4}-0.055$ for $Y>0.0031308$ and $V=12.92\,Y$ below. It was designed around display behavior, and it is close to a lightness scale: mid-gray code 128 of 255 is about 21.6% luminance and $L^{\ast}\approx53$. “The gamma correction in the sRGB standard approximates the conversion of luminance to lightness” is that statement. It is an approximation, and Section 6.8
returns to what “perceptual space” should mean. Almost all stored images are encoded, and almost all filtering code operates on whatever numbers it is given. The question of their Section 8.3 is on which numbers each linear operation should act. A prefilter computes, for each output pixel, a weighted average over a region of the input. The physical event being modeled is real: a sensor pixel integrates the light falling on its aperture; a distant or defocused view of a surface integrates the light from a patch of it; an eye looking at a display from far enough away integrates over several pixels. In each case what is averaged is radiant energy. “Prefiltering is an area integral of radiant energy” is meant literally. It follows that the average must be taken of quantities proportional to energy. Averaging encoded values computes something else, and because the encoding is concave the result is always too dark: for any set of values, the mean of the encoded values is at most the encoding of the mean (Jensen’s inequality), with equality only when the values are all equal. The error is therefore zero in flat regions, small in smooth ones, and large exactly where the prefilter is doing its job, in fine high-contrast detail. Their Figure 8.2(a) uses the cleanest possible test: a smooth ramp and a dithered ramp that look alike from a distance must still look alike after both are reduced. Figure 11 makes the same test quantitative. Figure 11. Top: a black-and-white source of fine patterns, its 4× box reduction computed on sRGB code values, and computed in linear light. Bottom: the same for a photograph, with the difference amplified 8×; and the mean displayed luminance of each level of a mip chain built either way. Measured: a one-pixel checkerboard (true mean luminance 0.5) becomes code 127.5, luminance 0.2140, when codes are averaged (−57.2%, 22.7 $L^{\ast}$ too dark) and code 187.5, luminance 0.5, in linear light. One-pixel lines with one in four lit: −79.6% (30.1 $L^{\ast}$). White text on black: −61.2%. Isolated white pixels covering 6% of a black ground: −88.1%. Black text on white: −3.4%. Photograph (512×512 to 128×128): mean difference 1.36 codes, largest 26.3 codes, and the linear-light result is never the darker one. Mip chain of the synthetic source: mean luminance 0.4627 at every level in linear light; 0.3204 at level 1 and 0.2993 at level 7 when built on code values. Mip chain of a photo crop: 0.3364 constant, against a fall to 0.3009. [Code] The numbers in Figure 11 define the size of the problem. On ordinary photographic content the mean error is about one code value, which is why the mistake survives in so much software: results look plausible. The error concentrates in specular highlights, whiskers, stars, lit windows at night, fine bright text, and any regular fine pattern, where it reaches tens of code values and changes the apparent brightness of whole regions. Bright detail on a dark ground loses 60 to 90% of its light; dark detail on a bright ground is nearly unaffected (3 to 7%), because the encoding is steep near black and shallow near white. A mip chain built on encoded values gets darker with every level, so a textured surface dims as it recedes. A separate analysis (more in Section 6.8
) reaches the same verdict, “Correct. Close to a theorem”, with the condition that makes it one: it holds whenever the output pixels, or the kernel footprint, are below the eye’s resolution, so that the eye would have performed the same average on the original. That condition is the definition of minification. Reconstruction is a different kind of operation. Between two samples there is no measurement; the interpolator invents values, and the invention is correct only under an assumption about the signal (band-limited, or a spline). No physical process is being modeled. The question “in which space?” then has no physical answer and has to be decided by what looks right. Nehab and Hoppe give one criterion, and the rest of the literature supplies a second. The midpoint argument (theirs). “The average between two function values should be perceived to have the intermediate value between them.” Take the two levels used in Figure 12, sRGB codes 64 and 224, of lightness 27.1 and 89.2. The perceptual midpoint is $L^{\ast}=58.1$. Averaging the code values gives code 144, $L^{\ast}\approx59.8$. Averaging the luminances gives luminance 0.398, $L^{\ast}\approx69.3$: the “halfway” value is eleven lightness units too light (arithmetic from the definitions in Section 6.2). An interpolated edge computed in linear light therefore spends most of its width looking bright, and its apparent position moves toward the dark side; a ramp interpolated in linear light does not look like a ramp. On encoded values the interpolated transition is close to uniform in appearance. The ringing argument (their aside on p. 11, Mitchell and Netravali, and Figure 12). A kernel with negative lobes overshoots by the same amount above and below, as a fraction of the step, in whatever space it is applied. In linear light, an undershoot of 7 to 12% of a step from dark gray to near white is many times the dark level itself; it lands at or below zero, is clipped to black, and appears as a hard dark halo, while the equal overshoot on the bright side changes the lightness by a few units and is hardly seen. In encoded values, equal excursions are roughly equal perceptual steps, so the two lobes look alike and neither is exaggerated. Figure 12. Top: bright-on-dark and dark-on-bright glyphs and a one-pixel line (levels sRGB 64 and 224) enlarged 6× with Keys, Lanczos-3 and the cardinal cubic B-spline, interpolating sRGB values (first row) or linear light (second row). Bottom: Lanczos-3 profiles of an edge, a bright line and a dark line in luminance and in $L^{\ast}$. Measured for the edge, undershoot on the dark side and overshoot on the bright side in $L^{\ast}$: Keys on sRGB values 5.32 and 4.17, in linear light 27.09 (clipped to black) and 2.36; Lanczos-3 8.55 and 6.61 against 27.09 and 3.71; cardinal B-spline 7.80 and 6.05 against 27.09 and 3.41. Width of a one-pixel line at half its $L^{\ast}$ contrast with Lanczos-3: bright line 1.19 source pixels on sRGB values and 1.44 in linear light; dark line 1.12 and 0.88. [Code] Figure 12 shows both effects. With every kernel the linear-light interpolation drives the dark side of the edge to black (an excursion of 27.09 $L^{\ast}$, the entire dark level) and leaves 2.4 to 3.7 $L^{\ast}$ on the bright side: an asymmetry of roughly ten to one. On encoded values the two sides are 5.3 and 4.2 for Keys, 8.6 and 6.6 for Lanczos-3. The line widths show the midpoint effect: in linear light a bright line on dark reads 1.44 pixels wide and a dark line on bright 0.88, a ratio of 1.6; on encoded values 1.19 and 1.12. Text enlarged in linear light looks bold in one polarity and thin in the other, with black fringes around the bright glyphs (visible in the second row of tiles). Note what the figure does not claim. Interpolating encoded values does not conserve light, and the excursions are not smaller in every sense: the bright-side overshoot is larger on encoded values (6.6 against 3.7 $L^{\ast}$ for Lanczos-3). The claim is symmetry and proportion, as seen. Antialiased rasterization computes the fraction of each pixel covered by a shape, an area integral, and blends foreground and background by it. That is a prefilter, and the blend belongs in linear light. Figure 13. The same coverage mask (text at 8, 10 and 13 pixels, one-pixel lines, a ring, a triangle) blended in linear light (left) and on sRGB values (right), white on black (top) and black on white (bottom). A half-covered pixel is code 187.5 in the linear blend and 127.5 in the encoded blend. Measured: white-on-black shapes blended on encoded values emit 0.667 of the correct light (mean code of partially covered pixels 113.8 against 165.4); black-on-white shapes remove 1.337 times the correct amount of light (141.2 against 187.4). [Code] Blending on encoded values makes light-on-dark text thin and dark-on-light text heavy. The general case can be quantified [Code]. For a black-to-white edge blended on sRGB values, the light deficit as a function of coverage peaks at 0.287 of the step at 53% coverage and averages 0.19 of a pixel’s light, which is a mean edge displacement of 0.19 pixel toward the light side. Because the map from coverage to light is not affine, a thin feature also changes brightness as it moves. A one-pixel white line delivers its full light when it is aligned with the pixel grid and 43% of it when it straddles two pixels equally, a factor of 2.3, so thin moving features shimmer with a one-pixel period. Ganacim and co-authors describe the same thing from the renderer’s side: evaluating the antialiasing integral in gamma space “leads to dark regions that look wider than intended … Unfortunately, many users have grown accustomed to incorrect rendering. As a compromise, we support both alternatives” [Ganacim 2014, p. 10, quotation not re-verified †]. That last remark matters in practice: fonts and hinting have been tuned for decades against encoded-space blending, so physically correct text can look too light or too heavy to users, and renderers compensate (stem darkening). Correct is not the same as preferred. Figure 14. A source with fine white detail on black (left half) and shapes at sRGB 64 and 224 (right half), reduced 4× with a box prefilter and enlarged 4× with Lanczos-3, each stage in linear light or on sRGB values. Mean luminance of the left half (source 0.1213) and halo sizes in the flat grounds (undershoot in the dark ground, overshoot in the bright ground, in $L^{\ast}$): linear prefilter and linear reconstruction 0.1230 (+1.4%), 27.09 and 4.34; linear prefilter and sRGB reconstruction 0.1199 (−1.1%), 11.77 and 6.82; sRGB prefilter and linear reconstruction 0.0268 (−77.9%), 27.09 and 4.79; sRGB prefilter and sRGB reconstruction 0.0254 (−79.1%), 10.48 and 7.47. [Code] Figure 14 crosses the two choices. The rows differ by brightness: prefiltering on encoded values loses 78 to 79% of the light of the fine detail whatever is done afterwards, and prefiltering in linear light keeps it within 1.4%. The columns differ by halo: reconstructing in linear light clips the dark side to black; reconstructing on encoded values gives 10 to 12 $L^{\ast}$ of undershoot and about 7 of overshoot. The recommended combination, linear prefilter and encoded reconstruction, is the top right tile, and the common default (everything on encoded values) is the bottom right: its only fault in this figure is the darkening. The mixed case that some “gamma-correct” pipelines produce by accident, and all strictly linear pipelines produce by design, is the left column. The figure also answers a natural worry, that reconstructing on encoded values gives away the brightness the linear prefilter preserved. It costs 1.1% here against +1.4% for the all-linear pipeline. The reconstruction stage redistributes light over a pixel or two; it does not average fine detail away, so its effect on local mean brightness is second order. Mip-maps and any image pyramid are repeated prefiltering: build them in linear light (Figure 11, bottom). The GPU’s sRGB texture formats exist for this: texels are decoded before filtering and the result is linear. Texture magnification through the same hardware path then also happens in linear light, which by Section 6.4
is the less attractive choice for strong magnification of high-contrast texels, and is seldom noticed with bilinear filtering because bilinear has no overshoot. Alpha and compositing. Alpha is coverage or opacity, a linear quantity, and is never gamma-encoded. Smith’s memo on the question is blunt: “One computes on linear data with linear alpha. If the standards committees of the world insist that nonlinear data be passed around, then it is passed around in display space where alpha doesn’t matter” [Smith 1998]. In an 8-bit sRGB pipeline that means: decode color, leave alpha alone, premultiply, composite, re-encode. Premultiplied alpha
also makes filtering and compositing commute and is the only form in which “over” is closed under composition [Smith 1995b]. Blur, glow, depth of field, motion blur model light spreading, and are prefilters in the sense that matters. High-dynamic-range resolve and chroma subsampling
are the two places where “average the light” needs a qualifier (practitioner sources, snippets only †). Averaging the samples of a pixel in linear HDR values is physically right and visually aliased, because one very bright sample dominates the mean; game renderers weight each sample by roughly $1/(1+\max(r,g,b))$, a reversible tone map applied before the average and undone after it (Karis; AMD’s “optimized reversible tonemapper for resolve”; Pettineo’s comparison of ten resolve filters). And chroma subsampling in a non-constant-luminance $Y'C_bC_r$ system averages encoded chroma, which produces large errors for saturated colors near the gamut
boundary, worst with the steep HDR transfer functions [Norkin 2016 †]. The companion article on color and HDR
shows the effect. Figure 15. Row 1: small bright disks on a dark ground blurred with a Gaussian ($\sigma=5$ pixels) on sRGB values and in linear light. Row 2: red|green and magenta|green edges blurred horizontally ($\sigma=6$). Row 3: a soft matte composited with “over”. Plots: luminance and $L^{\ast}$ across the blurred red|green edge. Measured: blurring sRGB values keeps 0.228 of the light of the spots (peak code of a small white spot 40.3 against 65.8; glow area 268 pixels against 1280). Across red|green the luminance dips to 0.1470, below both colors (red is 0.2126), a dark seam of $L^{\ast}$ 45.2 against red’s 53.2; in linear light the transition is monotonic. At the edge midpoint the code is (128, 128, 0), luminance 0.1998, against (187, 187, 0), luminance 0.4639. Compositing at 50% alpha: blue over yellow gives (128, 128, 127), luminance 0.215, against (188, 188, 187), luminance 0.501. [Code] The dark seam between saturated complementary colors is the same error seen per channel: each channel’s transition is too dark in the middle, and where one channel falls as another rises the two half-values sum to much less than either end. It is the origin of the dark fringes on blurred or resized colored edges, on chroma-subsampled video, and on soft brushes in paint programs that blend encoded values. The same rule holds for a fast Gaussian blur
: the sum must be formed in linear light. The stated reason and the usual reason differ. Nehab and Hoppe justify encoded-space reconstruction by midpoint uniformity. The ringing argument, which most people associate with the advice, appears only as an aside elsewhere in the monograph. Both point the same way, but they have different scopes: the midpoint argument applies to every interpolator including bilinear; the ringing argument only to kernels with negative lobes. The evidence is one figure. There is no metric, no derivation and no user study, and the caption of Figure 8.2 does not say in which space the zone plate
was synthesized, which matters, since a zone plate generated as a sinusoid in code values is a different signal from one generated in luminance. Most other sources say “always linear”. In the material read for this article, explicit statements about the space for filtering exist in only three places besides their Section 8.3: Heckbert (“it is essential that displayed intensity be proportional to the computed intensity … Without gamma correction, efforts toward antialiasing are hampered” [Heckbert 1989, p. 47]), Smith (“all algorithms that have been invented over the last two or three decades assume linear data” [Smith 1998]), and Ganacim and co-authors on coverage. None of them distinguishes reconstruction from prefiltering. The classic kernel papers (Keys, Mitchell and Netravali, Unser’s group, Sacht and Nehab, Rocha and co-authors) and the texture-filtering sources (PBRT’s chapter, Rutanen, Lensch) do not mention the question at all. The halftoning literature is firmly linear, for reasons given in Section 7.9
. So the recommendation to reconstruct in a perceptual space is a minority position in print, though a common practice in resizing software, and repeated searches found no peer-reviewed, controlled study of linear against perceptual interpolation at all. The clearest practitioner arguments are numerical examples: a texel of value 0.5 averaged on encoded values “will be displayed at only 25 percent of the brightness” at gamma 2.0 (Gritz and d’Eon, GPU Gems 3 †), and the FreeType documentation’s rule that text “must be rendered with linear alpha blending and gamma correction” †. There is also a counterexample to “always linear” from learned resamplers: the DLSS programming guide requires that low-dynamic-range input “must not be provided in linear space”, or the output shows banding and color shifts [NVIDIA 2026, p. 20]. What exists is practitioner material (search snippets only †). Robidoux’s ImageMagick forum posts report that enlarging in linear light with negative-lobe filters gives dark halos, that large gammas exaggerate light halos, and that his own “sigmoidal” resizing is not a general method; he suggested blending a linear-light and an encoded-space enlargement. That is the same observation as Figure 12 and the same lack of a criterion. Poynton’s “The Rehabilitation of Gamma” is often cited here, but it is about coding: encoded values are a perceptually uniform code in which 8 bits suffice where linear light needs 11 or more [Poynton 1998, pp. 2 and 6]. It does not discuss averaging, resizing or blending. GPU Gems 3, chapter 24 (not read) is the practitioner statement of the prefilter half. Blinn’s 1998 column remains unobtained (DOI 10.1109/38.637309). Font rasterizers’ gamma handling was not searched. Small factors and unresolved pixels. Reconstruction is “a guess judged by eye” only if the eye resolves what was guessed. When an image is rotated, shifted by a fraction of a pixel or scaled by a factor near one, and then viewed at a pixel density near or beyond the eye’s limit, the output pixels are themselves averaged by the eye, and the interpolation weights act partly as a prefilter. Then the linear-light argument applies to the interpolator too. One estimate puts the crossover at about twofold magnification on a 95 ppd phone display, with the perceptual space winning from fourfold up; it is an estimate, not a measurement, and no experiment in the material gathered tests it. This half of the advice is a good heuristic, not a law. The two halves do not compose cleanly. A pipeline that prefilters in one space and reconstructs in another is not a projection: it is not idempotent, and repeated round trips drift. A one-dimensional test shows the size [Code]. A one-pixel white line taken through eight round trips of 2× enlargement with the Keys kernel and 2× box reduction keeps 52% of its light under the mixed recipe (enlarge on encoded values, reduce in linear light), 33% when both steps act on encoded values, and 114% when both act on linear values with the negative lobes clipped. For a box-sampled black-to-white edge enlarged 16×, the point where the result crosses $L^{st}=50$ lies 0.27 pixel from the true edge under the mixed recipe, 0.42 pixel when the enlargement is done in linear light, and 0.04 pixel when sampling and enlargement both act on encoded values. Under heavy magnification, then, the perceived edge sits closest to the true one when the prefilter’s space matches the reconstruction’s. In Figure 14 the mixed recipe is nonetheless the best of the four by eye, so the incompatibility is a matter of residual bias, not of gross error. Where a generalized-sampling pipeline has a digital filter between the stages, there is the further question of which space the correction filter belongs in; the monograph does not say. Which perceptual space? sRGB encoding is the convenient choice because the data are already in it, not because it is the best lightness scale. It approximates $L^{\ast}$, which itself is a model for surface colors under a fixed adaptation, and neither accounts for the contrast-dependent, frequency-dependent visibility that actually governs whether ringing is seen (Section 8.6
). Interpolating per channel in encoded RGB also interpolates chroma along a path that is not perceptually straight; whether an opponent or uniform color space (CIELAB, Oklab
) would be better for reconstruction was not found addressed in the sources. For high-dynamic-range content the sRGB curve is the wrong model outright: the perceptual quantizer
(PQ) curve † and the PU21 encoding [Mantiuk 2021b] are the corresponding perceptual scales, and with displays that can show values above nominal white, some overshoot that would have been clipped becomes displayable [Rocha 2020, §4.3]. None of this was researched in depth. Clipping and negative values. The halo in linear light is partly an artifact of clipping at zero. Keeping negative intermediate values through a chain of operations and clipping once at the end avoids compounding, and Rocha and co-authors observe that for an orthogonal basis clamping negative values is the best non-negative approximation, while for a biorthogonal pair it need not be [Rocha 2020, §4.3]. Clamping between the two separable passes of a resampler is a common shortcut, and it is avoidable (Section 11
). Synthesis of the above: Resampling produces real numbers; storage and display take a few bits. This section follows the error from that last step: what it is, how dither makes it harmless, how shaping hides it, and on which brightness scale each step belongs. Halftoning (two levels, or a handful of inks) is the extreme case and gets most of the space. A uniform quantizer with step $\Delta$ makes an error $\epsilon$ that is a deterministic function of its input. The familiar model, error uniformly distributed with variance $\Delta^2/12$ and uncorrelated with the signal, is Bennett’s fine-quantization approximation †, and Widrow’s quantizing theorems † give the condition under which it is exact for the moments: the characteristic function of the input must vanish at multiples of $1/\Delta$. (The companion article on sampling theory
, Section 4
, treats this in detail; the statement of the moment condition here is from [Christou 2008, p. 8], which was read.) A smooth gradient across a few code values violates the condition completely: the error is a sawtooth locked to the signal, and its visible form is banding
(false contours). Two things can be done about it. Dither adds noise before the quantizer so that the error becomes statistically independent of the signal. Noise shaping
feeds past errors back so that the error spectrum is moved to where it matters least. They are independent, and the best systems do both. Whether a step of one code value is visible depends on how large it is in perceptual terms, and that depends on the encoding. Figure 16. Top: an sRGB ramp quantized to 16 levels equally spaced in sRGB code or in linear light, without and with triangular dither. Bottom left: the lightness step between adjacent codes for five encodings. Measured: with 16 sRGB-spaced levels the steps are 5.06 to 8.16 $L^{\ast}$; with 16 linear-spaced levels they run from 2.64 to 31.04 $L^{\ast}$, the first step from black alone being 31.04. Largest step between adjacent codes: 8-bit sRGB 0.509 $L^{\ast}$; 8-bit linear 3.542; 10-bit linear 0.883; 12-bit linear 0.221; 10-bit sRGB 0.127. Codes below $L^{\ast}=10$: 28 for 8-bit sRGB, 3 for 8-bit linear, 47 for 12-bit linear. [Code] Figure 16 gives the reason for nonlinear encodings in one row of numbers: 8-bit linear data has three codes for the darkest tenth of the lightness scale and steps of up to 3.5 $L^{\ast}$; 8-bit sRGB has 28 codes there and no step above 0.51 $L^{\ast}$. About 12 linear bits are needed to match 8 encoded bits in the shadows. Hence two rules: intermediate results in linear light need 16-bit or floating-point storage, and quantization to 8 or 10 bits should happen after encoding. Gjøl’s practical account for games says the same: without sRGB-aware stages or more than 8 bits there is a “terrible loss of precision”, and even with sRGB “dark areas remain” a problem [Gjøl 2016, slides 9–10]. For high dynamic range
, Miller, Nezamabadi and Daly found that a 12-bit gamma encoding sits near visibility thresholds at 100 cd/m²
but exceeds them at higher peak luminance, which motivated the perceptual quantizer curve [Miller 2012]. The journal version is Miller, Nezamabadi and Daly, SMPTE Motion Imaging J. 122(4), 2013. Measured treatments of step size against visibility do exist. Threshold experiments on minimum bit depth for HDR pixel encodings found the perceptual quantizer the most uniform but its “gain in bit-depth rather moderate”, a logarithmic encoding that “can be equally effective in some applications”, and luminance from 0.005 to 10,000 cd/m² encodable without perceptual loss in 11 bits or more [Mantiuk 2014, p. 5]. Kim, Azimi and Mantiuk measured banding directly: the contrast threshold is highest at 0.1 cd/m², falls up to about 100 cd/m², and is roughly constant from there to 10,000 cd/m² [Kim 2020, abstract; the body of the paper, p. 3, puts the flat region from about 10 cd/m²]; Denes and co-authors give a contrast-sensitivity-based model that predicts the probability of detecting a band, including chromatic ones [Denes 2019 †]. Figure 16’s $L^{\ast}$ steps remain a rough yardstick; visible banding on smooth gradients begins below one $L^{\ast}$ unit. Schuchman’s condition for subtractive dither (noise added before the quantizer and subtracted after) is that the dither’s characteristic function vanish at all nonzero multiples of $1/\Delta$; then the error is uniform and independent of the input, with variance $\Delta^2/12$, and rectangular (RPDF) dither one step wide is the lowest-power dither that does it [Christou 2008, p. 11]. (The primary statement is Theorem 4, “Schuchman’s Condition”, of [Lipshitz 1992, p. 363], read from the page image: an if-and-only-if condition for arbitrary input distributions; with such dither quantization “has thus merely added to the input signal a white-noise process of total power $\Delta^2/12$”.) Images are almost always dithered non-subtractively (the display does not know the noise), and there the statement is weaker and concerns moments: the $m$-th moment of the total error is independent of the input if and only if the $m$-th derivative of $\mathrm{sinc}(\Delta u)\,P_\nu(u)$ vanishes at all nonzero multiples of $1/\Delta$, with $P_\nu$ the characteristic function of the dither [Christou 2008, Theorem 1.3.1]. The consequences: Figure 17. A ramp requantized to 8 levels with no dither, white RPDF (1 LSB), white TPDF (2 LSB), a blue-noise mask used as RPDF, and the mask remapped to TPDF; and the measured mean and variance of the error against input level. Measured: no dither, mean error up to 0.5 LSB and zero variance (a deterministic sawtooth); white RPDF, mean error at most 0.0057 LSB, variance swinging from 0 to 0.25 LSB²; white TPDF, mean at most 0.0062, variance between 0.2458 and 0.2500; blue-noise RPDF 0.0001 and 0 to 0.25; blue-noise TPDF 0.0002 and 0.2498 to 0.2502. [Code] The measurements in Figure 17 match the theory to three decimals, and show one more thing: drawing the dither values from a blue-noise mask instead of a white generator changes none of these statistics. The mask controls where in the frequency plane the noise sits; the amplitude distribution controls its relation to the signal. The two choices are independent, and the usual practical recipe combines them: a blue-noise mask remapped to a triangular distribution [Gjøl 2016; Wronski 2016]. Practical rules, mostly from [Gjøl 2016]: With only two output levels, proper dither in the sense of Section 7.3
is impossible: there is no headroom, and “dither … does not apply to the half-toning problem since we cannot dither the signal properly without saturating the quantizer” [Christou 2008, p. 33]. Halftoning instead compares each pixel with a position-dependent threshold $t(x,y)$, output 1 where the input exceeds it. If the thresholds are uniformly distributed, the mean output equals the input, and all the design goes into their spatial arrangement. Figure 18. A photograph and a ramp halftoned to one bit eleven ways (view at 100%): fixed threshold, white-noise threshold, clustered-dot 8×8 at 45°, Bayer 8×8, a 64×64 void-and-cluster mask, Floyd–Steinberg (raster and serpentine), Jarvis–Judice–Ninke, Stucki, Atkinson, and Ostromoukhov’s variable-coefficient error diffusion. All operate on linear-light values. Measured on the photograph: the unweighted RMS error is 0.36 to 0.38 for every method that preserves tone; the fixed threshold has a mean error of +0.1875 and Atkinson −0.0478 (it loses shadows and highlights). Error weighted by a contrast-sensitivity model at 60 ppd: white noise 0.2004, clustered dot 0.2984, Bayer 0.1189, blue-noise mask 0.1403, Floyd–Steinberg 0.1173, Jarvis 0.1552, Stucki 0.1395, Atkinson 0.1395, Ostromoukhov 0.1235; at 120 ppd: white 0.1021, Bayer 0.0436, mask 0.0440, Floyd–Steinberg 0.0275, Ostromoukhov 0.0272. [Code] The table inside Figure 18 states the principle of the whole section: all tone-preserving halftones have about the same error (it cannot be otherwise when every pixel is forced to 0 or 1), and they differ by a factor of two to four once the error is weighted by what the eye can see. Ulichney’s term for the ideal dispersed-dot pattern: “the high-frequency complement of pink noise” [Ulichney 1988]. In Lau and Arce’s words, a “distribution of binary pixels where the minority pixels are spread as homogeneously as possible … aperiodic, isotropic (radially symmetric) and does not contain any low-frequency spectral components” [Lau 2008, p. 77]. For a gray level $g$ on a grid of pitch $D$, minority pixels spread evenly are separated on average by the principal wavelength $\lambda_b=D/\sqrt g$ ($g\le\tfrac12$; $D/\sqrt{1-g}$ above), giving the principal frequency $f_b=\sqrt g/D$. The pattern’s variance is $g(1-g)$, and its radially averaged power spectrum (RAPSD: the power spectrum averaged over annuli) should show “(a) little or no low-frequency spectral components; (b) a flat, high-frequency (blue-noise) spectral region; and (c) a spectral peak at cutoff frequency $f_b$” [Lau 2008, pp. 80–81]. A companion measure, the anisotropy (the relative variance of the spectrum around each annulus), should stay near the −10 dB expected of an isotropic pattern estimated from ten periodograms; directional structure becomes noticeable above 0 dB [Ulichney 1988; Lau 2008, p. 53]. Green noise is the clustered counterpart: blue-noise-distributed clusters of mean size $\bar M$ pixels, with principal frequency $\sqrt{g/\bar M}/D$ and energy concentrated at mid frequencies, for devices that cannot print isolated dots [Lau 2003]. Figure 19. Two-dimensional power spectra (top) and radially averaged spectra (bottom) of halftones of constant gray $g=1/8$, $1/4$ and $1/2$ for white noise, Bayer 8×8, the 64×64 void-and-cluster mask, and Floyd–Steinberg. Measured: at $g=1/8$ the spectral peak is at 0.354 cycles per pixel for the mask and 0.357 for Floyd–Steinberg (principal frequency $\sqrt{1/8}=0.354$); mean power below half the principal frequency, relative to white noise: mask −12.8 dB, Floyd–Steinberg −17.1 dB. At $g=1/4$: mask −12.7 dB; Floyd–Steinberg −18.6 dB but locked into a periodic pattern (line spectrum). At $g=1/2$: mask −8.1 dB; Floyd–Steinberg is a pure checkerboard with all its power at (0.5, 0.5). Bayer has power only at a few lattice frequencies. [Code] Two results in Figure 19 differ from the textbook picture. Floyd–Steinberg
shows the blue-noise profile only at $g=1/8$; at exactly $1/4$ and $1/2$ a constant input locks into periodic patterns, at $1/2$ a perfect checkerboard. This is the documented behavior (“texture shifts or banding at multiples of the 1/3 and 1/4 gray-levels”; “the most disturbing patterns evolve near $g=1/2$” [Lau 2008, pp. 82–83; Ulichney 1988]) and is why practical error diffusion perturbs something (Section 7.6
); a checkerboard is in one sense the ideal pattern at $g=1/2$, since $f_b=\sqrt{1/2}$ is the corner of the frequency square, and in another sense the worst, since any departure from it is a visible defect in a regular texture. And the 64×64 mask suppresses low frequencies by only 8 to 13 dB, less than error diffusion’s 17 dB: a mask is a fixed pattern that must serve every gray level at once. Error diffusion quantizes each pixel in scan order and distributes the resulting error over unprocessed neighbors (after [Lau 2008, eqs. 5.3–5.5], written here with the threshold at one half and the sign convention of the usual implementation):
The error carried forward is what the pixel was owed minus what it received. The weights, with Verified rows are from [Lau 2008, pp. 83–89] and [Ostromoukhov 2001]; rows marked † come from secondary listings and were not checked against primary sources. The Shiau–Fan row was checked against the text of US patent 5,353,127 (one half to the right; on the next line 1/16, 1/16, 1/8, 1/4 at offsets −3 to 0) and agrees with the matrix printed in [Krahmer 2024, p. 19]; the 4 sits directly below the current pixel. The same patent also gives a four-term set (one half right; 1/8, 1/8, 1/4 at offsets −2 to 0). No primary source exists for Burkes, Sierra or Atkinson: they are software lore. The original papers (Floyd and Steinberg 1976; Jarvis, Judice and Ninke 1976; Stucki 1981) are not obtained. Behavior and remedies: Write the quantizer as an additive error source, $y=u+\epsilon$ with $u$ its input. Error diffusion feeds $\epsilon$ back through the filter $h$: $u=x-h{\ast}\epsilon$. Then
the signal passes unchanged and the error is filtered by the noise transfer function $1-H$ [Christou 2008, pp. 20–21; Lau 2008, eqs. 5.9–5.10]. This is exactly the error-feedback form of a sigma-delta
modulator, in two dimensions. Krahmer and Veselovska’s survey makes the identification precise, error diffusion as first-order weighted sigma-delta quantization, and uses it to explain why higher-order shaping is rare in images: a direct higher-order scheme can lead to severe artifacts in black or white regions, where the quantizer has no amplitude margin, and needs rescaling and direction-dependent feedback filters to behave [Krahmer 2024, pp. 18–19]. For Floyd–Steinberg
[Christou 2008, p. 38], which is zero at $z_1=z_2=1$ because the weights sum to one: the mean is preserved, and error power is pushed away from DC. Sharma’s handbook: “the filter function … is a highpass filter … origin of the well known blue noise characteristics”. What shaping cannot do. For any causal, stable error filter (no term at the origin) the average of the logarithm of the noise transfer function over the frequency plane is non-negative,
with equality if and only if $1-H$ is minimum phase
(or marginally so). This is Christou’s two-dimensional generalization of Jensen’s theorem
[Christou 2008, Theorems 1.5.3–1.5.4 and 3.4.4]; the one-dimensional case is the content usually credited in audio to Gerzon and Craven, whose paper (AES Convention 87, 1989) is not free; its abstract says only that filtered error feedback is optimal when the filter satisfies a minimum-phase condition, and the log-integral form was found restated in patents, not in a citable derivation †. In decibels: every decibel of suppression at some frequencies is repaid by amplification at others, with equal volumes above and below 0 dB at best. Shaping moves noise; it cannot remove it. In power terms it adds: by Parseval
the mean of $|1-H|^2$ is $1+\sum h_i^2$, which is 1.328 (+1.2 dB) for Floyd–Steinberg (computed here; the measured RMS error on the photograph, 0.3722, compares with $\sqrt{0.1022\times1.328}=0.368$ predicted from the measured quantizer-error variance). The gain is entirely in where the error sits relative to the eye’s sensitivity, and the design problem is to fit the suppression to the visibility surface. Christou’s verdict on Floyd–Steinberg is that it meets the equality but wastes its suppression on a null at DC that is sharper than needed; his filter designed from a radially symmetric target is “a slight improvement”, and “greater noise shaping comes with a greater headroom requirement”. Figure 20. Top: $|1-H|$ over the frequency plane for Floyd–Steinberg, Jarvis–Judice–Ninke, Stucki and Atkinson. Bottom left: radial averages of $|1-H|^2$ (lines) and measured error spectra on a photograph (dots). Bottom right: smoothed halftone of a step edge. Measured, in the order Floyd–Steinberg, Jarvis, Stucki, Atkinson: largest $|1-H|$ 1.511, 1.208, 1.194, 1.281; value at DC 0, 0, 0, 0.250; mean $|1-H|^2$ below 0.1 cycles per pixel −15.3, −10.9, −11.3, −8.9 dB; fitted signal gain $K_s$ 1.83, 3.43, 3.29, 2.22; edge overshoot on the bright side 0.025, 0.063, 0.059, 0.099 (Atkinson’s includes a tone error of about 0.06: it reproduces 0.25 as 0.190 and 0.75 as 0.811). [Code] The sharpening side effect. The additive model assumes the quantizer error is uncorrelated with the signal. It is not: Kite, Evans and Bovik found it “highly correlated with the input image” [Kite 1997] and modeled the quantizer as a gain $K_s$ on the signal path plus uncorrelated noise. The signal transfer function becomes
unity at DC and rising at high frequency (they report gains of 4 for Floyd–Steinberg and 9 for Jarvis), while the noise still sees $1-H$ [Kite 1998, eqs. 3.11–3.16]. Their measured $K_s$ averaged over four images: 2.03 for Floyd–Steinberg, 4.37 for Jarvis, 3.94 for Stucki. Error diffusion therefore sharpens the image, more with larger filters. Figure 20 reproduces the pattern with a different estimator (a least-squares fit on one image): 1.83, 3.43 and 3.29, the same ordering at somewhat lower values. The measured error spectra follow $|1-H|^2$ at mid and high frequencies and exceed it at low frequencies, and the halftoned edge overshoots by 0.025 for Floyd–Steinberg and about 0.06 for Jarvis and Stucki. That is why the larger kernels look “crisper” in Figure 18, and also why they score worse on the weighted error there: some of what is measured as error is sharpened signal. The sharpening can be cancelled by threshold modulation with a negative constant, or kept as a feature. Atkinson’s filter diffuses only three quarters of the error. Its noise transfer function is 0.25 at DC, so it does not preserve the mean: shadows and highlights clip to solid black and white (mean error −0.048 in Figure 18, flat levels wrong by 0.06 in Figure 20). The look is high-contrast and clean, which is why it is liked; as a tone reproduction method it is deliberately lossy. Why moving the noise helps, and by how much. Figure 21. Top: white-noise and blue-noise halftones of gray 1/2 (identical mean squared error) and the same after filtering with a contrast-sensitivity low-pass at 60 ppd. Bottom: the sensitivity model, and weighted RMS error against pixels per degree. Measured ratio of weighted error power, white over blue-noise mask, at $g=1/2$: 0.1 dB at 20 ppd, 1.6 dB at 40, 4.5 dB at 60, 13.4 dB at 120 (at $g=1/4$: 0.1, 1.5, 4.1, 12.7). Weighted RMS at 60 ppd, $g=1/4$: white 0.2296, mask 0.1426, Floyd–Steinberg 0.0734, Bayer 0.0683. [Code] The sensitivity model is the Mannos–Sakrison formula made low-pass, isotropic, luminance only: a common simplification in halftoning work, not a calibrated visibility model. Figure 21 is honest about a point that enthusiastic accounts of blue noise skip: the advantage depends entirely on viewing conditions. At 20 ppd (large pixels, close viewing) white and blue noise are equally visible; at 40 ppd the difference is 1.5 dB; only at 60 ppd and beyond does it become large. The same figure rates Bayer dither better than the blue-noise mask at every distance, because all of Bayer’s error is at the highest frequencies. That is a failure of the measure, not a recommendation: a frequency-weighted error has no term for the conspicuousness of a regular texture, which viewers object to more than to fine irregular grain. Any single-number halftone metric built on a contrast sensitivity function
shares the defect. If the goal is to minimize what the eye sees, one can say so and optimize directly. The eye models are simple linear filters, converted from cycles per degree to cycles per pixel through the viewing distance and resolution. Those in use include the Mannos–Sakrison function $H(f)=2.6\,(0.0192+0.114f)\,e^{-(0.114f)^{1.1}}$ (peak near 8 cycles per degree), Campbell’s difference of exponentials, Näsänen’s exponential $e^{-k\rho}$, and Kim and Allebach’s mixed Gaussian, whose two published parameter sets trade smoothness (but “checkerboard artifacts”) against homogeneity [Lau 2008, pp. 66–75; Christou 2008, p. 37]. Kim and Allebach remark that the scale factor “serves more as a free parameter”, which is candid: these are tuning devices validated by the look of the result. Lieberman and Allebach’s observation that DBS under such a model also minimizes pointwise error at twice the viewing distance † is a reminder that the model’s distance parameter is not the physical one. Section 8.6
describes what current vision science would put in their place. A halftone reproduces a tone by having the eye average black and white dots. The average is of light. So the fraction of white dots must equal the target luminance, and the error being diffused must be an error in luminance. Kolpatzik and Bouman state it as a requirement: the data must be “transformed back to a linear format before processing. This transformation is important since our models for the human visual system and the display assume that the data are proportional to light intensity” [Kolpatzik 1992, p. 278]. S-CIELAB does its spatial filtering “in a linear color space using unit sum filters; consequently the spatial filtering preserves mean color values” [Zhang 1997]. Figure 22. An sRGB ramp halftoned by Floyd–Steinberg and by a blue-noise mask on code values and on linear light, to one bit and to four gray levels; and the resulting tone error. Measured: one bit on sRGB values, largest error 30.21 $L^{\ast}$ (at code 45, where 17.3% of dots are white instead of 2.6%), mean 17.83; one bit in linear light, largest 3.21 (a start-up effect at code 15), mean 0.41. Four levels (codes 0, 85, 170, 255) on sRGB values: largest 9.61, mean 3.00; in linear light 0.18 and 0.04. [Code] Assumes dots that add linearly (no dot gain). Dithering code values to one bit renders every mid-tone far too light: code 128 is 21% luminance and receives 50% white dots. Nayuki’s web note states the same case (about 22% white pixels, not 50%) and the corrected loop: carry the error in linear light, On real devices linearity in light is the starting point, not the end. Printed dots spread (dot gain) and reflect light scattered within the paper (the Yule–Nielsen effect), so the tone curve must be measured and corrected, and “dispersed dot methods are very sensitive to dot gain” [Pappas 2003]. Displays whose pixels interact (crosstalk between neighboring pixels of an e-paper
panel, for example) break additivity in the same way. Color halftoning has three additional difficulties: the output colors are few and fixed, the channels are perceptually unequal, and errors can leave the gamut. Channel-independent dithering (each of R, G, B separately) is the default and “not an optimal method” [Christou 2008]. Vector error diffusion quantizes the color vector to the nearest output color and diffuses a vector error. Its known failure is “smear and slow response” when input colors lie outside the convex hull
of the output colors: the error accumulates without bound until the scan reaches a region that can absorb it [Baqai 2005, p. 92]. Remedies are clipping the accumulated error, propagating it conditionally, and gamut-mapping the image into the hull first. On the space in which to work, the sources divide along the same line as Section 6
: Figure 23. A color photograph reduced to a six-color palette (black, white, red, green, blue, yellow): nearest color without dither by sRGB distance and by Oklab distance; Floyd–Steinberg on sRGB values; in linear light with linear-RGB matching; in linear light with Oklab matching. Measured mean Oklab error (×100) after a Gaussian blur of $\sigma=3$ pixels standing in for distance viewing, and mean luminance error: nearest by sRGB 18.44 and +0.196; nearest by Oklab 20.36 and +0.105; diffusion on sRGB values 18.16 and +0.236; diffusion in linear light with linear matching 0.76 and −0.0007; with Oklab matching 0.64 and +0.0008. [Code] Idealized saturated primaries; one image; the blur radius is arbitrary. Figure 23 reproduces the gray result in color and adds a surprise. Diffusing on encoded values leaves the picture washed out (mean luminance error +0.24) and is no better after blurring than no dither at all (18.2 against 18.4); diffusing in linear light brings the blurred error below one Oklab unit. The surprise is the third tile: nearest-color matching in Oklab without dither turns the gray-brown animal red, with 49% of pixels assigned to red, because red is the palette entry nearest in lightness to mid-tones, and a perceptual metric weights lightness heavily. A perceptual distance used for a hard per-pixel decision, with no mechanism to cancel the resulting hue error over a neighborhood, makes the result worse (20.4 against 18.4). Used inside error diffusion in linear light it helps a little (0.64 against 0.76). The lesson generalizes: a perceptual space is the right place to judge a difference and the wrong place to do arithmetic on light. Primary sources (abstracts, patents and web pages †). Palette design: Heckbert’s median cut
(SIGGRAPH 1982; not obtained), Gervautz and Purgathofer’s octree quantization (1988), Wu’s variance-minimizing bipartition (Graphics Gems II, 1991) and Orchard and Bouman’s tree-structured palettes (1991); the widely used pngquant is a modified median cut followed by a k-means
refinement, with dithering suppressed where the palette already fits. Color halftoning: the minimum-brightness-variation criterion, which restricts each pixel’s output to the four device colors that span its color with the least brightness variation, is documented only in a Hewlett-Packard patent (US 5,991,438); Neugebauer-primary error diffusion has an open-access treatment [Wu 2020]. Arbitrary palettes: Knoll’s pattern dithering is US patent 6,606,166; Yliluoma’s positional dithers are a web page. None of these sources settles the space question in the table above. E-paper. Six-color electrophoretic panels (black, white, red, yellow, blue, green) are a current hard case. What a survey of that problem adds to the above (checked against primary sources only in part): A display refreshed many times a second can spread quantization error over frames as well as over space. The material gathered covers only the rendering-oriented end of this. Interleaved gradient noise is animated by adding a per-frame offset ( Every ranking in Sections 4
and 5
was made with PSNR or SSIM. This section asks what those numbers measure, where they fail, what is better, and how much is actually known about predicting what a viewer sees. $\mathrm{MSE}=\frac1N\sum_i(x_i-y_i)^2$ and $\mathrm{PSNR}=10\log_{10}(L^2/\mathrm{MSE})$ with $L$ the peak value. PSNR is the energy of the difference signal, and it is the right measure for one question: how far is this array from that array, as vectors? It is convex, differentiable, additive over independent errors, and tied to the least-squares theory of Sections 2
to 5
, which is why it will not go away. As a predictor of appearance it makes four implicit assumptions, each false for vision [Wang 2009, p. 101]: that the error’s visibility does not depend on where the samples are in space, on the image content under it, on its sign, or on which samples carry it. In addition it depends on what the numbers are: PSNR on encoded values and on linear values rank images differently (Figure 25c), and it knows nothing of the display or the viewing distance. Huynh-Thu and Ghanbari’s summary is the usable rule: PSNR is a valid quality measure only while content and codec are held fixed (abstract only †). Within one family of algorithms on one image it orders results sensibly; across kinds of distortion it does not. The structural similarity index of Wang, Bovik, Sheikh and Simoncelli [Wang 2004] compares local statistics. For two aligned patches with means $\mu_x,\mu_y$, variances $\sigma_x^2,\sigma_y^2$ and covariance $\sigma_{xy}$,
the product of a luminance term $\frac{2\mu_x\mu_y+C_1}{\mu_x^2+\mu_y^2+C_1}$, a contrast term $\frac{2\sigma_x\sigma_y+C_2}{\sigma_x^2+\sigma_y^2+C_2}$ and a structure term $\frac{\sigma_{xy}+C_3}{\sigma_x\sigma_y+C_3}$ with $C_3=C_2/2$. The statistics are computed in an 11×11 circular-symmetric Gaussian window of standard deviation 1.5 normalized to unit sum; $K_1=0.01$, $K_2=0.03$; the image score is the mean over all window positions. The reference implementation also reduces the image, by box averaging, so that its smaller side is about 256 pixels. The original experiments used luma only, and the paper never says in which color space or encoding the inputs should be. Documented problems: Figure 24. After Wang and Bovik’s demonstration: seven distortions of a 256×256 photograph tuned to the same mean squared error (306.4; PSNR 23.27 dB), with their SSIM. Mean shift 0.9848 (its MSE is 322.9, 5% high, because an 8-bit shift must be a whole number of levels); contrast stretch 0.9247; salt-and-pepper noise 0.7855; translation by one pixel 0.6975; Gaussian noise 0.6416; 8×8 DCT quantization (JPEG-like) 0.5737; Gaussian blur 0.5213. [Code] Figure 24 reproduces the best-known argument against PSNR and for SSIM: one PSNR, seven visibly different images, and an SSIM that separates harmless changes (mean shift, contrast stretch) from damaging ones (blur, blocking). The values are close to the published ones for the Einstein image (contrast stretch 0.928, mean shift 0.987, noise 0.576, JPEG 0.580, blur 0.641 [Wang 2009, Fig. 2, checked against the figure]). The seventh tile is the counter-argument: a translation of one pixel, which no viewer would notice without a reference to flip against, scores 0.70, worse than visible salt-and-pepper noise. The published figure has the same lesson in its last five panels: a slight zoom, shifts and rotations have MSE of 577 to 873 and SSIM of 0.399 to 0.551, the lowest in the figure, while the complex-wavelet variant CW-SSIM gives them 0.916 to 0.933 [Wang 2009, Fig. 2]. Figure 25. (a) PSNR and SSIM between exact renderings of one scene displaced by sub-pixel amounts: 31.42 dB and 0.9585 at a quarter pixel; 25.87 and 0.8554 at half a pixel; 21.19 and 0.6026 at one pixel; 18.36 and 0.3731 at two. (b) A point-sampled half-size copy enlarged with bilinear (24.65 dB, SSIM 0.7007), with cardinal O-MOMS (23.90 dB, 0.7016) and with the unprefiltered B-spline (24.43 dB, 0.6579). (c) Noise confined to the dark half (image A) or the bright half (image B): PSNR on sRGB codes 31.13 and 33.05 dB; on linear light 36.04 and 30.52 dB; RMS error in $L^{\ast}$ 3.03 and 2.14. (d) White- and blue-noise halftones of gray 1/2: both 6.02 dB PSNR; contrast-sensitivity-weighted at 60 ppd, 11.56 against 16.02 dB. [Code] One image and one setting per case. Each panel of Figure 25 isolates one failure. (a) Registration. A quarter-pixel displacement, far below anything visible, costs PSNR down to 31 dB, which in a compression context would be called mediocre quality; half a pixel gives SSIM 0.86. Any comparison of resampling methods must therefore control registration exactly (Section 3.1
), and a metric used to judge warps, super-resolution or generative models must tolerate small shifts or it measures alignment. (b) Sharp but wrong beats nothing; blurry beats sharp. The blurrier bilinear enlargement outscores the sharper O-MOMS one by 0.75 dB, and even the unprefiltered B-spline, the blurriest of the three, beats O-MOMS in PSNR. SSIM rates the first two equal. Squared error punishes a sharp edge in a slightly wrong place more than a soft one, so on aliased input it rewards blur. This is the mechanism behind Figure 9, and behind the general finding that optimizing for PSNR produces soft images (Section 8.8
). (c) Encoding. The ranking of two noisy images reverses with the space in which PSNR is computed. On linear values a code step in the shadows is a tiny number, so shadow noise is nearly free. The $L^{\ast}$ error sides with the encoded-value ranking here. A PSNR is not interpretable without the transfer function of the data; for HDR data the standard practice is to encode with a perceptually uniform curve first (PU21 [Mantiuk 2021b]). On the HDR part of the UPIQ dataset, PSNR on PU21-encoded values reaches a rank correlation of about 0.57 and MS-SSIM about 0.82 (read from the paper’s Figure 6, which is the only place the values appear; ±0.01). (d) Spectrum. Two halftones with identical error energy differ by 4.5 dB once the error is weighted by a contrast-sensitivity curve at one particular viewing distance, and by a different amount at any other (Figure 21). MS-SSIM evaluates contrast and structure at five dyadic scales and luminance at the coarsest, combining them with fixed exponents [Venkataramanan 2021, p. 2]; it is consistently a little better than SSIM. VIF measures the mutual information between reference and distorted images under a Gaussian-scale-mixture wavelet model of natural images, as the ratio of the information extractable from the distorted image to that from the reference; a noise-free contrast enhancement scores above 1 [Sheikh 2006, pp. 4–5 and 11]. FSIM compares phase congruency and gradient magnitude; GMSD the standard deviation of a gradient-similarity map; VSI weights by a saliency model; MAD combines a detection model for near-threshold errors with an appearance model for large ones; NLPD compares images after a Laplacian pyramid
with divisive normalization, an explicit model of early visual processing [Laparra 2016]. SSIMULACRA 2 combines multi-scale SSIM-like, ringing and blur maps in an opponent space with weights fitted to compression datasets, and deliberately downsamples in linear light. Its validation is its author’s own: on a held-out set of 4,292 opinion scores it reaches a rank correlation of 0.89 against 0.79 for Butteraugli, which was itself tuned only around JPEG qualities 90 to 95 (project documentation †). VMAF
fuses VIF-like and detail features with a learned regressor for video at a fixed viewing geometry. None takes the viewing distance or display as input. All improve on SSIM in database correlation on at least some databases (Section 8.7
); each has a characteristic failure when used as an optimization target, catalogued by Ding and co-authors: mean-absolute-error, MS-SSIM and NLPD over-smooth; VIF manufactures local contrast; GMSD darkens (it ignores local luminance); FSIM and VSI produce artifacts; MAD leaves dots [Ding 2021]. Their human ranking of the optimized results is the more useful finding. MS-SSIM and mean absolute error were statistically tied on all four tasks (“quite surprising because MS-SSIM is widely regarded as a much better perceptual IQA model than MAE”); DISTS and LPIPS led for deblurring, super-resolution and compression, while MS-SSIM and MAE led for denoising; GMSD, VSI and FSIM were at or near the bottom everywhere; and “many recent models, despite their impressive abilities to explain existing IQA databases, do not offer additional perceptual gains over MAE (and may even reduce perceptual quality)” [Ding 2021, Fig. 8 and p. 10]. VMAF can be raised by simple contrast enhancement: by 5–6% in one study [Zvezdakova 2019] and by up to 218.8% with adaptive histogram equalization in another, where increases above 50% mostly lowered subjective quality; the hardened variant could still be raised by up to 23.6% [Siniukov 2021]. LPIPS [Zhang 2018] measures the distance between the activations of a pretrained classification network for the two images, with per-channel weights fitted to human two-alternative judgments (484,000 of them). Its central finding was that deep features from almost any network predict those judgments better than SSIM: even a randomly initialized network reaches 64.3% agreement, against SSIM’s 63.1%, and the calibrated versions about 70%, with the human ceiling at 73.9%. DISTS [Ding 2020] adds explicit tolerance to texture resampling by comparing spatial means and correlations of features, so that two different samples of the same grass count as similar. PieAPP learns from pairwise preferences; DreamSim [Fu 2023] targets mid-level similarity, which is not fidelity. Weaknesses: For generated images without a reference, FID compares feature statistics of two image sets. It is sensitive to the resizing function and to JPEG settings [Parmar 2022], can be lowered by matching class histograms without improving the images [Kynkäänniemi 2023], and in a large study “no existing metric strongly correlates with human evaluations” [Stein 2023]. No-reference indices (NIQE, PI) have near-zero correlation on GAN outputs (0.016 and 0.019 on PIPAL). The visibility of a spatial pattern depends on its frequency in cycles per degree at the eye, its contrast, the luminance to which the eye is adapted, where on the retina it falls, and what else is nearby. A picture file contains none of these. The same file shown on a phone at arm’s length and on a monitor at 50 cm presents the same pixel error at different retinal frequencies, and an error visible in one case is invisible in the other [Andersson 2020]. A measure that takes only two arrays of numbers cannot be a perceptual metric in any strict sense; it can at best be calibrated to the viewing conditions of the dataset it was fitted on. FovVideoVDP’s comparison table records that PSNR and MS-SSIM accept neither photometric nor geometric input [Mantiuk 2021a]. Figure 26. The Campbell–Robson chart: a sinusoidal luminance grating whose frequency rises logarithmically from left to right (1/256 to 1/4 cycle per pixel) and whose contrast falls logarithmically from bottom (1.0) to top (0.003). Contrast is constant along every horizontal line, yet the stripes appear to end along an inverted-U boundary: the viewer’s own contrast sensitivity function. At 60 ppd the horizontal midpoint is 1.87 cycles per degree and the right edge 14.97; stepping back moves the peak of the boundary to the left. View unscaled, on a display with a correct transfer function. Rendered through the sRGB encoding with triangular dither so that sub-code contrasts survive. [Code] Figure 26 lets the reader see the central fact. The contrast sensitivity function (CSF) is the reciprocal of the threshold contrast as a function of spatial frequency: band-pass for luminance, peaking at a few cycles per degree, and falling steeply above. What is established, with sources: Metrics built on these facts take the display’s size, resolution, peak luminance and the viewing distance as inputs and report in visibility units: These are the only class with a principled claim to the word perceptual, and they have limits of their own: they are calibrated on particular datasets, they model early vision (detection and masking) and not preference or semantic plausibility, and they are slower and less convenient as loss functions. A cheaper way to account for viewing distance is to rescale the images to the retinal resolution and then apply an ordinary metric. On three datasets that approach was statistically indistinguishable from the CSF-based metrics, and neither captured the measured change of quality with distance: “accounting for viewing distances requires better models” [Hammou 2023, p. 4; the paper’s abstract puts it more favorably to the CSF-based metrics]. A Fourier analysis by the same group shows the two approaches agree for high-frequency distortions when the viewer moves back and disagree when the viewer moves closer or the distortion is low-frequency [Hammou 2024 †]. Display brightness matters as well [Mikhailiuk 2021 †]. SRCC is the Spearman rank correlation
with human scores on LIVE / CSIQ / TID2013 from [Ding 2020, Table 1] unless noted. Correlations such as those above are fitted to databases of a few dozen reference images with a fixed list of synthetic distortions (noise, blur, compression) rated under one viewing condition. They establish that a metric orders those distortions on those images as people did. The table shows how little that transfers: a metric’s rank changes from database to database (VIF is among the top three on LIVE and next to last on TID2013; LPIPS falls below PSNR there), and the ordering collapses on distortion types absent from the fitting data: geometric changes (SSIM 0.190), GAN outputs (best 0.55), enhancement (VMAF), tone mapping
, high-fidelity compression near the visibility threshold, where behavior “is not well understood” [Mohammadi 2025]. The human ceiling is also lower than usually assumed: observers agree with each other 73.9% of the time on BAPPS and only 67–69% on the outputs of real algorithms [Zhang 2018], and in Rocha’s filter study individual subjects contradicted their own earlier choice in 33% of repeated pairs [Rocha 2020]. A separate line of work shows that the conflict between error and appearance is structural. Blau and Michaeli define perceptual quality as the distance between the distribution of outputs and the distribution of natural images, and prove that for any non-invertible degradation the achievable distortion and perceptual index trade off along a convex, monotone curve, for every distortion measure [Blau 2018]. The minimum-MSE estimate is the posterior mean, an average over all plausible originals, and averages are blurry; an output that looks like a real image must commit to one plausible original and pay for it in expected error. The cost is bounded: in lossy compression, perfect perceptual quality is attainable at the same rate with at most twice the MSE, that is 3 dB [Blau 2019], and the factor of two is exact for the lowest achievable MSE in the setting of [Yan 2021]; in restoration, sampling from the posterior has exactly twice the minimum MSE, while the least MSE compatible with perfect perceptual quality is typically strictly smaller [Ohayon 2025]; with Wasserstein-2 as the perceptual index the trade-off curve is exactly quadratic [Freirich 2021]. Two corollaries with practical bite: near-perfect perceptual quality together with consistency forces a restoration method to become arbitrarily sensitive to its input [Ohayon 2024], and it entails uncertainty of at least twice the inherent level, the theoretical form of “hallucination” [Cohen 2024]. PIPAL shows the trade-off in data: moving from an MSE-trained super-resolution network to a GAN-trained one lowered PSNR by about 2.6 dB while raising mean opinion scores [Gu 2020]. For this article’s subject the consequence is direct. A linear resampling kernel chosen to minimize squared error will lean toward blur on real (aliased, noisy) data, and a kernel that looks sharper will score worse. Neither number nor eye is wrong; they answer different questions. Reasonably well established. Detection thresholds for simple patterns as a function of frequency, luminance, color direction, eccentricity and size; the existence and rough form of contrast masking; the compressive response to luminance; that error energy alone does not predict visibility; that small misregistration is invisible but ruinous to pixel metrics; that every fitted metric can be gamed when optimized against. Not established. How threshold models extend to clearly visible distortions and to preference; how to pool local visibility into one judgment of quality; how to score plausible-but-wrong detail from generative methods; (first attempts appeared in 2025 and 2026: a language-model-based hallucination score, a dataset of semantic-fidelity ratings for super-resolution, a benchmark of 152,000 expert preferences over generative restoration, and synthetic hallucinations for testing; all report that existing metrics track these judgments weakly [Ren 2025; Rocafort 2025; Liu 2026; Kim 2025; abstracts only †]); whether any existing metric is reliable below one just-noticeable difference
; what metric suits multi-color halftones on reflective displays. For kernels specifically: no dataset in the material gathered measures human preference among classical resampling kernels beyond Mitchell and Netravali’s nine observers and Rocha’s 35. Common misstatements. A practical position: report PSNR for what it is (distance, with the encoding stated); never rank across distortion types with PSNR or SSIM; for visibility questions use a metric that takes pixels per degree and luminance; for texture-heavy or generative comparisons add DISTS or LPIPS and look at the images; and treat any single number as a screening device. What follows is partly a survey of what exists and partly synthesis; the synthesis is marked. Display-and-eye-referred prefilters. Kajiya and Ullner (1981; cited in [Nehab 2014, p. 11], not read) chose pixel values for a CRT’s Gaussian spot under a perceptual norm with non-negative coefficients. Rocha, Oliveira and Gastal do the modern version: the reconstruction kernel is the display’s box convolved with the eye’s point spread at a stated viewing distance, and the prefilter is the dual of that kernel, so the image on the retina is the least-squares approximation of the original [Rocha 2020]. Their measures, normalized to conventional filters, show what the criterion buys and costs: sharpness 1.514 for SBS3 against 1.010 for Mitchell–Netravali and 1.162 for the sinc; aliasing 0.451 against 0.172 and 0.168; ringing 0.074 against 0 and 1.000. In a study with 35 subjects, SBS3 was preferred to Mitchell–Netravali in 363 of 455 comparisons (18 the other way), and to a quasi-interpolator and to Öztireli–Gross downscaling by smaller margins, with one third of repeated judgments inconsistent. So observers preferred the filter with more than twice the aliasing measure, because it is sharper. Their appendix adds an observation that undercuts the Nyquist picture for displays: a cosine at 19/40 cycle per pixel shown at 1:1 visibly “beats”, because display plus eye is not a sinc reconstructor. Whether pixel values are treated as linear light in their model is not stated. Perceptual downscaling. Öztireli and Gross pose downscaling as maximizing SSIM between the original and the result and obtain a closed-form, non-linear filter built from local means and variances [Öztireli 2015]. With $\mu_h$ and $\sigma_h$ the mean and standard deviation of an input patch, $l_i$ the box average of the input pixels that map to output pixel $i$, and $\sigma_l$ the standard deviation of those averages over the patch, the SSIM-optimal output is $d_i=\mu_h+(\sigma_h/\sigma_l)(l_i-\mu_h)$, averaged over the overlapping 2×2 patches [Öztireli 2015, eqs. 7–8]. It is an adaptive unsharp mask on the box-averaged image: local contrast lost to averaging is restored by the factor $\sigma_h/\sigma_l$. Trentacoste and co-authors instead measure and preserve the apparent blur (Section 5.1
). Two other adaptive downscalers are known here only at second hand †: Kopf and co-authors align the kernel with local image features, which sharpens without ringing but can leave jagged edges, and Weber and co-authors give more weight to pixels that differ from their neighborhood, to keep small details [Öztireli 2015, p. 1; Liang 2024, p. 2]. Halftoning. The most developed case of optimizing for the viewer: model-based halftoning and direct binary search minimize error after an eye filter (Section 7.8
), blue-noise design targets the eye’s low-pass character, and the color versions weight luminance and chrominance errors separately. Chizhov and co-authors carry the same models into Monte Carlo rendering, distributing rendering error as blue noise in screen space [Chizhov 2022]. Perceptual quantization curves. The sRGB and PQ transfer functions are perceptual designs: they place code values at roughly equal visibility steps (Section 7.2
). PU21 does the same for computing metrics on HDR images [Mantiuk 2021b]. Perceptual losses and codecs. Mixed losses such as $\alpha\,\mathcal L^{\text{MS-SSIM}}+(1-\alpha)\,G\cdot\mathcal L^{\ell_1}$ with $\alpha=0.84$ [Zhao 2017], feature losses [Johnson 2016], divisive-normalization transforms in learned compression [Ballé 2017], and psychovisually tuned JPEG encoders (Guetzli, Jpegli) optimize against a perceptual measure instead of MSE. The evidence that this works is mixed, which is itself the lesson. For Jpegli, a pairwise study found it preferred 54% of the time at 2.8 bits per pixel against libjpeg-turbo at 3.8 and MozJPEG at 3.5 [Bruse 2024, abstract; a study by the encoder’s authors]. For the psychovisual rate–distortion mode of the x264 video encoder, long defended on the strength of its looks, whose authors’ rationale was that the eye wants an image of “similar complexity” and not merely a similar image (developer forum post †), an independent subjective test found that “the perceptual quality gain of Psy-RD ON versus Psy-RD OFF cases is negative on average” and that it “should be used with caution” [Duanmu 2017, abstract and p. 3]. A perceptual tool tested against people and found wanting is rare in print and worth remembering. Learned losses informed by masking models continue the line [Çoğalan 2025 †]. Human-vision-constrained super-resolution spends computation only where the eye can resolve the result, reducing the operation count by a factor of two or more without loss in a user study [Karpenko 2025]. The same reasoning allocates shading rate and refresh rate from a model of texture, motion, luminance and angular resolution [Jindal 2021 †], and temporal contrast-sensitivity models predict where flicker from variable refresh rates becomes visible [Cai 2025b †]. The generalized sampling framework has three parts: a space (the generator), a criterion (the $L_2$ norm), and algorithms that are cheap because both are shift-invariant and linear. A perceptual version must replace the criterion, and the difficulty is how much of the rest survives. No source in the material gathered gives: an error-kernel theory with a perceptual weight and its optimal kernels; a derivation of the space in which to interpolate (Section 6
rests on a heuristic); a treatment of ringing as an appearance phenomenon with a visibility model (the handbook chapter’s remark that ringing’s “perceptual counterpart is the Mach bands
phenomena” is as far as it goes); a perceptual account of the trade between aliasing and sharpness that explains Rocha’s user preferences; or a validated metric for comparing resampling kernels at stated viewing conditions. The question “does a sampling theory for perception exist?” was searched specifically in the second pass (about six queries). No paper was found that derives a reconstruction or prefilter kernel from a contrast sensitivity function or any other visual model for general resampling. The nearest are Öztireli and Gross, Trentacoste and co-authors, the viewing-distance studies of Section 8.6
, and the perceptual allocation of sampling rate in rendering [Jindal 2021 †]. The list above therefore probably describes the literature, on thin search evidence. Work from the last few years that bears on the questions above, with the result that matters in each case. Items marked † were read at the level of the abstract. Kernels, splines and resampling. Exact B-spline interpolation with boundary control and degrees up to 16 (Briand and Monasse, IPOL 2018) and its GPU implementation (Briand and Davy, IPOL 2019). The optimal shift for shifted-linear interpolation (Sadeghi, Yu and Wang, 2018). Catmull–Rom from four bilinear fetches (Csébfalvi 2018), and hardware support for high-order interpolation where the texture needs it (Lin, Seiler and Yuksel 2021). Display-referred prefilters (Rocha, Oliveira and Gastal 2020; Section 9.1
). Karpov designs piecewise-polynomial kernels symbolically to minimize staircasing; his optimized kernels beat Keys’ kernels of the same support, and across twelve metrics no single kernel wins: one of his under MS-SSIM and GMSD, the cubic B-spline under SSIM and LPIPS, Lanczos-3 under DISTS [Karpov 2021, pp. 11 and 14], which is Section 8
’s point in miniature. Costella shows that the “Magic Kernel” is the quadratic B-spline and that its sharpened form is one member of a sequence beginning with nearest neighbor and linear interpolation [Costella 2021, p. 1]. Learned resampling functions predict a steerable anisotropic kernel per pixel, usable at any scale (Li and co-authors 2023, 2024 †). Occorsio and co-authors’ Lagrange–Chebyshev resizing (2021) reports “infinite PSNR” on a downscaling test that undoes a bicubic upsampling, which says nothing about downscaling quality. Nothing from 2019–2026 was found that extends Nehab and Hoppe’s or Sacht and Nehab’s framework. Aliasing in learned systems. Neural networks rediscovered the prefilter. Temporal upscalers in games are a resampling problem with history. Karis’s temporal antialiasing replaces the box filter, which “is not stable under motion”, with a Gaussian-like reconstruction filter, reprojects the previous result, and clips it to the color range of the current neighborhood [Karis 2014, slides 23 and 34]. Catmull–Rom is commonly used for the reprojection, to avoid accumulating blur [Yang 2020, p. 9]. FSR 2 resamples with Lanczos and clamps the ringing [Riley 2022, slide 12]. DLSS publishes neither its kernel nor its rule for rejecting history. Texture filtering. Stochastic texture filtering chooses one texel per lookup with probability given by the filter weights and filters after shading, which is more accurate than filtering before it and makes high-order filters cheap: one voxel lookup replaces 64 for a tricubic filter, with a render 2.8 times faster in the authors’ volume scene [Fajardo 2023, Fig. 1; Pharr 2024]. Later work shares texel choices among neighboring pixels in magnified regions, recovering 10 to 12 dB of PSNR over plain stochastic filtering [Wronski 2025, Fig. 1], and shares decoded texels among pixels that need the same ones [Akenine-Möller 2025]. Dithering, blue noise and halftoning. Perceptual error optimization for Monte Carlo rendering (Chizhov and co-authors 2022) and blue-noise distribution of rendering error through the sampler (Heitz, Belcour and co-authors 2019, 2021). Spatiotemporal blue-noise masks (Wolfe and co-authors 2022; Section 7.11
). Gaussian blue noise (Ahmed and co-authors 2022 †) and filter-adapted spatiotemporal sampling (Donnelly and co-authors 2024 †). The mathematics of halftoning as sigma-delta quantization (Krahmer and Veselovska 2024; Section 7.7
). Non-subtractive dither with a shaped spectrum chosen from an input-dependent table (Faza, Leth and Eielsen †), the only recent optimal-dither result found. Learned halftoning by reinforcement learning and by differentiable dot models (Jiang and co-authors 2023, 2024 †), and learned inverse halftoning (Son 2019 †). A survey of forty years of color quantization (Celebi 2023). No error-diffusion or noise-shaping theory paper from 2025–2026 was found. Metrics and perception. Deep-feature metrics and their repairs (2018–2023), DISTS and PIPAL (2020), FLIP (2020), the visible-difference predictors (2021–2024) and castleCSF (2024) are treated in Section 8
, the perception–distortion results in Section 8.8
. Beyond those: Searched and found empty. Work from 2019–2026 extending Nehab and Hoppe’s framework; a controlled study of linear against perceptual interpolation; a kernel derived from a visual model; peer-reviewed studies of frame-rate-control flicker and of six-color e-paper dithering; a formal analysis of dither amplitude in an encoded space; an independent peer-reviewed critique of Oklab. Still not covered. Learned super-resolution and learned downscaling beyond the items above; the 2023–2026 literature on quality metrics, which is large and was sampled, not exhausted; HDR-specific resampling; demosaicing; subpixel rendering beyond two abstracts; and the paywalled classics (Bayer 1973, Floyd–Steinberg 1976, Knuth 1987, Heckbert 1982 and others). These come from writing and testing resamplers and metrics against the published numbers. Each is easy to get wrong and easy to test for. Experiments worth doing that are not in this article: whether generalized-sampling kernels improve three-shear rotation, and the remaining kernels of Nehab and Hoppe’s table (Condat, quasi-interpolators, quintic O-MOMS) in the repeated-transform test. Open questions, in rough order of how tractable they look: A dagger † marks items whose statements, as used here, rest on an abstract, a web page, a secondary source or memory, or which were not re-verified against the primary source. [—] marks sources that were not obtained and are cited from secondary sources. A [PDF] link is a copy hosted here; [DOI], [arXiv] and [link] go to the public source. Every claim in the text that rests on one of these was checked against the paper. Where no page is cited, the check was at the level of the abstract or first page †. Resampling Dithering Metrics and perception4.6 Quasi-interpolators and SBS3
$\varphi$
$Q(z)$, to be inverted
$\beta^1$
$\tfrac1{12}z^{-1}+\tfrac56+\tfrac1{12}z$
$\beta^2$
$-\tfrac7{1920}z^{-2}+\tfrac{67}{480}z^{-1}+\tfrac{233}{320}+\tfrac{67}{480}z-\tfrac7{1920}z^{2}$
$\beta^3$
$-\tfrac1{720}z^{-2}+\tfrac{31}{180}z^{-1}+\tfrac{79}{120}+\tfrac{31}{180}z-\tfrac1{720}z^{2}$
4.7 Three rankings side by side
Kernel
Time (s)
CT
Circular pattern
Lena
Nearest
0.24
19.44
4.96
18.57
Linear
0.36
24.21
7.13
21.98
Keys
0.90
38.74
15.00
28.16
Cubic Schaum
0.92
37.51
14.09
27.61
Quadratic spline
0.95
45.52
19.65
30.56
Quadratic O-MOMS
0.95
46.71
20.37
30.88
Cubic spline
1.17 (0.88 accelerated)
50.77
23.22
31.98
Cubic O-MOMS
1.17 (0.88 accelerated)
62.05
32.76
34.29
Quintic spline
1.64
63.61
35.01
34.81
Quintic O-MOMS
1.67
65.15
39.06
35.40
Sinc, Hamming window, $W=4$
2.33
33.18
12.35
26.50
5. Operations
5.1 Downscaling
tf.image, and only nearest-neighbor in Pillow [Quiring 2023, p. 6]. TensorFlow and OpenCV are susceptible by default, scikit-image only if its antialiasing is switched off, and Pillow’s multi-step downscaling makes it harder to attack [Lohn 2020, p. 7]. Upscaling is not vulnerable, because every source pixel is used [Quiring 2023, p. 10]. With antialiasing on, the attack “would require extensive perturbations to the source image” [Lohn 2020, p. 6]; detection is treated in [Quiring 2023].
5.2 Upscaling
5.3 Round trips
5.4 Translation, rotation and repeated transforms
5.5 Warps and texture filtering
6. Linear light versus perceptual encoding
6.1 The passage
6.2 The terms
6.3 The prefilter half: averaging is something light does
6.4 The reconstruction half: interpolation is a guess, judged by eye
6.5 Coverage and text: the prefilter rule in rasterization
6.6 Both stages together
6.7 Other practical consequences
6.8 Where the advice is debatable or incomplete
6.9 A working rule
Operation
What it models
Space
Downscaling prefilter, mip-map construction, supersample
resolve
Light averaged over an area
Linear
Blur, glow, defocus, motion blur
Light spreading
Linear
Coverage blending, alpha compositing
Area fractions of light
Linear (alpha itself is never encoded)
Halftone error diffusion
to few levels
The eye averaging dots
Linear (Section 7.9
)
Interpolation for magnification of about 2× or more, viewed resolved
A guess, judged by eye
Perceptual encoding (sRGB as the practical stand-in)
Interpolation at factors near 1 (rotation, sub-pixel shift), or viewed unresolved
Partly a guess, partly an average
Debatable; linear is defensible, and required if energy must be conserved
Quantization to 8 or 10 bits, with sub-step dither
A judgment of visible steps
Perceptual encoding (Section 7.2
)
Error metrics
A judgment
Never raw linear values; see Section 8
7. Quantization, dithering and noise shaping
7.1 What quantization error is
7.2 Banding and bit depth, in linear and encoded values
7.3 Dither for multi-bit quantization
trunc(f + rnd)).7.4 Thresholding against a pattern: ordered dither, noise functions, blue-noise masks
fract(52.9829189 * fract(0.06711056*x + 0.00583715*y)) (Jimenez’s 2014 SIGGRAPH course talk, which exists only as slides; the formula is as quoted and analyzed in a 2022 post by Wolfe †), designed so that every 3×3 neighborhood holds nearly evenly spaced values; Kolås’s a-dither is ((x ^ y*149) * 1234 & 511) / 511.0 [Kolås 2013]. Wronski’s comparison ranks them “blue noise
> IGN > white; avoid ordered Bayer” after low-pass filtering [Wronski 2016].
7.5 What “blue noise” means
7.6 Error diffusion
* the current pixel and the scan running left to right:
Kernel
Weights
Divisor
Status
Floyd–Steinberg
* 7 / 3 5 116
verified
Jarvis–Judice–Ninke
* 7 5 / 3 5 7 5 3 / 1 3 5 3 148
verified
Stucki
* 8 4 / 2 4 8 4 2 / 1 2 4 2 142
verified
Shiau–Fan
* 8 / 1 1 2 4 0 (second row starting three pixels to the left)16
verified against the patent
Burkes
* 8 4 / 2 4 8 4 232
†
Sierra (three-row)
* 5 3 / 2 4 5 4 2 / 0 2 3 2 032
†
Sierra Lite
* 2 / 1 14
†
Atkinson
* 1 1 / 1 1 1 / 0 1 08; only 6/8 of the error is propagated
†
7.7 Error diffusion as two-dimensional noise shaping
7.8 Model-based and optimization-based halftoning
7.9 Dithering in linear versus encoded values
err += toLinear(out) - lin, and choose the output level whose linear value is nearest. No peer-reviewed analysis of dither amplitude through a nonlinear encoding was found, so the amplitude argument that separates this rule from the one in Section 7.3
is this article’s own, supported by Figures 17 and 22 only. The nearest formal results are for other settings: subtractive dither applied in a companded domain [Akyol 2012 †], and the optimal variance of a small dither for a quantizer whose output is later inverted [Eielsen 2019 †]. On displays, a random dither of one step at 60 Hz “has no significant impact on contrast threshold and is not visible” [Allard 2008, abstract]. With more output levels the error shrinks, because the levels bracket the signal more closely and the encoding is nearly linear over each interval; at 256 levels it is negligible, which reconciles this figure with the rule of Section 7.3
. The crossover is a matter of degree: in Figure 22 four levels still give errors up to 9.6 $L^{\ast}$ in the shadows.7.10 Color, palettes and e-paper
Step
Recommended space
Source
Carry and diffuse the error
Linear (XYZ or linear RGB)
“(Lab space is not suitable for diffusing errors due to its nonlinear variation with intensity.)” [Damera-Venkata 2003]; the XYZ-to-Lab transform “does not preserve the spatially averaged tones” [Baqai 2005]
Shape the error by visibility
Opponent luminance–chrominance space built linearly from XYZ, with separate luminance and chrominance sensitivity filters (“more chromatic error than luminance error is allowed at lower spatial frequencies”)
[Baqai 2005; Kolpatzik 1992]
Choose the output color
Perceptual distance is plausible; one cited study found vector error diffusion in CIELAB best for color accuracy, which is in tension with the first row
Haneishi, cited in [Baqai 2005]
Design the palette (color quantization
)
No agreement; gamma-encoded RGB is “more uniform than the linear” and there is “anecdotal evidence that performing cq in an alternative color space does not necessarily give better results”
[Celebi 2023, p. 16]
7.11 Dithering over time
5.588238 * (frame % 64)) to the coordinates; a blue-noise mask can be advanced by adding multiples of the golden ratio modulo one, which keeps each frame blue but makes the temporal sequence per pixel well distributed [Wolfe 2017]; and spatiotemporal blue-noise masks are designed to be blue in space within each frame and across time at each pixel [Wolfe 2022]. Their construction is void-and-cluster with a Gaussian energy over $x$, $y$ and time in which pixels at different positions do not interact across time; each frame keeps a blue-noise spectrum, which a three-dimensional blue-noise mask does not, and each pixel’s sequence converges faster under temporal accumulation than independent masks or a golden-ratio offset [Wolfe 2022, pp. 2 and 5]. In Monte Carlo rendering the same idea is applied to the sampler instead of a mask: Heitz and co-authors optimize the per-pixel scrambling and ranking keys of a low-discrepancy sampler so that the rendering error itself is distributed as blue noise in screen space, for arbitrary sample counts [Heitz 2019, the two-page SIGGRAPH talk; Belcour 2021]. Their journal paper on permuting pixel seeds between frames (CGF 38(4), 2019) is not obtained. Display-side temporal dithering (frame-rate control
in LCD panels) was searched in the second pass and no peer-reviewed study of its visibility was found. Patents state the working assumptions: flicker sensitivity is greatest near 15 Hz (range roughly 4 to 30 Hz), so a 16-frame pattern at 60 Hz (3.75 Hz) is less visible than a 4-frame one, and the flicker shows most in dark tones (US 7,483,039 and 6,982,722 †). Subfield driving and HDR-specific dither were not searched.
8. Judging results: metrics and perception
8.1 PSNR
8.2 SSIM
8.3 Measured failure cases
8.4 Relatives of SSIM and other hand-designed metrics
8.5 Deep-feature metrics
8.6 Vision-model metrics, and why viewing conditions are not optional
8.7 A comparison table
Metric
What it models
Viewing conditions as input
Characteristic failure
SRCC
PSNR
Error energy
No
Equal MSE, different appearance; shifts; encoding dependence
0.873 / 0.810 / 0.687
SSIM
Local mean, variance, correlation
No
Close to MSE; shifts; dark regions; implementation-dependent
0.948 / 0.865 / 0.727
MS-SSIM
SSIM over five scales
No
Insensitive to uniform bias; over-smooths as a loss
0.951 / 0.906 / 0.786
VIF
Mutual information under a scene model
No
Rewards contrast enhancement
0.964 / 0.911 / 0.677
FSIMc
Phase congruency and gradient
No
Artifacts as a loss
0.965 / 0.931 / 0.851
GMSD
Gradient-similarity deviation
No
Ignores local luminance
0.960 / 0.950 / 0.804
NLPD
Laplacian pyramid, divisive normalization
No
Over-smooths texture
0.937 / 0.932 / 0.800
LPIPS
Deep-feature distance
No
Adversarially fragile; shift-sensitive; weak on TID2013
0.932 / 0.876 / 0.670
DISTS
Deep structure and texture statistics
No
Cannot flag hallucinated texture
0.954 / 0.929 / 0.830
PieAPP
Learned pairwise preference
No
Bound to its training data
0.919 / 0.892 / 0.876
VMAF
Fused features, learned regressor
Fixed (1080p at three picture heights)
Inflated by enhancement
TID2013 0.744 †
SSIMULACRA 2
Multi-scale structure, ringing, blur in an opponent space
No (1:1 viewing assumed)
Tuned to compression artifacts
TID2013 0.819 †
FLIP
CSF-filtered color difference and features
Yes (ppd)
Built for flip comparisons
not transcribed
HDR-VDP-3
Optics, adaptation, CSF, masking
Yes
Achromatic
error 0.957 JOD
ColorVideoVDP
Chromatic spatiotemporal CSF, masking
Yes
Calibrated on its training sets
error 0.661 JOD
FID
Feature statistics of image sets
No
Resizer and JPEG sensitive; class bias
not applicable
8.8 What the datasets establish, and the perception–distortion trade-off
8.9 Known, unknown, and commonly misstated
Claim
Status
“SSIM is a perceptual metric”
It is a structural statistic with no viewing model; analytically tied to PSNR for common distortions; 63.1% agreement on BAPPS against 63.2% for $L_2$
“Higher PSNR looks better”
True within one algorithm family at best; contradicted by PIPAL, by the perception–distortion theorems, and by Figures 9 and 25
“LPIPS is a metric” or “LPIPS is robust”
Neither; below PSNR on TID2013; adversarially fragile
“The eye cannot resolve more than 60 ppd”
Mean foveal limit 94 ppd achromatic [Ashraf 2025]
“Blue noise is less visible than white noise”
Only where the display is fine enough: 0.1 dB at 20 ppd, 13 dB at 120 ppd in Figure 21
“CIEDE2000 is a smooth function of its inputs”
It has three discontinuities; the largest, from the mean-hue computation for colors about 180° apart in hue, reaches 0.27 $\Delta E_{00}$ at chroma 2.5: “relatively small, although not negligible” [Sharma 2005]
“Oklab is a better color-difference measure than CIEDE2000”
Oklab was fitted for appearance uniformity (lightness, chroma, hue), not for discrimination. On the COMBVD set of 3,813 pairs its STRESS is 47.35 against 29.13 for CIEDE2000 (lower is better), though it does better on lightness-dominated pairs [Uchida 2026, single-author preprint †]
“A database correlation of 0.95 means the metric is trustworthy”
On that database’s distortions
“The cubic B-spline should be avoided because it blurs”
A consequence of omitting the prefilter [Thévenaz 2000b]
“sRGB is perceptually uniform”, “MSE on gamma-encoded values is perceptual”
Half-truths: the encoding is roughly lightness-uniform, which helps (Figure 25c), but uniformity of lightness steps is a small part of visibility. Not researched further for this article
9. Designing for perception instead of signal error
9.1 What exists
9.2 What a principled theory would need (synthesis)
9.3 What is missing
10. Recent research, 2018–2026
11. Lessons from implementing and testing these methods
12. Conclusions and open questions
References
Generalized sampling, kernels, interpolation
Texture filtering, warping, pixels, compositing
Quantization, dithering, halftoning
Metrics and perception
Additional sources
interpolate documentation; johncostella.com/magic.
Sources added in the final pass
Code
Comments