Image Resampling, Dithering and Evaluation: Best Practices and Gotchas

Chris Lomont, with Claude Fable 5.2

What this article is. The checklist in a set of three articles on image resampling, dithering and evaluation: what to do for each task, and the errors that are easiest to make.

Related articles. Expert notes: Image Processing Notes . General-audience summary: Image Processing Notes Summary . Companion topics: sampling theory (summary , checklist ); color spaces and HDR (summary , checklist ).

Contents

• Decision table
• Best practices by task
• Common errors and gotchas

Each item gives the rule, one line of why, and a pointer (§ = section of the expert notes, Fig. = figure in the expert notes). “Linear” means values proportional to light; “encoded” means sRGB code values (or another perceptual encoding). Items marked † rest on sources the project could not verify against a primary reference.

Decision table

If you are doing	Use	In which values
Downscaling by any factor	A kernel stretched by the scale factor, weights normalized per output pixel: Lanczos-3, or a cubic B-spline followed by its inverse filter on the output grid	Linear
Building mip-maps or pyramids	The same; at minimum a box, better the cardinal cubic B-spline recipe	Linear
Upscaling a photograph by 2× or more	Prefiltered cubic O-MOMS or B-spline, or Lanczos-3	Encoded (perceptual)
Upscaling text, line art, UI	A kernel with little or no overshoot (bilinear, Mitchell), or a dedicated method	Encoded
Rotating, sub-pixel shifting, mild warps, once	Prefiltered cubic B-spline or O-MOMS; Keys if no prefilter pass is possible	Either; linear if brightness of fine detail must be conserved
Repeated transforms (editing chains, registration loops)	Prefiltered cubic or quintic O-MOMS; never bilinear, Mitchell or an unprefiltered B-spline	Keep one space for the whole chain; float, no clamping between steps
Texture minification under perspective	Mip-maps with anisotropic filtering (or EWA)	Linear (sRGB texture formats)
Blur, glow, defocus, motion blur	Any low-pass	Linear
Antialiased coverage, alpha compositing	Premultiplied alpha, “over”	Linear color; alpha is never encoded
Reducing bit depth (16 or float to 8 or 10 bits)	Triangular dither, 2 steps wide, ideally from a blue-noise mask; add before rounding	Encoded (the space being quantized)
Halftoning to 1 bit or a few levels	Error diffusion (serpentine, perturbed or variable-coefficient) or a blue-noise mask	Error carried in linear
Dithering to a fixed color palette	Vector error diffusion with gamut handling	Error in linear; nearest-color decision may use a perceptual distance
Reporting a distance between two aligned images	PSNR, stating the encoding and peak	Encoded, or PU-encoded for HDR
Asking whether a difference is visible	A vision-model metric with pixels per degree and luminance (ColorVideoVDP, FLIP, HDR-VDP-3)	Calibrated display values
Comparing textured or generated results	DISTS or LPIPS as a screen, then look	As the metric specifies

Best practices by task

Downscale

1. Widen the kernel with the scale factor. A two- or four-tap “bilinear” or “bicubic” at any reduction above about 2 is point sampling: alias within 2 dB of nearest neighbor. §5.1 , Fig. 5.
2. Check what your library does. Default resize calls in several common libraries do not antialias, and they disagree with each other. †(per Parmar et al. 2022) §5.1 .
3. Filter linear values. Averaging code values darkens fine bright detail by 60–90% and a one-pixel checkerboard to 21% luminance instead of 50%. §6.3 , Fig. 11.
4. Normalize the weights at every output sample for non-integer factors and for any kernel that is not a partition of unity (all windowed sincs). §4.5 , §11 .
5. Prefer a prefilter with a flat pass band. Box leaves −11.7 dB of alias and droops 3 dB; tent and Mitchell lose over 5 dB of the top octave; Lanczos-3 and the cardinal B-spline prefilter reach about −24 dB alias with 1.5 dB droop. Fig. 5.
6. For small factors (below about 2) a strong antialiasing filter can cost more than it saves. Interpolation without prefilter was best at 1.5× and 2× in the round-trip test, and worst at 6×. §5.3 , Fig. 7.
7. If the result will be re-enlarged with a known kernel, use the least-squares (projection) reduction for that kernel. It is the ceiling; good prefilters come within 0.2–1.4 dB. §5.3 .

Upscale

8. Use a prefiltered spline, not a bigger interpolating kernel. Cubic O-MOMS and B-spline have order 4 at four taps; Keys has order 3; the cost of the prefilter is a few percent. §2.5 , Fig. 2.
9. Never apply a B-spline to pixel values without its prefilter. It becomes a smoothing filter worse than bilinear (rise width 1.31 pixels against 0.81). §4.3 , Fig. 6.
10. Expect overshoot from every sharp kernel: about 11–12% of a step for Lanczos-3 and cardinal splines, 7.4% for Keys, 3.6% for Mitchell, none for bilinear. Choose by content. Fig. 6.
11. Interpolate encoded values when enlarging visibly. In linear light the undershoot clips to black next to bright features and bright lines look 1.6 times wider than dark ones. §6.4 , Fig. 12.
12. Do not expect a kernel to add detail. Above 2–3× the limit is missing information; single-enlargement scores barely separate good kernels. §4.7 , §5.2 .

Rotate, translate, warp

13. Use half-pixel-center coordinates everywhere, and test with the identity transform. Output must equal input at angle zero and shift zero. §3.1 , Fig. 4.
14. Mirror (half-sample symmetric) boundaries for the spline prefilter, extended far enough for the precision wanted. §3.1 , §4.3 .
15. For chains of transforms, compose the geometry and resample once. If that is impossible, use the highest-quality kernel available: losses compound (31.3 dB for O-MOMS against 23.2 dB for bilinear after 15 rotations). Fig. 8.
16. Keep intermediates in floating point and do not clamp between steps or between separable passes. Clamp once at the end. §6.8 , §11 .
17. For warps that minify anywhere, the footprint must grow with the local scale (mip-map plus anisotropic taps, or EWA). §5.5 , Fig. 10.

Mip-maps and textures

18. Build the chain in linear light. A chain built on code values gets darker level by level (mean luminance 0.46 to 0.30 on the test pattern). Fig. 11.
19. Mark color textures as sRGB so the hardware decodes before filtering; leave data textures (normals, masks, alpha) linear. §6.7 .
20. Use anisotropic filtering for oblique surfaces. Trilinear alone blurs to the long axis of the footprint (far-field contrast 80 against 100 for the reference; 16 taps restore 96). Fig. 10.
21. A better chain than box is cheap: stretched cubic B-spline followed by the inverse filter, about 35 operations per output pixel. §5.1 .

Compositing, coverage, blur

22. Blend coverage and alpha in linear light. On code values, light text on dark emits two thirds of the correct light and dark text on light removes a third too much. §6.5 , Fig. 13.
23. Premultiply alpha before filtering or resampling, and never gamma-encode alpha. §6.7 .
24. Blur in linear light. On code values small highlights keep 23% of their light and saturated complementary colors get a dark seam between them. Fig. 15.
25. Know that “correct” text may not look familiar. Font rendering has been tuned against encoded-space blending; changing the space changes apparent weight. §6.5 .

Quantize and dither

26. Quantize after encoding, not in linear light. 8-bit linear has three codes in the darkest tenth of the lightness range; 8-bit sRGB has 28. About 12 linear bits match 8 encoded. §7.2 , Fig. 16.
27. Dither before rounding, with triangular noise two steps wide. Rectangular noise fixes the mean but leaves grain that comes and goes with the signal. §7.3 , Fig. 17.
28. Match the dither offset to the rounding rule (round: −1 to 1 step; truncate: −0.5 to 1.5), and fade to one-step noise near black and white so clipping does not bias the mean. §7.3 .
29. Take the dither from a blue-noise mask if you can. It changes the spectrum, not the statistics. Independent noise per channel. §7.3 , Fig. 17.
30. For halftones, diffuse the error in linear light. On code values a one-bit halftone is up to 30 $L^{\ast}$ too light; four levels still 10 $L^{\ast}$. §7.9 , Fig. 22.
31. Use serpentine scanning and some perturbation (or variable coefficients). Plain Floyd–Steinberg locks into periodic patterns at simple gray levels. §7.6 , Fig. 19.
32. Know your kernel’s side effects. Larger filters (Jarvis, Stucki) sharpen more; Atkinson drops a quarter of the error and loses shadows and highlights. §7.7 , Fig. 20.
33. For palettes, map or clip colors into the palette’s gamut, or vector error diffusion will smear. Measure the real primaries of the device. §7.10 .

Evaluate

34. Register exactly before computing any pixel metric. A quarter-pixel shift gives 31 dB; one pixel gives SSIM 0.60. Fig. 25a.
35. State the encoding, the peak value, the region scored and the implementation with every PSNR or SSIM. Exclude borders. §8.1 –8.2 .
36. Compare like with like. PSNR and SSIM order results within one kind of distortion on one image; they do not rank across kinds. §8.3 , Fig. 24.
37. Use test sources that are not themselves aliased when comparing interpolators, or know that blur will be rewarded. Fig. 9, Fig. 25b.
38. For visibility, give the metric the viewing conditions. If a metric has no input for pixels per degree, it is calibrated to someone else’s. §8.6 .
39. Look at the images, at 100% zoom, on the intended display. Every metric here has a documented way to be wrong. §8.9 .

Common errors and gotchas

Kernels and resampling

• “Bicubic” is not one filter. Keys with a = −0.5 (Catmull–Rom), a = −0.75, a = −1, Mitchell, and the B-spline are all called bicubic. Only a = −0.5 has third-order accuracy among Keys’ family. §4.2 .
• Mitchell–Netravali is a rendering reconstruction filter, deliberately a little blurry. It is near the bottom of every repeated-resampling ranking. §4.2 , Fig. 8.
• Lanczos does not sum to one. Unnormalized, flat areas ripple and the mean drifts under repetition; this, not the kernel’s shape, explains its worst published results. §1.4 , §4.5 .
• Spline coefficients are not pixel values. Do not display or store them as an image without knowing it; they leave the range of the data. §2.5 .
• “B-splines blur” is a symptom of a missing prefilter. §4.3 .
• Higher order is not always better on real images. Order governs smooth content far below Nyquist; on edges the ranking can invert. §4.7 .
• Bilinear can win PSNR while looking worst (29.2 dB against 24.7 for Lanczos-3 on an aliased pattern). §5.4 , Fig. 9.
• Nearest neighbor with sub-pixel shifts does nothing at all until the shift rounds; a low score may be misregistration, not blur. Fig. 8.
• Align-corners and index-scaling conventions shift or stretch the image by up to half an output pixel. Do not mix libraries without checking. Fig. 4.
• Single precision is enough for cubic prefilters to about 1e-6; high degrees need double. §4.3 .
• Published dB values are not comparable across papers. Protocol differences move them by several dB. §4.7 .
• Excluding the borders matters. Boundary handling alone is worth up to about 2 dB in rotation tests. §3.1 .

Linear versus encoded

• The mean error of gamma-space filtering on photographs is about one code value, which is why it goes unnoticed; the damage is in highlights and fine patterns (up to 26 codes). Fig. 11.
• “Always linear” is right for anything that averages light and not obviously right for interpolation. Strong enlargement in linear light gives dark halos. §6.4 , §6.8 .
• “Reconstruct in sRGB” is a heuristic. Near unit scale or at very fine pixel pitch the argument weakens, and a mixed pipeline is not idempotent. §6.8 .
• sRGB is not a 2.2 power law; use the piecewise function. And it is only an approximation to lightness. §6.2 .
• Luma computed from encoded RGB is not luminance. Gray conversions and metrics inherit the error. §8.2 , §11 .
• A perceptual space is for judging differences, not for arithmetic on light. Nearest-color matching in Oklab without error feedback turned a gray animal red. Fig. 23.
• HDR changes the perceptual curve (PQ † or PU21, not sRGB) and makes some overshoot displayable. §6.8 .
• Averaging HDR samples in linear light aliases on bright outliers. Weight each sample by about 1/(1 + max channel) before the average and undo it after. §6.7 .
• A resize without antialiasing is also a security hole: a second image can be hidden in the few pixels the resizer reads. §5.1 .

Dither and halftone

• Dither added after quantization does nothing. §7.3 .
• Sub-step dither goes in the quantizer’s space; halftone error goes in linear light. The two rules are consistent: it is a matter of amplitude. §7.3 , §7.9 .
• Noise shaping does not reduce error; it raises total error power (about 1.2 dB for Floyd–Steinberg) and moves it. §7.7 .
• Blue noise is only better than white where pixels are small: 0.1 dB at 20 pixels per degree, 4.5 dB at 60, 13 dB at 120. Fig. 21.
• TPDF dither is optimal only in a narrow sense: least added noise among dithers that make the error’s mean and variance independent of the input. It does not make the error independent of the signal. §7.3 .
• Shiau–Fan weights: 8 to the right; 1, 1, 2, 4 on the next row with the 4 directly below the current pixel (sixteenths), per the patent. §7.6 .
• A frequency-weighted error rates Bayer above blue noise. It has no term for visible regular texture. Do not select halftones by that number alone. Fig. 21.
• A 64×64 blue-noise mask suppresses low frequencies by only 8–13 dB, less than error diffusion. Fig. 19.
• Error diffusion sharpens. If you measure “error” against the input you are partly measuring sharpening. Fig. 20.
• Real devices are not linear in dot count. Dot gain on paper and pixel crosstalk on e-paper need a measured model. §7.9 , §7.10 .
• Error-diffusion weights from secondary listings (Burkes, Sierra, Atkinson) are widely copied and were not verified here against originals †. §7.6 .

Metrics

• SSIM is not perceptual: no viewing model, close to MSE for common distortions, 63.1% agreement with people where plain squared error gets 63.2%. §8.2 .
• SSIM implementations disagree (window, downsampling, color, borders). §8.2 .
• PSNR on linear values makes shadow noise nearly free; rankings can reverse with the encoding. Fig. 25c.
• LPIPS is not a metric, is fragile to small shifts and adversarial changes, and falls below PSNR on one standard database. §8.5 .
• Optimizing a network or a filter against any single metric finds that metric’s blind spot. VMAF can be raised by contrast enhancement that viewers dislike. §8.4 .
• As a training loss, MS-SSIM was no better than mean absolute error in human ranking on four tasks. DISTS and LPIPS led on three, MS-SSIM and MAE on denoising. §8.4 .
• “Perceptual” tuning needs a test with people. An independent study found x264’s psychovisual mode lowered subjective quality on average. §9.1 .
• Rescaling images for viewing distance works about as well as a CSF-based metric, and neither tracks distance correctly. §8.6 .
• Oklab is an appearance space, not a color-difference formula; on discrimination data it does much worse than CIEDE2000, which has small discontinuities of its own. §8.9 .
• Higher PSNR and “looks better” trade off in ill-posed problems; this is a theorem, not an accident. §8.8 .
• “The eye resolves 60 pixels per degree” understates it: measured mean 94 for achromatic detail. §8.6 .
• Human judgments are noisy too: observers agree with each other about 74% of the time on one large test, and contradicted themselves in a third of repeated pairs in another. §8.8 .