AIC2026 Exposes Metric Disagreement in Image Coding

High-fidelity image compression is now limited less by whether a codec can save bits than by whether evaluators can measure small perceptual differences consistently. Conventional codecs and learning-based codecs often fail in different ways, which makes a single aggregate score a poor proxy for visible quality. This paper introduces AIC2026, a dataset built for fine-grained assessment of image coding rather than broad, coarse ranking.

The dataset contains 70 source images selected from 2,787 candidates, then expanded into 9,618 distorted images using eight conventional and four learning-based codecs. The central design choice is controlled perceptual spacing: for each source-codec pair, the authors target 20 distortion levels from roughly 0.2 to 4.0 just-noticeable-difference units using ColorVideoVDP as the distortion estimator. That makes the benchmark useful for a narrower question than many compression datasets ask: whether codecs and image quality assessment methods can resolve small quality differences near visually plausible operating points.

Core Contribution

AIC2026 is best read as infrastructure for the next round of codec evaluation, especially JPEG AIC-4. Its main contribution is not a new compression method, but a deliberately sampled testbed where source content, codec family, bitrate, and perceptual distortion are varied in a controlled way.

Figure 1 situates the dataset between the prior JPEG AIC-3 assessment scope and the planned AIC-4 objective-metric evaluation, where the need is not just more images but better coverage of difficult perceptual cases.

The source-image selection is more careful than random sampling. The authors use semantic clustering over deep visual embeddings, inter-metric disagreement among objective IQA methods, and manual inspection to choose images that cover content diversity and stress current metrics. Figure 2 shows the UMAP projection used to inspect candidate coverage before settling on the final set.

Technical Approach

The construction pipeline starts from 2,787 candidate images and selects 70 sources. The paper reports that each source is encoded with seven codecs, drawn from a pool of 12 codecs: eight conventional systems and four learning-based systems. Across those codec-source combinations, the authors generate decoded images at 20 perceptually spaced levels rather than at arbitrary bitrates.

That distinction matters. A bitrate sweep can over-sample parts of the curve where visual quality barely changes and under-sample the region where small artifacts become visible. AIC2026 instead maps candidate encodes to ColorVideoVDP JND scores and chooses points along the distortion-rate curve. Figure 4 shows the achieved distribution of CVVDP-mapped scores: the target levels are marked separately from the medians, which makes clear that the spacing is approximate rather than perfectly controlled.

The paper also includes 840 × 944-pixel crops for each source and compressed image. Those crops are intended to support side-by-side inspection on standard Full HD displays and future crowdsourced studies. This is a practical choice: many codec artifacts are local, and full-resolution viewing can hide small failures unless the display protocol is constrained.

Results and Analysis

The empirical analysis evaluates 24 conventional IQA methods and 12 learning-based IQA methods on the constructed image set. The main result is disagreement, not a single winning metric. The paper reports substantial divergence among objective methods for fine-grained quality differences, with the problem more pronounced for artifacts from learning-based codecs. That finding is plausible given the dataset design: neural codecs can produce texture hallucination, smoothing, and semantic distortions that do not follow the blocking or ringing assumptions implicit in older metrics.

The rate-distortion examples make the issue concrete. The paper shows source-level cases where different IQA methods can imply different codec rankings on the same content, with reported metric-disagreement values such as 0.137 and 0.252. It also shows why sampling from a Pareto front needs judgment: for one JPEG AI BOP example, the highest-bitrate candidate is excluded because it gives only a small perceptual gain at high bitrate. That is the right editorial signal for a dataset paper: more bits are not automatically a better benchmark point if the perceptual change is negligible.

The evidence supports the paper’s narrower claims well. AIC2026 is a substantial public dataset with a documented selection pipeline, multiple codec families, dense distortion sampling, and an objective analysis across 36 IQA methods. It does not yet provide the final perceptual ground truth that would settle which objective metrics best match human judgments. The conclusion states that a large-scale crowdsourced subjective study, following the JPEG AIC-3 methodology, is planned. Until then, AIC2026 is strongest as a controlled benchmark for probing metric disagreement and preparing subjective evaluation, not as a completed human-opinion dataset.