Results per Dataset (Reproduction error)

This page provides the Reproduction Angular Errors of several illuminant estimation algorithms for different datasets. Reproduction Angular Error was proposed by Graham D. Finlayson and Roshanak Zakizadeh in British Machine Vision Conference (BMVC) 2014. The paper can be downloaded from here.

Data sets:

SFU Laboratory Images

Note that illuminant estimates are available for a wide range of parameter-settings (i.e. Minkowski-norm is varied between 1 and 15, and the smoothing value ? is varied between 1 and 12).

Method (applied to 321 images)  Mean error  Median error  Trimean error  95% quantile error
Grey-World: 10.1o 7.5o 8.3o 28o
White-Patch: 9.7o 7.4o 8.2o 27.3o
Shades-of-Grey: (p = 7) 6.9o 3.9o 4.8o 22.5o
general Grey-World (p = 10, ? = 4) 6.0o 3.9o 4.3o 20.2o
1st-order Grey-Edge (p = 14, ? = 4) 6.3o 3.6o 4.2o 21.8o
2nd-order Grey-Edge (p = 15, ? = 10) 5.8o 3.0o 3.8o 24.7o
Pixel-based Gamut (? = 4) 4.2o 2.8o 3.0o 14.8o
Edge-based Gamut (? = 2) 4.5o 2.7o 3.2o 14.3o
Union-based Gamut (? = 2) 5.2o 3.4o 3.9o 15.3o
Intersection-based Gamut (? = 3) 4.2o 2.5o 2.8o 17.6o
Inv. Intensity Chrom. Space 15.1o 9.3o 11.5o 43.2o
Spatial Correlations:
— HeavyTailed-based
6.2o 4.1o 5.0o 16.6o
Weighted Grey-Edge:
— p = 2, ? = 1, ? = 8
6.1o 3.6o 4.3o 19.3o

SFU Grey-ball Set (Original)

The original Grey-ball set consists of 11,346 images. However, because of the large correlation among some consecutive frames, various researchers decided to use a subset of this set. Feel free to use the following data to extract any results appropriate, since they are computed by processing the full set:

Method (applied to 11346 images)  Mean error  Median error  Trimean error  95% quantile error
Grey-World 8.7o 7.6o 7.9o 20.9o
White-Patch 7.1o 5.5o 6.0o 18.0o
Shades-of-Grey (p = 14) 6.5o 5.6o 5.8o 15.2o
general Grey-World (p = 12, ? = 0) 7.1o 6.2o 6.4o 16.6o
1st-order Grey-Edge (p = 2, ? = 1) 6.3o 4.8o 5.4o 15.7o
2nd-order Grey-Edge (p = 1, ? = 2) 6.5o 5.0o 5.6o 15.7o
Pixel-based Gamut (? = 2) 7.5o 5.9o 6.3o 19.6o
Edge-based Gamut (? = 1) 7.3o 5.8o 6.3o 18.1o
Intersection-based Gamut (? = 2) 7.5o 5.9o 6.4o 19.6o
Inv. Intensity Chrom. Space 7.0o 6.0o 6.2o 15.9o
Using Natural Image Statistics 5.5o 4.3o 4.7o 13.7o
Exemplar-Based Color Constancy 4.8o 3.7o 4.0o 12.5o
Color Cat 4.6o 3.4o 3.8o 12.6o
— Using WP / GW
4.9o 3.2o 3.8o 13.4o
 — Using ColorCat 5.7o 4.3o 4.7o 15.8o

Color-checker (by Shi)

Since the original color checker set wat generated from RAW data using automatic settings, Shi reprocessed the original RAW data and generated 12-bit PNG-images (with lossless compression).

Method (applied to 568 images)  Mean error  Median error  Trimean error  95% quantile error
Grey-World 7.0o 6.8o 6.9o 12.4o
White-Patch 8.1o 6.5o 7.1o 20.0o
Shades-of-Grey (p = 5) 5.8o 4.4o 4.9o 14.7o
General Grey-World (p = 9, ? = 9) 5.3o 4.0o 4.4o 14.0o
1st-order Grey-Edge (p = 1, ? = 9) 6.4o 4.9o 5.3o 15.8o
2nd-order Grey-Edge (p = 1, ? = 1) 6.0o 4.8o 5.2o 13.7o
Pixel-based Gamut (? = 4) 4.8o 2.7o 3.4o 15.8o
Edge-based Gamut (? = 3, edge type = 3) 8.0o 5.9o 6.6o 20.5o
Intersection-based Gamut (? = 4) 4.8o 2.7o 3.4o 16.4o
Inv. Intensity Chrom. Space 14.3o 13.6o 13.6o 22.6o
Regression (SVR) 8.8o 7.4o 7.9o 18.9o
Bayesian 5.6o 3.9o 4.4o 15.4o
Spatial Correlations:
— Gaussian-based
o o o o
 — HeavyTailed-based 4.2o 3.5o 3.6o 9.9o
Bottom-up (see here) 4.0o 3.0o 3.2o 11.6o
Top-down (see here) 4.3o 3.1o 3.2o 13.8o
Bottom-up+Top-down (see here) 4.0o 2.8o 3.0o 13.6o
Using Natural Image Statistics 4.8o 3.5o 3.9o 12.9o
CART-based Selection 5.2o 4.0o 4.1o 14.6o
CART-based Combination 4.5o 3.5o 3.8o 11.4o
Exemplar-Based Color Constancy 3.4o 2.6o 2.9o 8.3o
Using CNNs:
— AlexNet+SVR
5.5o 3.6o 4.1o 16.3o
 — Deep color constancy 3.1o 2.2o 2.5o 9.0o


ColorChecker (RECommended)

The table below was updated on March 24, 2018.

We re-processed the Gehler data to address the problem raised in G. D. Finlayson, G. Hemrit, A. Gijsenij, and P. Gehler, “A Curious Problem with Using the Colour Checker Dataset for Illuminant Estimation,” in Color and Imaging Conference, 2017, pp. 64–69.

If you use the data and results below then please cite G. Hemrit et al., “Rehabilitating the ColorChecker Dataset for Illuminant Estimation,” in Color and Imaging Conference, 2018. see paper here

Finlayson et al. demonstrated that the previous rankings and performance evaluation of algorithms for the ColorChecker dataset are ill-founded, because of mixing different ground-truths. The following results are obtained using the RECommended ground-truth, calculated on the re-processed images from Gheler’s raw images, according to the calculation methodology described by Shi and Funt.

Information on the re-processed dataset as well as the images, are accessible on the Datasets page. If you have any contribution to bring to this work, feel free to send a message to the contact person of this web-site.

Method (applied to 568 images)  Mean error  Median error  Trimean error  95% quantile error  Max error
Grey-World 10.7o 10.6o 10.7o 17.4o 27.0o
White-Patch 9.8o 8.0o 8.9o 22.3o 32.6o
Shades-of-Grey (p = 5) 8.3o 7.5o 7.8o 17.0o 23.0o
General Grey-World (p = 9, ? = 9) 7.6o 6.7o 7.0o 16.5o 22.9o
1st-order Grey-Edge (p = 1, ? = 9) 5.0o 3.7o 4.1o 13.3o 28.8o
2nd-order Grey-Edge (p = 1, ? = 1) 5.4o 4.5o 4.8o 12.8o 22.6o
Pixel-based Gamut (? = 4) 6.9o 5.2o 5.7o 18.2o 28.9o
Edge-based Gamut (? = 3, edge type = 3) 6.9o 4.6o 5.2o 20.6o 40.1o
Intersection-based Gamut (? = 4) 6.8o 5.2o 5.6o 18.3o 30.1o
Inv. Intensity Chrom. Space 16.3o 12.8o 14.3o 34.0o 38.3o
Regression (SVR) 11.8o 10.3o 10.9o 22.1o 29.8o
Bayesian 6.6o 4.6o 5.2o 18.4o 30.4o
 Spatial Correlations: — HeavyTailed-based 6.8o 5.8o 6.1o 15.9o 21.9o
Bottom-up 6.7o 5.9o 6.1o 14.7o 22.4o
Top-down 7.0o 5.4o 5.7o 17.7o 31.1o
Bottom-up+Top-down 6.6o 5.4o 5.6o 15.8o 31.1o
Using Natural Image Statistics 6.5o 5.5o 5.7o 16.2o 22.7o
CART-based Selection 7.1o 6.1o 6.4o 16.2o 25.3o
CART-based Combination 7.2o 6.6o 6.8o 14.0o 21.3o
Exemplar-Based Color Constancy 6.0o 5.2o 5.4o 13.2o 19.5o
Using CNNs:
— AlexNet+SVR
8.2o 6.3o 6.8o 20.4o 31.6o
 — Deep color constancy 5.7o 4.7o 5.0o 12.4o 18.9o
Fast Fourier Color Constancy (model Q) 2.5o 1.4o 1.8o 8.8o 27.8o