Results per Dataset (Reproduction error)

This page provides the Reproduction Angular Errors of several illuminant estimation algorithms for different datasets. Reproduction Angular Error was proposed by Graham D. Finlayson and Roshanak Zakizadeh in British Machine Vision Conference (BMVC) 2014. The paper can be downloaded from here.

Data sets:

SFU Laboratory Images

Note that illuminant estimates are available for a wide range of parameter-settings (i.e. Minkowski-norm is varied between 1 and 15, and the smoothing value ? is varied between 1 and 12).

Method (applied to 321 images)  Mean error  Median error  Trimean error  95% quantile error
Grey-World: 10.1o 7.5o 8.3o 28o
White-Patch: 9.7o 7.4o 8.2o 27.3o
Shades-of-Grey: (p = 7) 6.9o 3.9o 4.8o 22.5o
general Grey-World (p = 10, ? = 4) 6.0o 3.9o 4.3o 20.2o
1st-order Grey-Edge (p = 14, ? = 4) 6.3o 3.6o 4.2o 21.8o
2nd-order Grey-Edge (p = 15, ? = 10) 5.8o 3.0o 3.8o 24.7o
Pixel-based Gamut (? = 4) 4.2o 2.8o 3.0o 14.8o
Edge-based Gamut (? = 2) 4.5o 2.7o 3.2o 14.3o
Union-based Gamut (? = 2) 5.2o 3.4o 3.9o 15.3o
Intersection-based Gamut (? = 3) 4.2o 2.5o 2.8o 17.6o
Inv. Intensity Chrom. Space 15.1o 9.3o 11.5o 43.2o
Spatial Correlations:
— HeavyTailed-based
6.2o 4.1o 5.0o 16.6o
Weighted Grey-Edge:
— p = 2, ? = 1, ? = 8
6.1o 3.6o 4.3o 19.3o

SFU Grey-ball Set (Original)

The original Grey-ball set consists of 11,346 images. However, because of the large correlation among some consecutive frames, various researchers decided to use a subset of this set. Feel free to use the following data to extract any results appropriate, since they are computed by processing the full set:

Method (applied to 11346 images)  Mean error  Median error  Trimean error  95% quantile error
Grey-World 8.7o 7.6o 7.9o 20.9o
White-Patch 7.1o 5.5o 6.0o 18.0o
Shades-of-Grey (p = 14) 6.5o 5.6o 5.8o 15.2o
general Grey-World (p = 12, ? = 0) 7.1o 6.2o 6.4o 16.6o
1st-order Grey-Edge (p = 2, ? = 1) 6.3o 4.8o 5.4o 15.7o
2nd-order Grey-Edge (p = 1, ? = 2) 6.5o 5.0o 5.6o 15.7o
Pixel-based Gamut (? = 2) 7.5o 5.9o 6.3o 19.6o
Edge-based Gamut (? = 1) 7.3o 5.8o 6.3o 18.1o
Intersection-based Gamut (? = 2) 7.5o 5.9o 6.4o 19.6o
Inv. Intensity Chrom. Space 7.0o 6.0o 6.2o 15.9o
Using Natural Image Statistics 5.5o 4.3o 4.7o 13.7o
Exemplar-Based Color Constancy 4.8o 3.7o 4.0o 12.5o
Color Cat 4.6o 3.4o 3.8o 12.6o
— Using WP / GW
4.9o 3.2o 3.8o 13.4o
 — Using ColorCat 5.7o 4.3o 4.7o 15.8o

Color-checker (by Shi)

Please note that the table below was updated on March 24, 2018, see this paper to know more. The previous results prior the update will soon be available again. Please contact the administrator if you have any questions.

Since the original color checker set wat generated from RAW data using automatic settings, Shi reprocessed the original RAW data and generated 12-bit PNG-images (with lossless compression).

Method (applied to 568 images)  Mean error  Median error  Trimean error  95% quantile error
Grey-World 10.7o 10.6o 10.7o 17.4o
White-Patch 9.8o 8.0o 8.9o 22.3o
Shades-of-Grey (p = 5) 8.3o 7.5o 7.8o 17.0o
general Grey-World (p = 9, ? = 9) 7.6o 6.7o 7.0o 16.5o
1st-order Grey-Edge (p = 1, ? = 9) 5.0o 3.7o 4.1o 13.3o
2nd-order Grey-Edge (p = 1, ? = 1) 5.4o 4.5o 4.8o 12.8o
Pixel-based Gamut (? = 4) 6.9o 5.2o 5.7o 18.2o
Edge-based Gamut (? = 3, edge type = 3) 6.9o 4.6o 5.2o 20.6o
Intersection-based Gamut (? = 4, intersection = 2) 6.8o 5.2o 5.6o 18.3o
Inv. Intensity Chrom. Space 16.3o 12.8o 14.3o 34.0o
Regression (SVR) 11.8o 10.3o 10.9o 22.1o
Bayesian 6.6o 4.6o 5.2o 18.4o
Spatial Correlations:
— Gaussian-based
o o o o
 — HeavyTailed-based 6.8o 5.8o 6.1o 15.9o
Bottom-up (see here) 6.7o 5.9o 6.1o 14.7o
Top-down (see here) 7.0o 5.4o 5.7o 17.7o
Bottom-up+Top-down (see here) 6.6o 5.4o 5.6o 15.8o
Using Natural Image Statistics 6.5o 5.5o 5.7o 16.2o
CART-based Selection 7.1o 6.1o 6.4o 16.2o
CART-based Combination 7.2o 6.6o 6.8o 14.0o
Exemplar-Based Color Constancy 6.0o 5.2o 5.4o 13.2o
Using CNNs:
— AlexNet+SVR
8.2o 6.3o 6.8o 20.4o
 — Deep color constancy 5.7o 4.7o 5.0o 12.4o