Results per Dataset (Recovery error)

Until now, only a limited number of algorithms on a limited number of data sets are available. If you have or know of any additions, please let me know.

You can browse the results per data set or per method. The first shows the results of all methods that are available for a particular data set. The second shows the results of one particular method on all data sets.

Data sets:

SFU Laboratory Images

Results on this full set are available for the following low-level methods. Note that illuminant estimates are available for a wide range of parameter-settings (i.e. Minkowski-norm is varied between 1 and 15, and the smoothing value ? is varied between 1 and 12).

Method (applied to 321 images)  Mean error  Median error  Trimean error  Max error
Grey-World: 9.8o 7.0o 7.6o 37.3o
White-Patch: 9.1o 6.5o 7.5o 36.2o
Shades-of-Grey: (p = 7) 6.4o 3.7o 4.6o 29.6o Influence of parameters on angular error
general Grey-World (p = 10, ? = 4) 5.4o 3.3o 3.8o 28.9o Influence of parameters on angular error
1st-order Grey-Edge (p = 7, ? = 4) 5.6o 3.2o 3.7o 31.6o Influence of parameters on angular error
2nd-order Grey-Edge (p = 7, ? = 5) 5.2o 2.7o 3.3o 26.7o Influence of parameters on angular error

 

Results of the Gamut mapping are also available, but this method requires training data. The following results are obtained using 31 images (all images recorded under the “syl-50MR16Q”-illuminant) to construct the canonical gamut. This canonical gamut is then used to estimate the illuminant for all images, so including the training images. This is done to be able to make a comparison with other methods; if you would like to compute the performance of merely the test image, simply download the results and exclude the results of the 31 training images.

Method (applied to 321 images)  Mean error  Median error  Trimean error  Max error
Pixel-based Gamut (? = 4) 3.7o 2.3o 2.5o 27.1o Influence of parameters on angular error
Edge-based Gamut (? = 2) 3.9o 2.3o 2.7o 29.7o Influence of parameters on angular error
Union-based Gamut (? = 2) 4.7o 3.0o 3.4o 27.7o Influence of parameters on angular error
Intersection-based Gamut (? = 4) 3.6o 2.1o 2.4o 27.1o Influence of parameters on angular error

 

Various other methods have been applied to this set. If you would like to have your results reported here, feel free to send a message to the contact person of this web site.

Method (applied to 321 images)  Mean error  Median error  Trimean error  Max error
Inv. Intensity Chrom. Space 15.5o 8.2o 10.7o 80.9o
Spatial Correlations:
— HeavyTailed-based
5.6o 3.5o 4.3o 21.6o
Weighted Grey-Edge:
— p = 7, ? = 5, ? = 8
5.6o 2.4o 2.9o 43.8o

 

The results of Spatial Correlations denoted “HeavyTailed-cbased” is using the newest implementation and can be found in “Color Constancy with Spatio-spectral Statistics”, in IEEE TPAMI. More elaborate results by Chakrabarti et al. can be downloaded here. These results also contain slightly different results for the Grey-Edge-type methods.

 


SFU Grey-ball Set (Original)

The original Grey-ball set consists of 11,346 images. However, because of the large correlation among some consecutive frames, various researchers decided to use a subset of this set. Feel free to use the following data to extract any results appropriate, since they are computed by processing the full set:

Method (applied to 11346 images)  Mean error  Median error  Trimean error  Max error
Grey-World 7.9o 7.0o 7.1o 48.1o
White-Patch 6.8o 5.3o 5.8o 38.7o
Shades-of-Grey (p = 12) 6.1o 5.3o 5.5o 41.2o Influence of parameters on angular error
general Grey-World (p = 12, ? = 0) 6.1o 5.3o 5.5o 41.2o Influence of parameters on angular error
1st-order Grey-Edge (p = 1, ? = 1) 5.9o 4.7o 5.1o 41.2o Influence of parameters on angular error
2nd-order Grey-Edge (p = 1, ? = 2) 6.1o 4.9o 5.3o 41.7o Influence of parameters on angular error

 

When obtaining results of the Gamut Mapping (or any other learning-based method) on the Grey-ball Set, one should carefully take the correlation between consecutive frames into account. We propose to use 15-fold cross-validation. The 15 different folds are constructed by grouping images of the same scene, ensuring the correlated images to be either all in the test set or all in the training set. (Alternatively, one can choose to create a subset where the correlation is eliminated). Results of the 15-fold cross-validation are shown here:

Method (applied to 11346 images)  Mean error  Median error  Trimean error  Max error
Pixel-based Gamut (? = 5) 7.1o 5.8o 6.1o 41.9o Influence of parameters on angular error
Edge-based Gamut (? = 3) 6.8o 5.8o 6.0o 40.3o Influence of parameters on angular error
Intersection-based Gamut (? = 9) 6.9o 5.8o 6.1o 41.9o Influence of parameters on angular error

 

Various other methods have been applied to this set. If you would like to have your results reported here, feel free to send a message to the contact person of this web site.

Method (applied to 11346 images)  Mean error  Median error  Trimean error  Max error
Inv. Intensity Chrom. Space 6.6o 5.6o 5.8o 76.2o
Using Natural Image Statistics 5.2o 3.9o 4.3o 44.5o
Exemplar-Based Color Constancy 4.4o 3.4o 3.7o 45.6o
Various methods (Uni. Zagreb):
— ColorCat
4.2o 3.2o 3.5o 43.7o
— SmartColorCat 4.6o 3.5o 3.8o 46.2o
— ColorDog Using WP / GW 5.3o 3.7o 4.2o 46.8o
 — ColorDog Using ColorCat 4.5o 2.9o 3.5o 46.2o
 — ColorDog Using Sm.ColorCat 4.8o 3.1o 3.7o 48.7o
 — ColorAnts 5.0o 3.7o 4.1o 46.5o

 


SFU Grey-ball Set (Linear)

Since the original Grey-ball Set was stored in a non-linear device-RGB color space (NTSC=RGB), we created a modification of this set by applying gamma-correction (with ? = 2.2). For consistency, we recomputed the ground truth on the linear images (click here to download this new ground truth). Results on this modified data set show all methods obtain considerably higher angular errors:

Method (applied to 11346 images)  Mean error  Median error  Trimean error  Max error
Grey-World 13.0o 11.0o 11.5o 63.0o
White-Patch 12.7o 10.5o 11.3o 46.5o
Shades-of-Grey (p = 4) 11.6o 9.7o 10.2o 58.1o Influence of parameters on angular error
general Grey-World (p = 4, ? = 0) 11.6o 9.7o 10.2o 58.1o Influence of parameters on angular error
1st-order Grey-Edge (p = 1, ? = 1) 10.6o 8.8o 9.2o 58.4o Influence of parameters on angular error
2nd-order Grey-Edge (p = 1, ? = 1) 10.7o 9.0o 9.4o 56.0o Influence of parameters on angular error

 

The same restrictions as for the original Grey-ball Set apply. Results of the Gamut mapping using 15-fold cross-validation are shown here:

Method (applied to 11346 images)  Mean error  Median error  Trimean error  Max error
Pixel-based Gamut (? = 9) 11.8o 8.9o 10.0o 49.0o Influence of parameters on angular error
Edge-based Gamut (? = 9) 12.8o 10.9o 11.4o 58.3o Influence of parameters on angular error
Intersection-based Gamut (? = 9) 11.8o 8.9o 10.0o 47.5o Influence of parameters on angular error

 

Various other methods have been applied to this set. If you would like to have your results reported here, feel free to send a message to the contact person of this web site.

Method (applied to 11346 images)  Mean error  Median error  Trimean error  Max error
Inv. Intensity Chrom. Space 14.7o 11.0o 11.6o 86.8o
Regression (SVR) 13.1o 11.2o 11.8o 59.6o
Spatial Correlations:
— Gaussian-based
12.7o 10.8o 11.5o 41.2o
 — HeavyTailed-based 10.3o 8.9o 9.2o 53.6o
Bottom-up (see here) 10.0o 8.0o 8.5o 58.5o
Top-down (see here) 10.2o 8.3o 8.7o 63.0o
Bottom-up+Top-down (see here) 9.7o 7.7o 8.2o 60.0o
Using Natural Image Statistics 9.9o 7.7o 8.3o 56.1o
Exemplar-Based Color Constancy 8.0o 6.5o 6.8o 53.6o
Various methods (Uni. Zagreb):
— ColorCat
8.7o 7.1o 7.4o 52.4o
— SmartColorCat 8.2o 6.3o 6.7o 54.5o
— ColorDog Using WP / GW

10.3o 7.3o 8.2o 53.6o
 — ColorDog Using ColorCat 8.8o 6.0o 7.0o 54.8o
 — ColorDog Using Sm.ColorCat 8.5o 5.6o 6.6o 56.2o
 — ColorAnts 8.9o 6.5o 7.1o 57.6o

 

The results of the Spatial Correlations denoted “Gaussian-based” are computed by Gijsenij et al. using an earlier implementation of the Spatial Correlations method. The results of Spatial Correlations denoted “HeavyTailed-based” is using the newest implementation and can be found in “Color Constancy with Spatio-spectral Statistics”, in IEEE TPAMI. More elaborate results by Chakrabarti et al. can be downloaded here. These results also contain slightly different results for the Grey-Edge-type methods.

 


Color-checker (Original)

Using the original images (gamma-corrected from sRGB to linear-RGB), we obtain the following results for the low-level methods (do not forget to mask the color checker!):

Method (applied to 568 images)  Mean error  Median error  Trimean error  Max error
Grey-World 9.8o 7.4o 8.2o 46.0o
White-Patch 8.1o 6.0o 6.4o 36.3o
Shades-of-Grey (p = 6) 7.0o 5.3o 5.6o 36.6o Influence of parameters on angular error
general Grey-World (p = 6, ? = 0) 7.0o 5.3o 5.6o 36.6o Influence of parameters on angular error
1st-order Grey-Edge (p = 4, ? = 1) 7.0o 5.2o 5.5o 36.3o Influence of parameters on angular error
2nd-order Grey-Edge (p = 4;, ? = 7) 7.0o 5.0o 5.4o 38.0o Influence of parameters on angular error

 

Using three-fold cross-validation (with the folds supplied along with the original data set and ground truth) to learn the canonical gamut, we obtain the following results for the Gamut mapping:

Method (applied to 568 images)  Mean error  Median error  Trimean error  Max error
Pixel-based Gamut (? = 5) 6.9o 4.9o 5.2o 37.1o Influence of parameters on angular error
Edge-based Gamut (? = 7) 7.7o 5.0o 5.7o 44.4o Influence of parameters on angular error
Intersection-based Gamut (? = 5) 6.9o 4.9o 5.2o 37.1o Influence of parameters on angular error

 

Various other methods have been applied to this set. Please note that any algorithm that requires learning is trained using the same three folds as were used for the Gamut mapping. If you would like to have your results reported here, feel free to send a message to the contact person of this web site.

Method (applied to 568 images)  Mean error  Median error  Trimean error  Max error
Inv. Intensity Chrom. Space 9.7o 6.0o 6.7o 61.3o
Bayesian 6.7o 4.7o 5.0o 39.4o
Bottom-up (see here) 6.6o 4.9o 5.2o 36.5o
Top-down (see here) 6.7o 4.7o 5.1o 43.8o
Bottom-up+Top-down (see here) 6.4o 4.5o 5.0o 33.6o
Using Natural Image Statistics 6.1o 4.5o 4.9o 36.8o
Weighted Grey-Edge
(p=2, ? = 1, ? = 20)
6.6o 4.7o 5.1o 44.3o

 


Color-checker (by Shi)

Since the original color checker set wat generated from RAW data using automatic settings, Shi reprocessed the original RAW data and generated 12-bit PNG-images (with lossless compression). Using these new images (and a new ground truth), the following results are obtained for the low-level methods:

Method (applied to 568 images)  Mean error  Median error  Trimean error  Max error
Grey-World 6.4o 6.3o 6.3o 24.8o
White-Patch 7.6o 5.7o 6.4o 40.6o
Shades-of-Grey (p = 4) 4.9o 4.0o 4.2o 22.4o Influence of parameters on angular error
general Grey-World (p = 9, ? = 9) 4.7o 3.5o 3.8o 22.0o Influence of parameters on angular error
1st-order Grey-Edge (p = 1, ? = 6) 5.3o 4.5o 4.7o 26.4o Influence of parameters on angular error
2nd-order Grey-Edge (p = 1, ? = 1) 5.1o 4.4o 4.6o 23.9o Influence of parameters on angular error

 

Using the same three-fold cross-validation as before, we obtain the following results for the Gamut mapping:

Method (applied to 568 images)  Mean error  Median error  Trimean error  Max error
Pixel-based Gamut (? = 4) 4.2o 2.3o 2.9o 23.2o Influence of parameters on angular error
Edge-based Gamut (? = 4) 6.5o 5.0o 5.4o 29.0o Influence of parameters on angular error
Intersection-based Gamut (? = 9) 4.2o 2.3o 2.9o 24.2o Influence of parameters on angular error

 

Various other methods have been applied to this set. Please note that any algorithm that requires learning is trained using the same three folds as were used for the Gamut mapping. If you would like to have your results reported here, feel free to send a message to the contact person of this web site.

Method (applied to 568 images)  Mean error  Median error  Trimean error  Max error
Inv. Intensity Chrom. Space 13.6o 13.6o 13.5o 56.7o
Regression (SVR) 8.1o 6.7o 7.2o 32.0o
Bayesian 4.8o 3.5o 3.9o 24.5o
Spatial Correlations:
— Gaussian-based
4.0o 3.1o 3.3o 19.8o
 — HeavyTailed-based 3.6o 3.0o 3.0o 21.6o
Bottom-up (see here) 3.4o 2.6o 2.7o 20.6o
Top-down (see here) 3.8o 2.6o 2.8o 25.2o
Bottom-up+Top-down (see here) 3.5o 2.5o 2.6o 25.2o
Using Natural Image Statistics 4.2o 3.1o 3.5o 26.2o
CART-based Selection 4.5o 3.9o 3.3o 22.3o
CART-based Combination 3.9o 2.9o 3.3o 22.3o
Exemplar-Based Color Constancy 2.9o 2.3o 2.4o 19.4o
Using CNNs:
— AlexNet+SVR
4.7o 3.1o 3.5o 29.2o
 — Deep color constancy 2.6o 2.0o 2.1o 14.8o

 

Notes

– The results of the Spatial Correlations denoted “Gaussian-based” are computed by Gijsenij et al. using an earlier implementation of the Spatial Correlations method. The results of Spatial Correlations denoted “HeavyTailed-based” is using the newest implementation and can be found in “Color Constancy with Spatio-spectral Statistics”, in IEEE TPAMI. More elaborate results by Chakrabarti et al. can be downloaded here. These results also contain slightly different results for the Grey-Edge-type methods.
– The results of the CART-based Selection and Combination methods are delivered by Bianco et al. They also computed results for the reprocessed Color-checker set using an alternative ground truth, these results are specified here. This alternative ground truth and the results can be downloaded here.