Edit #2: I posted the gist below.
After observing Photoshop's "Save for Web" JPEG performance and comparing it to the JPEGs produced by ImageMagick, it's obvious (to me at least) that Photoshop has superior visual quality at lower filesizes. I attribute it entirely to Adobe's well-honed q-tables, as I can't imagine any other way one would psychovisually tweak a JPEG encoder. However, the question is how to present q-table performance in a quantifiable form.
To that end, I whipped up a small comparison using a state-of-the-art perceptual metric, IW-SSIM, on IM and Photoshop and threw in WebP (which should be better tuned by now) and Kakadu JP2K (which does visual weighting by default). IW-SSIM should outperform PSNR/SSIM/MS-SSIM in almost all situations.
Note that this is far from exhaustive, and I don't claim that it's even conclusive; it's just something I did for my own ends and might as well share.
Reference pictures: Run, Old, Fish
Code: Select all
convert 6.8.0-10 $f -define jpeg:dct-method=float -quality $i kdu_compress 7.1 -full -precise -i $f -rate $i webpe 0.2.1 -preset default -m 6 -segments 4 -af -q $i -pre 1 -pass 5 -jpeg_like CS6 ExportOptionsSaveForWeb(); [ ... ] optimized = true; [ ... ] quality = i; <---- q = 0 to 50 here is about equal to 15 to 75 in IM JPGs were also ran through ImageOptim ( JpegOptim + jpegrescan + jpegtran )
Each of the windows above should represent 30 to 50 different sample points for each different encoder.
The metric numbers are meant to be analyzed in relative terms in standardized test conditions rather than absolute terms. However, based on IW-SSIM's performance on various subjective databases, below 0.95 is almost certainly bad and the threshold into probably bad is around 0.97. Anything above 0.99 is not worth analyzing, as it's entirely "good enough".
About a dozen sanity checks were done in the lower bpp ranges by comparing the various pictures. I entirely agree with IW-SSIM's rankings.
Edit: Well, I guess I missed the obvious fact that Photoshop automatically uses different quality factors for luma and chroma, and structural similarity completely focuses on luma. It could be conceivable that Photoshop simply focuses more on luma, but that is obviously not the case: 502048 B Photoshop vs 49808 B ImageMagick. It just seem strange that q-tables, by themselves, can do so much.
I do already have PS, so I'm just going to ascribe this to Adobe witchcraft and move on. However, it'd be nice if my open-source image encoding tools can go toe to toe with my overpriced proprietary tools.