Averaging (convert mean/median) 80000 16bit PNGs

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Post Reply
pinktank
Posts: 8
Joined: 2015-04-17T09:08:57-07:00
Authentication code: 6789

Averaging (convert mean/median) 80000 16bit PNGs

Post by pinktank » 2019-09-20T21:53:28-07:00

Hello All,
I'm trying to average a folder of 80000 16bit PNGS but running out of memory and harddrive space. The files are about 800GB. It also takes a million years. Is there a better way to do this with a running average or something? I'm on OSX

User avatar
fmw42
Posts: 25556
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Averaging (convert mean/median) 80000 16bit PNGs

Post by fmw42 » 2019-09-20T22:00:16-07:00

You probably need to send your policy.xml file arguments to be able to process images using disk space if you are running out of RAM. Also you should do a running average. But that would require some scripting that is OS dependent. You would definitely run out of RAM trying to load all 80,000 PNGs. How much RAM do you have? What platform/OS? What version of ImageMagick?

See https://imagemagick.org/Usage/files/#massive

pinktank
Posts: 8
Joined: 2015-04-17T09:08:57-07:00
Authentication code: 6789

Re: Averaging (convert mean/median) 80000 16bit PNGs

Post by pinktank » 2019-09-20T22:05:52-07:00

32GB Ram, OSX High Sierra, ImageMagick 7.0.8-59 Q16 x86_64

I tried something like
find . -type f -name "*.png" | xargs -n 100 sh -c 'convert "$0" "$@" -evaluate-sequence mean outdir/"$0" '
but it gave the following error
convert: unable to open image 'outdir/./out0000001.png': No such file or directory @ error/blob.c/OpenBlob/3497.
convert: WriteBlob Failed `outdir/./out0000001.png' @ error/png.c/MagickPNGErrorHandler/1715.

User avatar
fmw42
Posts: 25556
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Averaging (convert mean/median) 80000 16bit PNGs

Post by fmw42 » 2019-09-20T23:13:14-07:00

do you have outdir created? ImageMagick will not create a directory. But that is not going to do a running average because you have to divide by different numbers of images each time. see cumulative average at https://en.wikipedia.org/wiki/Moving_average and https://stackoverflow.com/questions/126 ... data-total and https://en.wikiversity.org/wiki/Moving_Average

For each new value:
new_average = (previous_average*previous_count + new_value) / (previous_count+1)

first_average = (0*0 + first_value)/(0+1) = first_value/1
second_average = (first_average*1 + second_value)/2
third_average = (second_average*2 + third_value)/3
etc

pinktank
Posts: 8
Joined: 2015-04-17T09:08:57-07:00
Authentication code: 6789

Re: Averaging (convert mean/median) 80000 16bit PNGs

Post by pinktank » 2019-09-21T06:29:54-07:00

Yeah I was just sharing it as another way to deal with memory issues since it would dump after each sequence.
My one concern with a moving average is that it is going to have a lot of rounding errors at 8000 16bit limited multiplications. Does Imagemagick carry that math out in 16bits or 64?

There are some tries here that I might look at.
https://www.imagemagick.org/discourse-s ... hp?t=33188
https://www.imagemagick.org/discourse-s ... hp?t=21279

User avatar
fmw42
Posts: 25556
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Averaging (convert mean/median) 80000 16bit PNGs

Post by fmw42 » 2019-09-21T10:38:52-07:00

Computations are done to double precision (64-bits), I believe. But returning and storing in a variable may be limited. But you can use -precision to keep more decimal places if you use fx calculations. Or if you use bc, you can set the scale.

snibgo
Posts: 12152
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Averaging (convert mean/median) 80000 16bit PNGs

Post by snibgo » 2019-09-21T11:31:58-07:00

There are many methods for calculating the mean image from a large set, eg all at once, or rolling average, or in batches of say 100, or some combination. How large are the images, in pixels? That determines how many will fit in memory at once.

I suggest you start work with small images, eg "-crop 1x1+0+0" from each image in your set. Then you can easily experiment with timings and precision. Finally, use your chosen method on the full-size inputs.

PNGs are only 16 bits/channel/pixel. "-evaluate-sequence" uses long double, which are 64-bits. I doubt that precision will be a problem.
snibgo's IM pages: im.snibgo.com

User avatar
fmw42
Posts: 25556
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Averaging (convert mean/median) 80000 16bit PNGs

Post by fmw42 » 2019-09-21T11:39:21-07:00

One way would be to use mogrify (see https://imagemagick.org/Usage/basics/#mogrify) with -scale 1x1! to average all your images to 1 pixel. The -scale does a simple average. See https://imagemagick.org/script/command- ... .php#scale . Then average all the 1 pixel results to get the complete average of the full set. If you need more precision in your average of each image to 1 pixel, then store the results as 1pixel float TIFFS (or MIFF or PFM) using an HDRI compiled ImageMagick. IM 7 is HDRI by default. See TIFF defines for floating point results at https://imagemagick.org/script/formats.php or https://imagemagick.org/script/command- ... php#define. See -define quantum:format=type

snibgo
Posts: 12152
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Averaging (convert mean/median) 80000 16bit PNGs

Post by snibgo » 2019-09-21T11:48:39-07:00

I think the OP wants an average image, not a simple number. My suggestion for 1x1 images is only for development and testing.
snibgo's IM pages: im.snibgo.com

User avatar
fmw42
Posts: 25556
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Averaging (convert mean/median) 80000 16bit PNGs

Post by fmw42 » 2019-09-21T13:11:51-07:00

snibgo wrote:
2019-09-21T11:48:39-07:00
I think the OP wants an average image, not a simple number. My suggestion for 1x1 images is only for development and testing.
My misunderstanding. Thanks.

Post Reply