Large 1bit Tiffs take forever

[DustinYoder]

I have very high res TIF files that need to be rotated. So I notice that magick is creating a temp file to do this as the file is over 1GB in size on the drive. So I just do a convert {INFILE} -rotate 270 {OUTFILE} and it processes small 1 bit tiffs fairly well, still not super fast, but the larger ones take FOREVER. Like hours. I can add -debug All to the command and see that it is still making progress, but very slowly. It keeps writing to the pixel cache.
So any ideas? I'm increasing my memory limits in the policy.xml file as well as making it a single thread. The main issue with these tiffs that may be unique is that they are 1 bit or 2 color only. Black and White, no gray. Any thing I can do to speed this up? Is there a way to rotate that doesn't take much calculation? Thanks.

[snibgo]

I suppose you don't have enough memory. I suggest you try a specialist program such as tiffcrop.

[fmw42]

What version of ImageMagick and what platform? IM 6 will treat it as color (>=3 channels). But IM 7 will treat it as one channel, assuming no transparency. So IM 6 will need more RAM. How big is your image in width and height? How much RAM do you have?

[DustinYoder]

The image is 40" x 30" at 2540 ppi so in the range of 101,600 x 76,200 pixels. I tried the latest version 7 as well as the latest version 6. The computer is pretty new, it's got an i7 6 core 3.2ghz with 32 GB of ram. I tried zip, rle, lzw and no compression in the tiffs. All seem to just never finish processing. I can see the log writing to the image cache temp file every 5 or 10 second or so when I debug.

[DustinYoder]

Where could I download a windows version of tiffcrop? Having trouble finding that.

[fmw42]

It comes with libtiff, the delegate that ImageMagick uses for processing tiff files. Search Google for libtiff

[DustinYoder]

windows binaries for libtiff are at http://gnuwin32.sourceforge.net/packages/tiff.htm, but after installing this, I still don't see a tiffcrop.exe anywhere. Anyone have a compiled version for windows that I can install with all required dll's etc?

I tried using 2TIFF to rotate and that is basically just taking forever to execute too. I'm afraid we're going to end up paying heidelberg $5000 for some custom software they sell for this purpose which can actually handle this type of thing. It does it very quickly too, so I think there is an easy way to do 90 degree increments that these libraries are not taking advantage of.

[fmw42]

100,000x75,000 = 7.5 GB as a single channel 8-bit image. Even though you have binary data, ImageMagick will convert that to 32*bits internally to process, I believe, So 7.5*4=30 GB. So you are likely thrashing to disk a lot. If you use ImageMagick 7, I think ImageMagick will keep it as a single 8-bit image. So you will have 4x less space required. You probably can modify your policy.xml file for processing huge images. See https://imagemagick.org/Usage/files/#massive and policy.xml file at https://imagemagick.org/script/resources.php. Sorry, I am not an expert on this subject. So cannot advise on the resource tuning.

[snibgo]

Tiffcrop is available as a tool in the Cygwin package for Windows. I expect tiffcrop can be downloaded without all the other Cygwin stuff.

However, sadly, tiffcrop gave me an "integer overflow" when I tried it on a 100000x10000 pixel image.


You have 32GB of memory, but how much free memory? Did you have any other programs running, eg web browsers? Did you try Q8 IM, non-HDRI?

If that needs a temporary file on disk, then try converting the input TIFF to a MPC file, and use that as input to a rotation.

If the rotation still isn't fast, perhaps try: chop the input into four (or 9 or 16) pieces, then rotate each piece, then reassemble.

[Werty]

Since on windows you can try opening it with windows photo viewer and rotate it there, i just tried with a 24.000 x 24.000 NASA TIFF image of the moon and it only took a couple minutes, this on i7 4 core on a 32bit win7 with 4gb ram and a mechanical harddrive. Loading took the most of the time, the rotating process was near instant, and saving took 5-10 seconds

[DustinYoder]

Thanks for the replies, I'll try a few of these ideas. I think I can use powershell to using photoviewer to rotate via the command line. So I could try that as well. I need to use command line because I'm running this off of a watched hotfolder, so it can be automatic. I'm going to check the policy.xml and there is a -depth 1 setting to set output to 1 bit depth, but I don't know if that will help during rotation or not. Thanks for the ideas, I'll try to let you know what I find out.

[fmw42]

-depth 1 will only help when writing the output, not while doing the rotation. If you cannot tune the policy.xml file to get adequate speed, you might consider other tools such as VIPS, but I have not experience with it.

[DustinYoder]

Well I was able get my limits set to the point where it didn't need to write to disk. Now I rotate the image in about 2.5 minutes. So I think that will work. I had to put the -limit parameters in the command because my policy.xml file was not being used for some reason. I tried uncommenting the xml comments in that file, but identify limits kept showing lower numbers
So now I'm good except the output tiff is TIFF64 format and so is considered invalid by my destination software. The input tiff is simply TIFF format. How can I force TIFF output and not TIFF64. I think that is the only reason the output file is not opening, it's not opening in photoshop either though so I'm worried the TIFF64 thing is not the only problem. The identify [file] command shows everything same except for the format. Thanks.

[magick]

Thanks for the problem report. We can reproduce it and will have a patch to fix it in GIT master branch @ https://github.com/ImageMagick/ImageMagick later today. The patch will be available in the beta releases of ImageMagick @ https://www.imagemagick.org/download/beta/ by sometime tomorrow.

[DustinYoder]

Thanks! I appreciate that. Would anyone be able to compile that beta version for 8 bit imagemagick, 64bit Windows? I could use this today and I probably would spend much of the day getting it to compile properly. Or instructions on compiling this for windows in the easiest way. Thanks again.