ImageMagick-6.9.3 reuses user lock after freed

[jhowarth]

While testing builds of ImageMagick-6.9.3 against the upcoming llvm 3.8 release RC1 clang compiler and libomp support, I discovered a large number of regressions in the resulting ImageMagick build's test suite...

https://llvm.org/bugs/show_bug.cgi?id=26184

The llvm OpenMP maintainer has analyzed these segfaulting test cases...

https://llvm.org/bugs/show_bug.cgi?id=26184#c15

and believes this is a bug in ImageMagick attempting to reuse the user lock after it has been freed. The ImageMagick developers really should attempt to build the current ImageMagick-6.9.3 release against the llvm 3.8 RC compilers as they do seem to expose latent bugs in the openmp support.

[jhowarth]

Confirmed on x86_64-apple-darwin15 with clang/openmp 3.8 branch svn that using...

make -k -j1 check KMP_LOCK_KIND=tas

makes the bug go latent again and produces clean test suite results for ImageMagick-6.9.3 .

[magick]

Thanks for the problem report. We can reproduce it and have a patch to fix it in the GIT master branch @ https://github.com/ImageMagick/ImageMagick. The patch will be available in the beta releases of ImageMagick @ http://www.imagemagick.org/download/beta/ by sometime tomorrow.

[jhowarth]

Current ImageMagick-6 still seems to show segfaults in a variable number of test cases despite the recent commit...

FAIL: Magick++/tests/tests.tap 3
FAIL: Magick++/tests/tests.tap 5
FAIL: Magick++/tests/tests.tap 6
FAIL: Magick++/tests/tests.tap 11
FAIL: Magick++/demo/demos.tap 17
FAIL: Magick++/demo/demos.tap 20

when built with clang/libomp 3.8svn on x86_64 darwin.

[jhowarth]

These failures back trace differently now...

# ./averageImages
(lldb) target create "/sw/src/fink.build/libmagickcore6.9.q16.2-shlibs-6.9.3.3-1/ImageMagick-6.9.3-3/Magick++/tests/.libs/averageImages"
Current executable set to '/sw/src/fink.build/libmagickcore6.9.q16.2-shlibs-6.9.3.3-1/ImageMagick-6.9.3-3/Magick++/tests/.libs/averageImages' (x86_64).
(lldb) r
Process 72931 launched: '/sw/src/fink.build/libmagickcore6.9.q16.2-shlibs-6.9.3.3-1/ImageMagick-6.9.3-3/Magick++/tests/.libs/averageImages' (x86_64)
Process 72931 stopped
* thread #1: tid = 0x1053970, 0x0000000100b55eee libomp.dylib`__kmp_acquire_ticket_lock + 14, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
frame #0: 0x0000000100b55eee libomp.dylib`__kmp_acquire_ticket_lock + 14
libomp.dylib`__kmp_acquire_ticket_lock:
-> 0x100b55eee <+14>: lock
0x100b55eef <+15>: xaddl %esi, 0x10(%rbx)
0x100b55ef3 <+19>: movl 0x14(%rbx), %eax
0x100b55ef6 <+22>: cmpl %esi, %eax
(lldb) bt
error: libMagickCore-6.Q16HDRI.2.dylib debug map object file '/sw/src/fink.build/libmagickcore6.9.q16.2-shlibs-6.9.3.3-1/ImageMagick-6.9.3-3/magick/.libs/magick_libMagickCore_6_Q16HDRI_la-magick.o' has changed (actual time is 0x56a40017, debug map time is 0x56a3fbc3) since this executable was linked, file will be ignored
* thread #1: tid = 0x1053970, 0x0000000100b55eee libomp.dylib`__kmp_acquire_ticket_lock + 14, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
* frame #0: 0x0000000100b55eee libomp.dylib`__kmp_acquire_ticket_lock + 14
frame #1: 0x0000000100b1a034 libomp.dylib`__kmpc_set_lock + 84
frame #2: 0x00000001003215ef libMagickCore-6.Q16HDRI.2.dylib`MagickCoreTerminus + 47
frame #3: 0x00007fff8b00646b libsystem_c.dylib`__cxa_finalize_ranges + 345
frame #4: 0x00007fff8b00676f libsystem_c.dylib`exit + 55
frame #5: 0x00007fff9ef1c5b4 libdyld.dylib`start + 8
frame #6: 0x00007fff9ef1c5ad libdyld.dylib`start + 1

[jhowarth]

These new regressions only occur in ImageMagick-6 branch. Master built the same way against clang/libomp 3.8svn shows clean test suite results.

[jhowarth]

FYI, prebuilt linux binaries for llvm 3.8' RC1s clang compiler (including the openmp libomp) are available for download at...
http://llvm.org/pre-releases/3.8.0/

[jhowarth]

Commenting the line...

/* SemaphoreComponentTerminus(); */

in magick/magick.c/MagickCoreTerminus()/1406: doesn't suppress these regressions in ImageMagick-6 branch when built against clang/libomp 3.8 RC1. These failures do seem to be very random so you might need to try to repeatedly execute one of them, such as appendImage, after running make check.

[magick]

Ok, got LLVM 3.8.0 RC1 built from source under Fedora 23 along with the latest OpenMP. The unit tests did pass. We'll try rerunning and see if we can get it to fail.

[jhowarth]

Make sure that the ImageMagick-6 branch build is actually modified to link against libomp. On darwin, I used...

perl -pi -e 's|-lgomp|-L/sw/opt/llvm-3.8/lib -lomp|g' configure

[jhowarth]

Also, in case it matters, the following are the configure options that I am using on x86_64-apple-darwin15...

$ ./configure --prefix=/sw --prefix=/sw/lib/ImageMagick-6.9 --bindir=/sw/bin --docdir=/sw/share/doc/ImageMagick-6 --includedir=/sw/include --mandir=/sw/share/man --disable-silent-rules --enable-dependency-tracking --x-includes=/opt/X11/include --x-libraries=/opt/X11/lib --enable-openmp --disable-opencl --enable-shared --disable-static --enable-hdri --with-modules --with-quantum-depth=16 --with-frozenpaths --with-magick-plus-plus --without-perl --with-bzlib --with-x --without-autotrace --without-dps --with-djvu --with-fftw --without-fpx --without-gslib --with-gs-font-dir=/sw/share/ghostscript/fonts --without-gvc --with-jbig --with-jpeg --without-openjp2 --with-lcms --with-lqr --with-lzma --with-openexr --without-pango --with-png --with-rsvg --with-tiff --with-webp --with-windows-font-dir=/Library/Fonts --with-wmf --with-xml

[magick]

We verified ImageMagick is linked to the LLVM OpenMP:

• ldd convert
  libomp.so => /usr/local/lib/libomp.so (0x00007f7b20d5e000)

And 'configure -list configure' returns:

• CONFIGURE ./configure '--enable-delegate-build' '--enable-shared' '--disable-static' '--with-perl' '--with-wmf' '--with-quantum-depth=16' '--with-modules' 'CC=/usr/local/bin/clang' 'CFLAGS=-O2 -g' 'LDFLAGS=' 'CXX=/usr/local/bin/clang++' 'CXXFLAGS=-O2 -g' 'PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/'

We'll continue to run the unit tests, but after hundreds of runs so far, none of them have failed.

[jhowarth]

Okay, I am checking with Jonathan Peyton to see if his reproduction of the original failures that I reported was done on darwin or linux (in case that these problems are only triggered on that target). FYI, I also see failures in current ImageMagick-6 branch using llvm/libomp 3.7.1 (which requires -fopenmp=libomp as it isn't the default there).

FAIL: Magick++/tests/tests.tap 6
FAIL: Magick++/tests/tests.tap 8
FAIL: Magick++/tests/tests.tap 11
FAIL: Magick++/tests/tests.tap 12

These failures back trace differently as...

# ./color
(lldb) target create "/sw/src/fink.build/libmagickcore6.9.q16.2-shlibs-6.9.3.3-1/ImageMagick-6.9.3-3/Magick++/tests/.libs/color"
Current executable set to '/sw/src/fink.build/libmagickcore6.9.q16.2-shlibs-6.9.3.3-1/ImageMagick-6.9.3-3/Magick++/tests/.libs/color' (x86_64).
(lldb) r
Process 72396 launched: '/sw/src/fink.build/libmagickcore6.9.q16.2-shlibs-6.9.3.3-1/ImageMagick-6.9.3-3/Magick++/tests/.libs/color' (x86_64)
Process 72396 stopped
* thread #1: tid = 0x10f615a, 0x0000000100b55922 libomp.dylib`void __kmp_acquire_queuing_lock_timed_template<false>(kmp_queuing_lock*, int) + 274, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x110f18eb8)
frame #0: 0x0000000100b55922 libomp.dylib`void __kmp_acquire_queuing_lock_timed_template<false>(kmp_queuing_lock*, int) + 274
libomp.dylib`void __kmp_acquire_queuing_lock_timed_template<false>(kmp_queuing_lock*, int):
-> 0x100b55922 <+274>: movq -0x8(%rax,%rcx,8), %rax
0x100b55927 <+279>: movl -0x34(%rbp), %ecx
0x100b5592a <+282>: movl %ecx, 0x400(%rax)
0x100b55930 <+288>: leaq -0x1567(%rip), %rdx ; __kmp_eq_4
(lldb) bt
error: libMagickCore-6.Q16HDRI.2.dylib debug map object file '/sw/src/fink.build/libmagickcore6.9.q16.2-shlibs-6.9.3.3-1/ImageMagick-6.9.3-3/magick/.libs/magick_libMagickCore_6_Q16HDRI_la-registry.o' has changed (actual time is 0x56a5277d, debug map time is 0x56a52647) since this executable was linked, file will be ignored
error: libMagickCore-6.Q16HDRI.2.dylib debug map object file '/sw/src/fink.build/libmagickcore6.9.q16.2-shlibs-6.9.3.3-1/ImageMagick-6.9.3-3/magick/.libs/magick_libMagickCore_6_Q16HDRI_la-magick.o' has changed (actual time is 0x56a5277a, debug map time is 0x56a52644) since this executable was linked, file will be ignored
* thread #1: tid = 0x10f615a, 0x0000000100b55922 libomp.dylib`void __kmp_acquire_queuing_lock_timed_template<false>(kmp_queuing_lock*, int) + 274, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x110f18eb8)
* frame #0: 0x0000000100b55922 libomp.dylib`void __kmp_acquire_queuing_lock_timed_template<false>(kmp_queuing_lock*, int) + 274
frame #1: 0x0000000100b1c0fe libomp.dylib`__kmpc_set_lock + 78
frame #2: 0x000000010038301a libMagickCore-6.Q16HDRI.2.dylib`RegistryComponentTerminus + 42
frame #3: 0x0000000100323741 libMagickCore-6.Q16HDRI.2.dylib`MagickCoreTerminus + 65
frame #4: 0x00007fff8b00646b libsystem_c.dylib`__cxa_finalize_ranges + 345
frame #5: 0x00007fff8b00676f libsystem_c.dylib`exit + 55
frame #6: 0x00007fff9ef1c5b4 libdyld.dylib`start + 8
frame #7: 0x00007fff9ef1c5ad libdyld.dylib`start + 1

[magick]

And you say the master branch works? That's ImageMagick 7.0.0-0. Its curious because both use the same locking code in semaphore.c and they use the same OMP pragmas.

[jhowarth]

Yes, master branch on darwin works without any changes. I am still waiting for a response in https://llvm.org/bugs/show_bug.cgi?id=26184 as to whether the OpenMP developers reproduced the original lock issues on darwin or linux.