fuzzing-created crashers

[setharnold]

Hello; there's a renewed interest in adding openjpeg2 to Ubuntu's 'main' archive so that it can be used in Ubuntu's PDF reader, graphics programs, etc. As part of this process I've run the OpenJPEG 2.1.1 release through the AFL (American Fuzzy Lop) guided fuzzing tool and generated the following set of files that crash the opj_decompress tool. (Presumably this does something similar to what would be used by document viewers.)

The Ubuntu bug for the main inclusion process is at https://bugs.launchpad.net/ubuntu/+source/openjpeg2/+bug/711061

The crashing inputs are saved in a compressed tarball on launchpad: https://bugs.launchpad.net/ubuntu/+source/openjpeg2/+bug/711061/+attachment/4723094/+files/crashes-openjpeg-2.1.1.tar.gz

Please consider adding these images to the testing frameworks.

Thanks

[malaterre]

Could you please also document the AFL cmd line you used ? In particular how long have you been runnning it ?

[setharnold]

On Aug 17, 2016 22:57, "Mathieu Malaterre" [email protected] wrote:

Could you please also document the AFL cmd line you used ? In particular
how long have you been runnning it ?

It was along the lines of:

screen
afl-fuzz -M master -i in -s out -f foo.j2k ../bin/opj_decompress -i @@ -o
foo.bmp
for i in seq -w 1 29 ; do screen afl-fuzz -S slave$i -i in -s out -f
foo$i.j2k ../bin/opj_decompress -i @@ -o foo$i.bmp ; done

I ran this for about two days; afl-whatsup reported somewhere around two
months of CPU time. It was still finding new crashes every few minutes but
the weather is turning too hot to leave this running.

Change the 29 to something that's a good fit for your hardware. Roughly the
number of hyperthreads on your computer is a good starting point.

This returned roughly 4500 unique crashes. afl-cmin was able to reduce this
to a minimized set of roughly 600 input files that together cover all
"edges" covered by the full set. The afl-cmin command was along the lines
of:

mkdir collected min
mv out//crash/id: collected/
afl-cmin -i collected/ -o min/ -f foo.j2k ../bin/opj_decompress -i @@ -o
foo.bmp

I started with the same relax.j2k image as last time. I think AFL could
make faster progress if it starts with a smaller input file. Next time
starting with a 1*1 pixel image might be more productive more quickly.

Note that this corpus of inputs is entirely devoted to openjpeg to BMP
crashes. Converting to other formats, or from other formats to openjpeg, is
liable to discover entirely new crashes.

Thanks

[boxerab]

All crashes seem to stem from the same root cause : trying to apply an ICC profile
when subsampling is not uniform across components. Should be pretty simple to fix.

[trylab]

All the crashes seem division-by-zero problems which I've reported on the issue list before.

[boxerab]

@setharnold @trylab I've fixed these crashes in my Grok J2K codec: https://github.com/GrokImageCompression/grok
Do you have time to run your fuzzer on this codec ? The API is the same as OpenJPEG, so I think
it would be pretty easy to set things up.

[trylab]

@boxerab OK. I'll have a try when I'm available.

[setharnold]

boxerab, sorry, I couldn't get AFL to run with grok:

$ afl-fuzz -i fuzz/in -o fuzz/out -f foo.jp2 ./bin/opj_decompress -i @@ -o foo.bmp            
afl-fuzz 1.96b by <[email protected]>
[+] You have 30 CPU cores and 1 runnable tasks (utilization: 3%).
[+] Try parallel jobs - see /usr/share/doc/afl/parallel_fuzzing.txt.
[*] Checking core_pattern...
[*] Setting up output directories...
[+] Output directory exists but deemed OK to reuse.
[*] Deleting old session data...
[+] Output dir cleanup successful.
[*] Scanning 'fuzz/in'...
[+] No auto-generated dictionary tokens to reuse.
[*] Creating hard links for all input files...
[*] Validating target binary...
[*] Attempting dry run with 'id:000000,orig:relax.jp2'...
[*] Spinning up the fork server...

[-] Hmm, looks like the target binary terminated before we could complete a
    handshake with the injected code. There are two probable explanations:

    - The current memory limit (50.0 MB) is too restrictive, causing an OOM
      fault in the dynamic linker. This can be fixed with the -m option. A
      simple way to confirm the diagnosis may be:

      ( ulimit -Sv $[49 << 10]; /path/to/fuzzed_app )

      Tip: you can use http://jwilk.net/software/recidivm to quickly
      estimate the required amount of virtual memory for the binary.

    - Less likely, there is a horrible bug in the fuzzer. If other options
      fail, poke <[email protected]> for troubleshooting tips.

[-] PROGRAM ABORT : Fork server handshake failed
         Location : init_forkserver(), afl-fuzz.c:2108

[boxerab]

Thanks, guys. I've moved the AFL issue with Grok over to the Grok repo:

https://github.com/GrokImageCompression/grok/issues/22

[asarubbo]

The crashing inputs are saved in a compressed tarball on launchpad: https://bugs.launchpad.net/ubuntu/+source/openjpeg2/+bug/711061/+attachment/4723094/+files/crashes-openjpeg-2.1.1.tar.gz

Please consider adding these images to the testing frameworks.

@setharnold
Your archive contains 646 images, does each image cause a problem which have a different nature? I guess no.
For better tracking of issues and give people (mainly who fuzzes) a way to check if the issue was already reported, I suggest to open one issue per problem. Few lines of bash will help you to delete the duplicates from the crashes/ directory. Thanks.

[setharnold]

@asarubbo, AFL generated over 4000 crashing images; the tarball of 646 images has already been reduced by afl-cmin to the smallest set of images that covered all paths discovered by the total corpus. By that measure, they are all unique crashing inputs, however it's true that a single fix may address multiple input files. That's exceedingly difficult to determine short of actually preparing fixes.

What exactly are you looking for? I'd like to provide something that's actually useful to your team, but opening 600-odd issues seems like the wrong approach.

Thanks

[detonin]

@mayeut Would you have time to have a look at this one ? would be great to fix it before 2.1.2

[szukw000]

@detonin, @mayeut,

I have asked for two pull requests:

issue #869
Changes for issues #863 and #862
This patch is reported to prevent SEGFAULTS introduced with files PoC1 and PoC2.

issue #873
These changes repair bugs of #871 and #872
Repair was possible when using the types of LIBTIFF.

Then I found the following

issue #811
fuzzing-created crashers
setharnold opened this Issue on Aug 18

I have now tested all files of the archive 'crashes-openjpeg-2.1.1.tar.gz'
and would like to remove the above mentioned pull requests.

The changes proposed in the above mentioned pull requests are now part of
the changes made for #811.

I you agreed, I could make another pull request for #811, #862, #863, #871
and #872.

winfried

[szukw000]

@detonin , @mayeut , @setharnold ,

after having solved the crashes of 'crashes-openjpeg-2.1.1.tar.gz'
(see #875: Patch for crashes #811,#862,#863,#871 and #872)
I ran afl-gcc myself and found some other crashes. These I could solve
with patches not plublicly available.

Then I ran afl-gcc a second time: no crash, but some hangs.

One of the 'out/hangs' example:

jp2.c:600: IHDR w(32) h(2147483680) bpc(7)
j2k.c:2095: IMAGE w(32) h(1056) TILE w(32) h(32)
j2k.c:2576: m_layer(0) numlayers(65281)

[ERROR] IHDR w(32) h(2147483680) != IMAGE w(32) h(1056)
ERROR -> opj_decompress: failed to decode image!

The original image is:

in/not_kitty.jp2

jp2.c:600: IHDR w(32) h(32) bpc(7)
j2k.c:2095: IMAGE w(32) h(32) TILE w(32) h(32)
j2k.c:2576: m_layer(0) numlayers(1)

winfried

[setharnold]

Impressive set of patches. It feels like nitpicking but some of the indentation doesn't match the surroundings well. (It's more obvious in the GitHub web view than it would be in your editor configured with your favorite settings :) Thanks

[rouault]

With latest master, "for i in ../crashes-openjpeg-2.1.1/*; do echo $i; bin/opj_decompress -quiet -i $i -o out.bmp; done >log.txt 2>&1" on a -fsanitize=address,undefined now works cleanly

[asarubbo]

if possible would be great have a new stable release.