bpo-29505: Add fuzz tests for float(str), int(str), unicode(str) by ssbr · Pull Request #2878 · python/cpython

added 6 commits

July 21, 2017 00:37
This is an easy place to start, and these functions are probably safe.
Please review with it in mind that I want to add more fuzz tests later.

While the fuzz tests are included in CPython and compiled / tested on a
very basic level inside CPython itself, the actual fuzzing happens as
part of oss-fuzz (https://github.com/google/oss-fuzz). The reason to
include the tests in CPython is to make sure that they're maintained
as part of the CPython project, especially when (as some will) they
use internal implementation details in the test.

(This will be necessary sometimes because e.g. the fuzz test should
never enter Python's interpreter loop, whereas some APIs only expose
themselves publicly as Python functions.)

This particular set of changes is part of testing Python's builtins,
tracked internally at Google by b/37562550.
… occurs.

I'm still not happy with how many times I need to repeat the fuzz test name.
This can go wrong way too easily. :/
It's possible there's a way to compile this with clang and then link it with
clang++, but I don't know how to do that.

Specifically, compilation fails with this:

    $CC $CFLAGS \
        -D _Py_FUZZ_ONE -D _Py_FUZZ_$fuzz_test \
        -Wno-unused-function \
        $($OUT/bin/python3-config --cflags) -g -O1 \
        $FUZZ_DIR/fuzzer.c -o $OUT/$fuzz_test -lFuzzingEngine \
        $($OUT/bin/python3-config --ldflags)

With errors like:

    /usr/local/bin/../include/c++/v1/new:234: undefined reference to `operator delete(void*)'

But it works just fine with this:

    $CXX $CXXFLAGS \
        -D _Py_FUZZ_ONE -D _Py_FUZZ_$fuzz_test \
        -Wno-unused-function \
        $($OUT/bin/python3-config --cflags) -g -O1 \
        $FUZZ_DIR/fuzzer.c -o $OUT/$fuzz_test -lFuzzingEngine \
        $($OUT/bin/python3-config --ldflags)

Presumably there are ways to do this in C, so for expediency I'm doing it in C++
right now, with the minimal possible C++ changes (extern "C"), so that if I need
to change it back to C in code review, it won't be too hard.
(i.e. just to get increased confidence we won't immediately crash on fuzzing.)

I'm using s# because I'd like to minimize the diff between Python 2 and 3.

@ssbr ssbr mentioned this pull request

Jul 26, 2017

alex

@ssbr

(Screwed up my refactoring when I moved the fuzz definition to CPython.)

alex

terryjreedy

@ssbr

larryhastings

@ssbr

Not sure how I didn't notice it. :/

@ssbr

@ssbr

@ssbr

(Despite not understanding how to compile C code.)

@ssbr

@ssbr

Since it's now required in setup.py, we know that the test will run on linux.
On Windows we don't really care if it runs or not -- the fuzz tests are on a
linux machine as far as I'm aware -- and I don't have a windows box to mess with
Visual Studio files in.

Big thanks to zware on #python-dev for walking me through this.

@ssbr

@ssbr

…ryError on some machines.

(But not mine, so I can't debug. If anyone can debug, I'm sure this would be
 better off fixed than ignored...)

See discussion on python#2878

gpshead

Testing it resulted in a few cleanups such as PY_SSIZE_T_CLEAN being
required to avoid from crashing in an opt build.
That was just hiding the actual error in the first place that happened
due to PY_SSIZE_T_CLEAN not being there.

@ssbr ssbr mentioned this pull request

Sep 7, 2017