Initial Migration of Fuzz Tests & Integration Scripts From the OSS-Fuzz Project Repo by DaveLak · Pull Request #1901 · gitpython-developers/GitPython

added 5 commits

April 11, 2024 19:55
Migrates the OSS-Fuzz tests and setup scripts from the OSS-Fuzz
repository to GitPython's repo as discussed here:
gitpython-developers#1887 (comment)

These files include the changes that were originally proposed in:
google/oss-fuzz#11763

Additional changes include:
- A first pass at documenting the contents of the fuzzing set up in a
  dedicated README.md
- Adding the dictionary files to this repo for improved visibility. Seed
  corpra zips are still located in an external repo pending further
 discussion regarding where those should live in the long term.
Adds additional documentation links and fixes some typos.
- Updates the fuzzing documentation to include steps for working with
  locally modified versions of the gitpython repository.
- Updates the build.sh script to make the fuzz target search path more
  specific, reducing the risk of local OSS-Fuzz builds picking up
  files located outside of where we expect them (for example, in a .venv
  directory.)
- add artifacts produced by local OSS-Fuzz runs to gitignore
- Fix typos in the documentation on dictionaries
- Link to the fuzzing directory in the main README where it is
  referenced.

@DaveLak DaveLak marked this pull request as ready for review

April 12, 2024 05:00

DaveLak added a commit to DaveLak/oss-fuzz that referenced this pull request

Apr 12, 2024
Updates the gitpython project files to enable migrating and maintaining
fuzz targets and build scripts upstream.

Related PR in the upstream repo: gitpython-developers/GitPython#1901

Byron

@DaveLak

EliahKagan

DaveLak added a commit to DaveLak/GitPython that referenced this pull request

Apr 16, 2024
Addresses feedback and encorperates suggestions from PR gitpython-developers#1901 to ensure
that the Apache License requirements are met for the two files that they
apply to, and the documentation pertaining to licensing of the files in
this repository is clear and concise.

@DaveLak

EliahKagan

DaveLak added a commit to DaveLak/GitPython that referenced this pull request

Apr 16, 2024
Prefer executing these files using the OSS-Fuzz or `python` command
methods outlined in the `fuzzing/README`.

Based on feedback and discussion on:
gitpython-developers#1901

DaveLak added a commit to DaveLak/GitPython that referenced this pull request

Apr 16, 2024
This script is meant to be sourced by the OSS-Fuzz file of the same
name, rather than executed directly. The shebang may lead to the
incorrect assumption that the script is meant for direct execution.
Replacing it with this directive instructs ShellCheck to treat
the script as a Bash script, regardless of how it is executed.

Based @EliahKagan's suggestion and feedback on:
gitpython-developers#1901

DaveLak added a commit to DaveLak/GitPython that referenced this pull request

Apr 16, 2024
This script is executed directly, not sourced as is the case with
`build.sh`, so it should have an executable bit set to avoid ambiguity.

Based @EliahKagan's suggestion and feedback on:
gitpython-developers#1901

DaveLak added a commit to DaveLak/GitPython that referenced this pull request

Apr 16, 2024
- Make the link text for the OSS-Fuzz test status URL more descriptive
- Fix capitalization of GitPython repository name

Based @EliahKagan's suggestion and feedback on:
gitpython-developers#1901

DaveLak added a commit to DaveLak/GitPython that referenced this pull request

Apr 16, 2024
Replaces the null character delimiter `-d $'\0'` with the simpler
empty string `-d ''` in the fuzzing harness build loop.

This changes leverages the Bash `read` builtin behavior to avoid
unnecessary complexity and improving script readability.

Based @EliahKagan's suggestion and feedback on:
gitpython-developers#1901

DaveLak added a commit to DaveLak/GitPython that referenced this pull request

Apr 16, 2024

DaveLak added a commit to DaveLak/GitPython that referenced this pull request

Apr 16, 2024

@DaveLak

Byron

DavidKorczynski pushed a commit to google/oss-fuzz that referenced this pull request

Apr 18, 2024
Updates the gitpython project files to enable migrating and maintaining
fuzz targets and build scripts upstream.

Related PR in the upstream repo:
gitpython-developers/GitPython#1901

`project.yaml` updates:

- @Byron, the maintainer of GitPython, is added as the primary contact.
- @EliahKagan and myself are added to the `auto_ccs` list as discussed
with @Byron here:
gitpython-developers/GitPython#1889 (comment)
- @DavidKorczynski I removed what I believe is your email from the
`vendor_ccs` because it looked like you were included as the default
when no other contacts were listed. If this was a mistake on my part and
you want to remain listed as a CC, please let me know and I'll correct
it.

Thanks!

DaveLak added a commit to DaveLak/GitPython that referenced this pull request

Apr 20, 2024
Adds a Dockerfile to enable easily executing the fuzz targets directly
inside a container environment instead of directly on a host machine.
This addresses concerns raised in PR gitpython-developers#1901 related to how `fuzz_tree.py`
writes to the real `/tmp` directory of the file system it is executed on
as part of setting up its own test fixtures, but also makes for an
easier to use development workflow.

See this related comment on PR gitpython-developers#1901 for additional context:
gitpython-developers#1901 (comment)

@DaveLak DaveLak deleted the oss-fuzz-test-harness-upstreaming branch

April 22, 2024 20:16

DaveLak added a commit to DaveLak/GitPython that referenced this pull request

Apr 22, 2024
These files are already BSD-3-Clause even without the headers, but
adding these comments and the `LICENSE-BSD` symlink to the root level
`LICENSE` file are helpful to reinforce that there are only two
particular files in the `fuzzing/` that are not under BSD-3-Clause.

See:
gitpython-developers#1901 (comment)

DaveLak added a commit to DaveLak/GitPython that referenced this pull request

Apr 22, 2024
While discussing adding similar license comments to the shell scripts
introduced in PR gitpython-developers#1901, it was noticed that the shell scripts in the
repository root directory did not have such comments and suggested that
we could add them when the scripts in the `fuzzing/` directory were
updated, so this commit does just that.

See:
gitpython-developers#1901 (comment)

DaveLak added a commit to DaveLak/GitPython that referenced this pull request

Apr 29, 2024

DaveLak added a commit to DaveLak/GitPython that referenced this pull request

Apr 29, 2024
As discussed in the initial fuzzing integration PR[^1], `fuzz_tree.py`'s
implementation was not ideal in terms of coverage and its reading/writing to
hard-coded paths inside `/tmp` was problematic as (among other concerns), it
causes intermittent crashes on ClusterFuzz[^2] when multiple workers execute
the test at the same time on the same machine.

The changes here replace `fuzz_tree.py` completely with a completely new
`fuzz_repo.py` fuzz target which:

- Uses `tempfile.TemporaryDirectory()` to safely manage tmpdir creation and
  tear down, including during multi-worker execution runs.
- Retains the same feature coverage as `fuzz_tree.py`, but it also adds
  considerably more from much smaller data inputs and with less memory consumed
  (and it doesn't even have a seed corpus or target specific dictionary yet.)
- Can likely be improved further in the future by exercising additional features
  of `Repo` to the harness.

Because `fuzz_tree.py` was removed and `fuzz_repo.py` was not derived from it,
the Apache License call outs in the docs were also updated as they only apply to
the singe `fuzz_config.py` file now.

[^1]: gitpython-developers#1901 (comment)
[^2]: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=68355