Implement #85440 (Random test ordering) by smoelius · Pull Request #89082 · rust-lang/rust

@smoelius smoelius changed the title Implement #85440 Implement #85440 (Random test ordering)

Sep 18, 2021

@smoelius smoelius changed the title Implement #85440 (Random test ordering) WIP: Implement #85440 (Random test ordering)

Sep 19, 2021

lukaslueg

@smoelius smoelius changed the title WIP: Implement #85440 (Random test ordering) Implement #85440 (Random test ordering)

Sep 20, 2021

kennytm

@smoelius

@smoelius

kennytm

@smoelius

@smoelius

@smoelius

@bors bors added S-waiting-on-bors

Status: Waiting on bors to run and complete tests. Bors will change the label on completion.

and removed S-waiting-on-review

Status: Awaiting review from the assignee but also interested parties.

labels

Oct 7, 2021

Manishearth added a commit to Manishearth/rust that referenced this pull request

Oct 7, 2021
Implement rust-lang#85440 (Random test ordering)

This PR adds `--shuffle` and `--shuffle-seed` options to `libtest`. The options are similar to the [`-shuffle` option](https://github.com/golang/go/blob/c894b442d1e5e150ad33fa3ce13dbfab1c037b3a/src/testing/testing.go#L1482-L1499) that was recently added to Go.

Here are the relevant parts of the help message:
```
        --shuffle       Run tests in random order
        --shuffle-seed SEED
                        Run tests in random order; seed the random number
                        generator with SEED
...
By default, the tests are run in alphabetical order. Use --shuffle or set
RUST_TEST_SHUFFLE to run the tests in random order. Pass the generated
"shuffle seed" to --shuffle-seed (or set RUST_TEST_SHUFFLE_SEED) to run the
tests in the same order again. Note that --shuffle and --shuffle-seed do not
affect whether the tests are run in parallel.
```
Is an RFC needed for this?

Manishearth added a commit to Manishearth/rust that referenced this pull request

Oct 8, 2021
Implement rust-lang#85440 (Random test ordering)

This PR adds `--shuffle` and `--shuffle-seed` options to `libtest`. The options are similar to the [`-shuffle` option](https://github.com/golang/go/blob/c894b442d1e5e150ad33fa3ce13dbfab1c037b3a/src/testing/testing.go#L1482-L1499) that was recently added to Go.

Here are the relevant parts of the help message:
```
        --shuffle       Run tests in random order
        --shuffle-seed SEED
                        Run tests in random order; seed the random number
                        generator with SEED
...
By default, the tests are run in alphabetical order. Use --shuffle or set
RUST_TEST_SHUFFLE to run the tests in random order. Pass the generated
"shuffle seed" to --shuffle-seed (or set RUST_TEST_SHUFFLE_SEED) to run the
tests in the same order again. Note that --shuffle and --shuffle-seed do not
affect whether the tests are run in parallel.
```
Is an RFC needed for this?

bors added a commit to rust-lang-ci/rust that referenced this pull request

Oct 8, 2021

@epage epage mentioned this pull request

Jul 11, 2025

epage added a commit to epage/rust that referenced this pull request

Aug 11, 2025
Like with rust-lang#139224, this is a documentation-only deprecation for now.
Over time, we can
- warn and then remove on use of unstable environment variables
- warn on use of stable environment variables (no plan to remove due to
  compatibility)

Longer term, we expect test runners, like `cargo test`, to provide the
necessary mechanisms for environmental or persistent configuration (e.g.
using cargo config which supports `.cargo/config.toml` as well as
environment variables).

This would include:
- `RUST_TEST_THREADS`
- `RUST_TEST_NOCAPTURE`
- `RUST_TEST_SHUFFLE` (unstable)
- `RUST_TEST_SHUFFLE_SEED` (unstable)

The primary outcomes for this change are
- Reducing the scope of what is expected for custom test harnesses to
  implement
- Reduce the mechanisms that test runners, like `cargo test`, are
  expected to track when they are being bypassed to protect against
  negative interactions, e.g.  `RUST_TEST_NOCAPTURE=1` when json output
  is being read.

For testing-devex FCP, see rust-lang/testing-devex-team#10

Fixes rust-lang/testing-devex-team#10

History
-------

At each step, I could not find evidence of design discussions on whether
to support CLI, env, or both.  The first env variable seems to come from
the fact that it was being forked out of an existing env variable that
had a much wider scope.

At best, this seems like a way to offer a more persistent configuration
for these flags but environment variables hidden away in libtest is a
bit clunky and this seems like the wrong layer to handle this problem.

**Originally:** `RUST_THREADS` was respected by the Rust runtime and
libextra/test got this for free

**2013:** rust-lang#7335 suggested splitting `RUST_TEST_TASKS` out
of `RUST_THREADS`.  In that issue and the implementation
(rust-lang#8823).

**2014:** rust-lang#13374 ask for support to disable capturing of
stdout/stderr.  `--nocapture` and `RUST_TEST_NOCAPTURE` were added
together.

**2015:** rust-lang#23525 renamed `RUST_TEST_TASKS` to
`RUST_TEST_THREADS`

**2016:** rust-lang#25636 asked to configure `RUST_TEST_THREADS`
via `--test-threads` which was implemented in rust-lang#35414

**2021:** rust-lang#85440 asked for test randomization which was
implemented in rust-lang#89082, adding `--shuffle` /
RUST_TEST_SHUFFLE` and `--shuffle-seed SEED` / `RUST_TEST_SHUFFLE_SEED`

Potentially relevant issues
---------------------------

- rust-lang#74845

@epage epage mentioned this pull request

Aug 11, 2025

Kobzol added a commit to Kobzol/rust that referenced this pull request

Aug 12, 2025
Deprecate RUST_TEST_* env variables

Like with rust-lang#139224, this is a documentation-only deprecation for now. Over time, we can
- warn and then remove on use of unstable environment variables
- warn on use of stable environment variables (no plan to remove due to compatibility)

Longer term, we expect test runners, like `cargo test`, to provide the necessary mechanisms for environmental or persistent configuration (e.g. using cargo config which supports `.cargo/config.toml` as well as environment variables).

This would include:
- `RUST_TEST_THREADS`
- `RUST_TEST_NOCAPTURE`
- `RUST_TEST_SHUFFLE` (unstable)
- `RUST_TEST_SHUFFLE_SEED` (unstable)

The primary outcomes for this change are
- Reducing the scope of what is expected for custom test harnesses to implement
- Reduce the mechanisms that test runners, like `cargo test`, are expected to track when they are being bypassed to protect against negative interactions, e.g.  `RUST_TEST_NOCAPTURE=1` when json output is being read.

For testing-devex FCP, see rust-lang/testing-devex-team#10

Fixes rust-lang/testing-devex-team#10

History
-------

At each step, I could not find evidence of design discussions on whether to support CLI, env, or both.  The first env variable seems to come from the fact that it was being forked out of an existing env variable that had a much wider scope.

At best, this seems like a way to offer a more persistent configuration for these flags but environment variables hidden away in libtest is a bit clunky and this seems like the wrong layer to handle this problem.

**Originally:** `RUST_THREADS` was respected by the Rust runtime and libextra/test got this for free

**2013:** rust-lang#7335 suggested splitting `RUST_TEST_TASKS` out of `RUST_THREADS`.  In that issue and the implementation (rust-lang#8823).

**2014:** rust-lang#13374 ask for support to disable capturing of stdout/stderr.  `--nocapture` and `RUST_TEST_NOCAPTURE` were added together.

**2015:** rust-lang#23525 renamed `RUST_TEST_TASKS` to `RUST_TEST_THREADS`

**2016:** rust-lang#25636 asked to configure `RUST_TEST_THREADS` via `--test-threads` which was implemented in rust-lang#35414

**2021:** rust-lang#85440 asked for test randomization which was implemented in rust-lang#89082, adding `--shuffle` / RUST_TEST_SHUFFLE` and `--shuffle-seed SEED` / `RUST_TEST_SHUFFLE_SEED`

Potentially relevant issues
---------------------------

- rust-lang#74845

jhpratt added a commit to jhpratt/rust that referenced this pull request

Aug 12, 2025
Deprecate RUST_TEST_* env variables

Like with rust-lang#139224, this is a documentation-only deprecation for now. Over time, we can
- warn and then remove on use of unstable environment variables
- warn on use of stable environment variables (no plan to remove due to compatibility)

Longer term, we expect test runners, like `cargo test`, to provide the necessary mechanisms for environmental or persistent configuration (e.g. using cargo config which supports `.cargo/config.toml` as well as environment variables).

This would include:
- `RUST_TEST_THREADS`
- `RUST_TEST_NOCAPTURE`
- `RUST_TEST_SHUFFLE` (unstable)
- `RUST_TEST_SHUFFLE_SEED` (unstable)

The primary outcomes for this change are
- Reducing the scope of what is expected for custom test harnesses to implement
- Reduce the mechanisms that test runners, like `cargo test`, are expected to track when they are being bypassed to protect against negative interactions, e.g.  `RUST_TEST_NOCAPTURE=1` when json output is being read.

For testing-devex FCP, see rust-lang/testing-devex-team#10

Fixes rust-lang/testing-devex-team#10

History
-------

At each step, I could not find evidence of design discussions on whether to support CLI, env, or both.  The first env variable seems to come from the fact that it was being forked out of an existing env variable that had a much wider scope.

At best, this seems like a way to offer a more persistent configuration for these flags but environment variables hidden away in libtest is a bit clunky and this seems like the wrong layer to handle this problem.

**Originally:** `RUST_THREADS` was respected by the Rust runtime and libextra/test got this for free

**2013:** rust-lang#7335 suggested splitting `RUST_TEST_TASKS` out of `RUST_THREADS`.  In that issue and the implementation (rust-lang#8823).

**2014:** rust-lang#13374 ask for support to disable capturing of stdout/stderr.  `--nocapture` and `RUST_TEST_NOCAPTURE` were added together.

**2015:** rust-lang#23525 renamed `RUST_TEST_TASKS` to `RUST_TEST_THREADS`

**2016:** rust-lang#25636 asked to configure `RUST_TEST_THREADS` via `--test-threads` which was implemented in rust-lang#35414

**2021:** rust-lang#85440 asked for test randomization which was implemented in rust-lang#89082, adding `--shuffle` / RUST_TEST_SHUFFLE` and `--shuffle-seed SEED` / `RUST_TEST_SHUFFLE_SEED`

Potentially relevant issues
---------------------------

- rust-lang#74845

jhpratt added a commit to jhpratt/rust that referenced this pull request

Aug 12, 2025
Deprecate RUST_TEST_* env variables

Like with rust-lang#139224, this is a documentation-only deprecation for now. Over time, we can
- warn and then remove on use of unstable environment variables
- warn on use of stable environment variables (no plan to remove due to compatibility)

Longer term, we expect test runners, like `cargo test`, to provide the necessary mechanisms for environmental or persistent configuration (e.g. using cargo config which supports `.cargo/config.toml` as well as environment variables).

This would include:
- `RUST_TEST_THREADS`
- `RUST_TEST_NOCAPTURE`
- `RUST_TEST_SHUFFLE` (unstable)
- `RUST_TEST_SHUFFLE_SEED` (unstable)

The primary outcomes for this change are
- Reducing the scope of what is expected for custom test harnesses to implement
- Reduce the mechanisms that test runners, like `cargo test`, are expected to track when they are being bypassed to protect against negative interactions, e.g.  `RUST_TEST_NOCAPTURE=1` when json output is being read.

For testing-devex FCP, see rust-lang/testing-devex-team#10

Fixes rust-lang/testing-devex-team#10

History
-------

At each step, I could not find evidence of design discussions on whether to support CLI, env, or both.  The first env variable seems to come from the fact that it was being forked out of an existing env variable that had a much wider scope.

At best, this seems like a way to offer a more persistent configuration for these flags but environment variables hidden away in libtest is a bit clunky and this seems like the wrong layer to handle this problem.

**Originally:** `RUST_THREADS` was respected by the Rust runtime and libextra/test got this for free

**2013:** rust-lang#7335 suggested splitting `RUST_TEST_TASKS` out of `RUST_THREADS`.  In that issue and the implementation (rust-lang#8823).

**2014:** rust-lang#13374 ask for support to disable capturing of stdout/stderr.  `--nocapture` and `RUST_TEST_NOCAPTURE` were added together.

**2015:** rust-lang#23525 renamed `RUST_TEST_TASKS` to `RUST_TEST_THREADS`

**2016:** rust-lang#25636 asked to configure `RUST_TEST_THREADS` via `--test-threads` which was implemented in rust-lang#35414

**2021:** rust-lang#85440 asked for test randomization which was implemented in rust-lang#89082, adding `--shuffle` / RUST_TEST_SHUFFLE` and `--shuffle-seed SEED` / `RUST_TEST_SHUFFLE_SEED`

Potentially relevant issues
---------------------------

- rust-lang#74845

jhpratt added a commit to jhpratt/rust that referenced this pull request

Aug 12, 2025
Deprecate RUST_TEST_* env variables

Like with rust-lang#139224, this is a documentation-only deprecation for now. Over time, we can
- warn and then remove on use of unstable environment variables
- warn on use of stable environment variables (no plan to remove due to compatibility)

Longer term, we expect test runners, like `cargo test`, to provide the necessary mechanisms for environmental or persistent configuration (e.g. using cargo config which supports `.cargo/config.toml` as well as environment variables).

This would include:
- `RUST_TEST_THREADS`
- `RUST_TEST_NOCAPTURE`
- `RUST_TEST_SHUFFLE` (unstable)
- `RUST_TEST_SHUFFLE_SEED` (unstable)

The primary outcomes for this change are
- Reducing the scope of what is expected for custom test harnesses to implement
- Reduce the mechanisms that test runners, like `cargo test`, are expected to track when they are being bypassed to protect against negative interactions, e.g.  `RUST_TEST_NOCAPTURE=1` when json output is being read.

For testing-devex FCP, see rust-lang/testing-devex-team#10

Fixes rust-lang/testing-devex-team#10

History
-------

At each step, I could not find evidence of design discussions on whether to support CLI, env, or both.  The first env variable seems to come from the fact that it was being forked out of an existing env variable that had a much wider scope.

At best, this seems like a way to offer a more persistent configuration for these flags but environment variables hidden away in libtest is a bit clunky and this seems like the wrong layer to handle this problem.

**Originally:** `RUST_THREADS` was respected by the Rust runtime and libextra/test got this for free

**2013:** rust-lang#7335 suggested splitting `RUST_TEST_TASKS` out of `RUST_THREADS`.  In that issue and the implementation (rust-lang#8823).

**2014:** rust-lang#13374 ask for support to disable capturing of stdout/stderr.  `--nocapture` and `RUST_TEST_NOCAPTURE` were added together.

**2015:** rust-lang#23525 renamed `RUST_TEST_TASKS` to `RUST_TEST_THREADS`

**2016:** rust-lang#25636 asked to configure `RUST_TEST_THREADS` via `--test-threads` which was implemented in rust-lang#35414

**2021:** rust-lang#85440 asked for test randomization which was implemented in rust-lang#89082, adding `--shuffle` / RUST_TEST_SHUFFLE` and `--shuffle-seed SEED` / `RUST_TEST_SHUFFLE_SEED`

Potentially relevant issues
---------------------------

- rust-lang#74845

Kobzol added a commit to Kobzol/rust that referenced this pull request

Aug 13, 2025
Deprecate RUST_TEST_* env variables

Like with rust-lang#139224, this is a documentation-only deprecation for now. Over time, we can
- warn and then remove on use of unstable environment variables
- warn on use of stable environment variables (no plan to remove due to compatibility)

Longer term, we expect test runners, like `cargo test`, to provide the necessary mechanisms for environmental or persistent configuration (e.g. using cargo config which supports `.cargo/config.toml` as well as environment variables).

This would include:
- `RUST_TEST_THREADS`
- `RUST_TEST_NOCAPTURE`
- `RUST_TEST_SHUFFLE` (unstable)
- `RUST_TEST_SHUFFLE_SEED` (unstable)

The primary outcomes for this change are
- Reducing the scope of what is expected for custom test harnesses to implement
- Reduce the mechanisms that test runners, like `cargo test`, are expected to track when they are being bypassed to protect against negative interactions, e.g.  `RUST_TEST_NOCAPTURE=1` when json output is being read.

For testing-devex FCP, see rust-lang/testing-devex-team#10

Fixes rust-lang/testing-devex-team#10

History
-------

At each step, I could not find evidence of design discussions on whether to support CLI, env, or both.  The first env variable seems to come from the fact that it was being forked out of an existing env variable that had a much wider scope.

At best, this seems like a way to offer a more persistent configuration for these flags but environment variables hidden away in libtest is a bit clunky and this seems like the wrong layer to handle this problem.

**Originally:** `RUST_THREADS` was respected by the Rust runtime and libextra/test got this for free

**2013:** rust-lang#7335 suggested splitting `RUST_TEST_TASKS` out of `RUST_THREADS`.  In that issue and the implementation (rust-lang#8823).

**2014:** rust-lang#13374 ask for support to disable capturing of stdout/stderr.  `--nocapture` and `RUST_TEST_NOCAPTURE` were added together.

**2015:** rust-lang#23525 renamed `RUST_TEST_TASKS` to `RUST_TEST_THREADS`

**2016:** rust-lang#25636 asked to configure `RUST_TEST_THREADS` via `--test-threads` which was implemented in rust-lang#35414

**2021:** rust-lang#85440 asked for test randomization which was implemented in rust-lang#89082, adding `--shuffle` / RUST_TEST_SHUFFLE` and `--shuffle-seed SEED` / `RUST_TEST_SHUFFLE_SEED`

Potentially relevant issues
---------------------------

- rust-lang#74845

rust-timer added a commit that referenced this pull request

Aug 13, 2025
Rollup merge of #145269 - epage:test-env, r=jhpratt

Deprecate RUST_TEST_* env variables

Like with #139224, this is a documentation-only deprecation for now. Over time, we can
- warn and then remove on use of unstable environment variables
- warn on use of stable environment variables (no plan to remove due to compatibility)

Longer term, we expect test runners, like `cargo test`, to provide the necessary mechanisms for environmental or persistent configuration (e.g. using cargo config which supports `.cargo/config.toml` as well as environment variables).

This would include:
- `RUST_TEST_THREADS`
- `RUST_TEST_NOCAPTURE`
- `RUST_TEST_SHUFFLE` (unstable)
- `RUST_TEST_SHUFFLE_SEED` (unstable)

The primary outcomes for this change are
- Reducing the scope of what is expected for custom test harnesses to implement
- Reduce the mechanisms that test runners, like `cargo test`, are expected to track when they are being bypassed to protect against negative interactions, e.g.  `RUST_TEST_NOCAPTURE=1` when json output is being read.

For testing-devex FCP, see rust-lang/testing-devex-team#10

Fixes rust-lang/testing-devex-team#10

History
-------

At each step, I could not find evidence of design discussions on whether to support CLI, env, or both.  The first env variable seems to come from the fact that it was being forked out of an existing env variable that had a much wider scope.

At best, this seems like a way to offer a more persistent configuration for these flags but environment variables hidden away in libtest is a bit clunky and this seems like the wrong layer to handle this problem.

**Originally:** `RUST_THREADS` was respected by the Rust runtime and libextra/test got this for free

**2013:** #7335 suggested splitting `RUST_TEST_TASKS` out of `RUST_THREADS`.  In that issue and the implementation (#8823).

**2014:** #13374 ask for support to disable capturing of stdout/stderr.  `--nocapture` and `RUST_TEST_NOCAPTURE` were added together.

**2015:** #23525 renamed `RUST_TEST_TASKS` to `RUST_TEST_THREADS`

**2016:** #25636 asked to configure `RUST_TEST_THREADS` via `--test-threads` which was implemented in #35414

**2021:** #85440 asked for test randomization which was implemented in #89082, adding `--shuffle` / RUST_TEST_SHUFFLE` and `--shuffle-seed SEED` / `RUST_TEST_SHUFFLE_SEED`

Potentially relevant issues
---------------------------

- #74845

github-actions bot pushed a commit to model-checking/verify-rust-std that referenced this pull request

Aug 18, 2025
Deprecate RUST_TEST_* env variables

Like with rust-lang#139224, this is a documentation-only deprecation for now. Over time, we can
- warn and then remove on use of unstable environment variables
- warn on use of stable environment variables (no plan to remove due to compatibility)

Longer term, we expect test runners, like `cargo test`, to provide the necessary mechanisms for environmental or persistent configuration (e.g. using cargo config which supports `.cargo/config.toml` as well as environment variables).

This would include:
- `RUST_TEST_THREADS`
- `RUST_TEST_NOCAPTURE`
- `RUST_TEST_SHUFFLE` (unstable)
- `RUST_TEST_SHUFFLE_SEED` (unstable)

The primary outcomes for this change are
- Reducing the scope of what is expected for custom test harnesses to implement
- Reduce the mechanisms that test runners, like `cargo test`, are expected to track when they are being bypassed to protect against negative interactions, e.g.  `RUST_TEST_NOCAPTURE=1` when json output is being read.

For testing-devex FCP, see rust-lang/testing-devex-team#10

Fixes rust-lang/testing-devex-team#10

History
-------

At each step, I could not find evidence of design discussions on whether to support CLI, env, or both.  The first env variable seems to come from the fact that it was being forked out of an existing env variable that had a much wider scope.

At best, this seems like a way to offer a more persistent configuration for these flags but environment variables hidden away in libtest is a bit clunky and this seems like the wrong layer to handle this problem.

**Originally:** `RUST_THREADS` was respected by the Rust runtime and libextra/test got this for free

**2013:** rust-lang#7335 suggested splitting `RUST_TEST_TASKS` out of `RUST_THREADS`.  In that issue and the implementation (rust-lang#8823).

**2014:** rust-lang#13374 ask for support to disable capturing of stdout/stderr.  `--nocapture` and `RUST_TEST_NOCAPTURE` were added together.

**2015:** rust-lang#23525 renamed `RUST_TEST_TASKS` to `RUST_TEST_THREADS`

**2016:** rust-lang#25636 asked to configure `RUST_TEST_THREADS` via `--test-threads` which was implemented in rust-lang#35414

**2021:** rust-lang#85440 asked for test randomization which was implemented in rust-lang#89082, adding `--shuffle` / RUST_TEST_SHUFFLE` and `--shuffle-seed SEED` / `RUST_TEST_SHUFFLE_SEED`

Potentially relevant issues
---------------------------

- rust-lang#74845