tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) by Enselic · Pull Request #143002 · rust-lang/rust

@rustbot rustbot added A-compiletest

Area: The compiletest test runner

A-testsuite

Area: The testsuite used to check the correctness of rustc

S-waiting-on-review

Status: Awaiting review from the assignee but also interested parties.

T-bootstrap

Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

T-compiler

Relevant to the compiler team, which will review and decide on the PR/issue.

labels

Jun 25, 2025

@Enselic Enselic added S-waiting-on-author

Status: This is awaiting some action (such as code changes or more information) from the author.

and removed S-waiting-on-review

Status: Awaiting review from the assignee but also interested parties.

labels

Jun 25, 2025

@rustbot rustbot added the T-infra

Relevant to the infrastructure team, which will review and decide on the PR/issue.

label

Jun 25, 2025

rust-bors bot added a commit that referenced this pull request

Jun 25, 2025

@rust-bors

rust-bors bot added a commit that referenced this pull request

Jun 25, 2025

@rust-bors

@rustbot rustbot added S-waiting-on-author

Status: This is awaiting some action (such as code changes or more information) from the author.

and removed S-waiting-on-review

Status: Awaiting review from the assignee but also interested parties.

labels

Jul 17, 2025

@Enselic

…t ok)

And introduce two new directives for ui tests:
* `run-crash`
* `run-fail-or-crash`

Normally a `run-fail` ui test like tests that panic shall not be
terminated by a signal like `SIGABRT`. So begin having that as a hard
requirement.

Some of our current tests do terminate by a signal/crash however.
Introduce and use `run-crash` for those tests. Note that Windows crashes
are not handled by signals but by certain high bits set on the process
exit code. Example exit code for crash on Windows: `0xc000001d`.
Because of this, we define "crash" on all platforms as "not exit with
success and not exit with a regular failure code in the range 1..=127".

Some tests behave differently on different targets:
* Targets without unwind support will abort (crash) instead of exit with
  failure code 101 after panicking. As a special case, allow crashes for
  `run-fail` tests for such targets.
* Different sanitizer implementations handle detected memory problems
  differently. Some abort (crash) the process while others exit with
  failure code 1. Introduce and use `run-fail-or-crash` for such tests.

@bors bors added S-waiting-on-bors

Status: Waiting on bors to run and complete tests. Bors will change the label on completion.

and removed S-waiting-on-author

Status: This is awaiting some action (such as code changes or more information) from the author.

labels

Jul 19, 2025

@bors bors mentioned this pull request

Jul 20, 2025

@bors bors mentioned this pull request

Jul 20, 2025

github-actions bot pushed a commit to rust-lang/rustc-dev-guide that referenced this pull request

Jul 21, 2025
…etrochenkov

tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok)

And introduce two new directives for ui tests:
* `run-crash`
* `run-fail-or-crash`

Normally a `run-fail` ui test like tests that panic shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement.

Some of our current tests do terminate by a signal/crash however.  Introduce and use `run-crash` for those tests. Note that Windows crashes are not handled by signals but by certain high bits set on the process  exit code. Example exit code for crash on Windows: `0xc000001d` (`STATUS_ILLEGAL_INSTRUCTION`).  Because of this, we define "crash" on all platforms as "not exit with success and not exit with a regular failure code in the range 1..=127".

Some tests behave differently on different targets:
* Targets without unwind support will abort (crash) instead of exit with   failure code 101 after panicking. As a special case, allow crashes for   `run-fail` tests for such targets.
* Different sanitizer implementations handle detected memory problems   differently. Some abort (crash) the process while others exit with   failure code 1. Introduce and use `run-fail-or-crash` for such tests.

This adds further (cc rust-lang/rust#142304, rust-lang/rust#142886) protection against the regression in rust-lang/rust#123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally).

### TODO
- [x] **Q:** what about on Windows?
    **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix.
- [x] test all permutations of actual vs expected
    **Done:** See rust-lang/rust#143002 (comment).
- [x] Handle targets without unwind support
- [x] Add `run-fail-or-crash` for some sanitizer tests
- [x] remote-test-client. See rust-lang/rust#143448

### Zulip discussion

See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235

try-job: aarch64-apple
try-job: x86_64-msvc-1
try-job: x86_64-gnu
try-job: dist-i586-gnu-i586-i686-musl
try-job: test-various
try-job: armhf-gnu

Muscraft pushed a commit to Muscraft/rust that referenced this pull request

Jul 21, 2025
…, r=Mark-Simulacrum

remote-test-client: Exit code `128 + <signal-number>` instead of `3`

If the remote process is terminated by a signal, make `remote-test-client` exit with the code `128 + <signal-number>` instead of always `3`. This follows common practice among tools such as bash [^1]:

> When a command terminates on a fatal signal whose number is N, Bash uses the
> value 128+N as the exit status.

It also allows us to differentiate between `run-pass` and `run-crash` ui tests without special case code in compiletest for that when `remote-test-client` is used. See rust-lang#143002 and in particular rust-lang#143002 (comment).

Exiting with code `3` has been done from the start (see rust-lang#39400) and seems arbitrary rather than a deliberate design decision, so changing it does not seem like an extraordinarily big deal.

### Regression testing

Note that rust-lang#143002 will act as a regression test once it is rebased on this PR.

### Why a separate PR

I think it is comforting to know that CI does not break with just this change. But if my reviewer prefers, we can move this commit to be part of rust-lang#143002 instead.

[^1]: https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html

Muscraft pushed a commit to Muscraft/rust that referenced this pull request

Jul 21, 2025
…gnal, r=petrochenkov

tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok)

And introduce two new directives for ui tests:
* `run-crash`
* `run-fail-or-crash`

Normally a `run-fail` ui test like tests that panic shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement.

Some of our current tests do terminate by a signal/crash however.  Introduce and use `run-crash` for those tests. Note that Windows crashes are not handled by signals but by certain high bits set on the process  exit code. Example exit code for crash on Windows: `0xc000001d` (`STATUS_ILLEGAL_INSTRUCTION`).  Because of this, we define "crash" on all platforms as "not exit with success and not exit with a regular failure code in the range 1..=127".

Some tests behave differently on different targets:
* Targets without unwind support will abort (crash) instead of exit with   failure code 101 after panicking. As a special case, allow crashes for   `run-fail` tests for such targets.
* Different sanitizer implementations handle detected memory problems   differently. Some abort (crash) the process while others exit with   failure code 1. Introduce and use `run-fail-or-crash` for such tests.

This adds further (cc rust-lang#142304, rust-lang#142886) protection against the regression in rust-lang#123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally).

### TODO
- [x] **Q:** what about on Windows?
    **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix.
- [x] test all permutations of actual vs expected
    **Done:** See rust-lang#143002 (comment).
- [x] Handle targets without unwind support
- [x] Add `run-fail-or-crash` for some sanitizer tests
- [x] remote-test-client. See rust-lang#143448

### Zulip discussion

See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235

try-job: aarch64-apple
try-job: x86_64-msvc-1
try-job: x86_64-gnu
try-job: dist-i586-gnu-i586-i686-musl
try-job: test-various
try-job: armhf-gnu