tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) by Enselic · Pull Request #143002 · rust-lang/rust
rustbot
added
A-compiletest
labels
Jun 25, 2025
Enselic
added
S-waiting-on-author
and removed S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.labels
Jun 25, 2025label
Jun 25, 2025
rustbot
added
S-waiting-on-author
and removed S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.labels
Jul 17, 2025…t ok) And introduce two new directives for ui tests: * `run-crash` * `run-fail-or-crash` Normally a `run-fail` ui test like tests that panic shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal/crash however. Introduce and use `run-crash` for those tests. Note that Windows crashes are not handled by signals but by certain high bits set on the process exit code. Example exit code for crash on Windows: `0xc000001d`. Because of this, we define "crash" on all platforms as "not exit with success and not exit with a regular failure code in the range 1..=127". Some tests behave differently on different targets: * Targets without unwind support will abort (crash) instead of exit with failure code 101 after panicking. As a special case, allow crashes for `run-fail` tests for such targets. * Different sanitizer implementations handle detected memory problems differently. Some abort (crash) the process while others exit with failure code 1. Introduce and use `run-fail-or-crash` for such tests.
bors
added
S-waiting-on-bors
and removed S-waiting-on-author
Status: This is awaiting some action (such as code changes or more information) from the author.labels
Jul 19, 2025
bors
mentioned this pull request
bors
mentioned this pull request
github-actions bot pushed a commit to rust-lang/rustc-dev-guide that referenced this pull request
Jul 21, 2025…etrochenkov tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) And introduce two new directives for ui tests: * `run-crash` * `run-fail-or-crash` Normally a `run-fail` ui test like tests that panic shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal/crash however. Introduce and use `run-crash` for those tests. Note that Windows crashes are not handled by signals but by certain high bits set on the process exit code. Example exit code for crash on Windows: `0xc000001d` (`STATUS_ILLEGAL_INSTRUCTION`). Because of this, we define "crash" on all platforms as "not exit with success and not exit with a regular failure code in the range 1..=127". Some tests behave differently on different targets: * Targets without unwind support will abort (crash) instead of exit with failure code 101 after panicking. As a special case, allow crashes for `run-fail` tests for such targets. * Different sanitizer implementations handle detected memory problems differently. Some abort (crash) the process while others exit with failure code 1. Introduce and use `run-fail-or-crash` for such tests. This adds further (cc rust-lang/rust#142304, rust-lang/rust#142886) protection against the regression in rust-lang/rust#123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [x] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [x] test all permutations of actual vs expected **Done:** See rust-lang/rust#143002 (comment). - [x] Handle targets without unwind support - [x] Add `run-fail-or-crash` for some sanitizer tests - [x] remote-test-client. See rust-lang/rust#143448 ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 try-job: aarch64-apple try-job: x86_64-msvc-1 try-job: x86_64-gnu try-job: dist-i586-gnu-i586-i686-musl try-job: test-various try-job: armhf-gnu
Muscraft pushed a commit to Muscraft/rust that referenced this pull request
Jul 21, 2025…, r=Mark-Simulacrum remote-test-client: Exit code `128 + <signal-number>` instead of `3` If the remote process is terminated by a signal, make `remote-test-client` exit with the code `128 + <signal-number>` instead of always `3`. This follows common practice among tools such as bash [^1]: > When a command terminates on a fatal signal whose number is N, Bash uses the > value 128+N as the exit status. It also allows us to differentiate between `run-pass` and `run-crash` ui tests without special case code in compiletest for that when `remote-test-client` is used. See rust-lang#143002 and in particular rust-lang#143002 (comment). Exiting with code `3` has been done from the start (see rust-lang#39400) and seems arbitrary rather than a deliberate design decision, so changing it does not seem like an extraordinarily big deal. ### Regression testing Note that rust-lang#143002 will act as a regression test once it is rebased on this PR. ### Why a separate PR I think it is comforting to know that CI does not break with just this change. But if my reviewer prefers, we can move this commit to be part of rust-lang#143002 instead. [^1]: https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html
Muscraft pushed a commit to Muscraft/rust that referenced this pull request
Jul 21, 2025…gnal, r=petrochenkov tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) And introduce two new directives for ui tests: * `run-crash` * `run-fail-or-crash` Normally a `run-fail` ui test like tests that panic shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal/crash however. Introduce and use `run-crash` for those tests. Note that Windows crashes are not handled by signals but by certain high bits set on the process exit code. Example exit code for crash on Windows: `0xc000001d` (`STATUS_ILLEGAL_INSTRUCTION`). Because of this, we define "crash" on all platforms as "not exit with success and not exit with a regular failure code in the range 1..=127". Some tests behave differently on different targets: * Targets without unwind support will abort (crash) instead of exit with failure code 101 after panicking. As a special case, allow crashes for `run-fail` tests for such targets. * Different sanitizer implementations handle detected memory problems differently. Some abort (crash) the process while others exit with failure code 1. Introduce and use `run-fail-or-crash` for such tests. This adds further (cc rust-lang#142304, rust-lang#142886) protection against the regression in rust-lang#123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [x] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [x] test all permutations of actual vs expected **Done:** See rust-lang#143002 (comment). - [x] Handle targets without unwind support - [x] Add `run-fail-or-crash` for some sanitizer tests - [x] remote-test-client. See rust-lang#143448 ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 try-job: aarch64-apple try-job: x86_64-msvc-1 try-job: x86_64-gnu try-job: dist-i586-gnu-i586-i686-musl try-job: test-various try-job: armhf-gnu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters