Include executed tests in the build metrics (and use a custom test display impl) by emilyalbini · Pull Request #108659 · rust-lang/rust
added 3 commits
March 2, 2023 16:20
rustbot
added
A-testsuite
labels
Mar 2, 2023Behind the scenes Clippy uses compiletest-rs, which doesn't support the --json flag we added to Rust's compiletest.
This was referenced
Mar 21, 2023
ehuss
mentioned this pull request
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request
Mar 25, 2023…_output, r=pietroalbini Bugfix: avoid panic on invalid json output from libtest rust-lang#108659 introduces a custom test display implementation. It does so by using libtest to output json. The stdout is read and parsed; The code trims the line read and checks whether it starts with a `{` and ends with a `}`. If so, it concludes that it must be a json encoded `Message`. Unfortunately, this does not work in all cases: - This assumes that tests running with `--nocapture` will never start and end lines with `{` and `}` characters - Output is generated by issuing multiple `write_message` [statements](https://github.com/rust-lang/rust/blob/master/library/test/src/formatters/json.rs#L33-L60). Where only the last one issues a `\n`. This likely results in a race condition as we see multiple json outputs on the same line when running tests for the `x86_64-fortanix-unknown-sgx` target: ``` 10:21:04 �[0m�[0m�[1m�[32m Running�[0m tests/run-time-detect.rs (build/x86_64-unknown-linux-gnu/stage1-std/x86_64-fortanix-unknown-sgx/release/deps/run_time_detect-8c66026bd4b1871a) 10:21:04 10:21:04 running 1 tests 10:21:04 test x86_all ... ok 10:21:04 �[0m�[0m�[1m�[32m Running�[0m tests/thread.rs (build/x86_64-unknown-linux-gnu/stage1-std/x86_64-fortanix-unknown-sgx/release/deps/thread-ed5456a7d80a6193) 10:21:04 thread 'main' panicked at 'failed to parse libtest json output; error: trailing characters at line 1 column 135, line: "{ \"type\": \"suite\", \"event\": \"ok\", \"passed\": 1, \"failed\": 0, \"ignored\": 0, \"measured\": 0, \"filtered_out\": 0, \"exec_time\": 0.000725911 }{ \"type\": \"suite\", \"event\": \"started\", \"test_count\": 1 }\n"', render_tests.rs:108:25 ``` This PR implements a partial fix by being much more conservative of what it asserts is a valid json encoded `Message`. This prevents panics, but still does not resolve the race condition. A discussion is needed where this race condition comes from exactly and how it best can be avoided. cc: `@jethrogb,` `@pietroalbini`
bors added a commit to rust-lang-ci/rust that referenced this pull request
Apr 5, 2023…huss Validate `ignore` and `only` compiletest directive, and add human-readable ignore reasons This PR adds strict validation for the `ignore` and `only` compiletest directives, failing if an unknown value is provided to them. Doing so uncovered 79 tests in `tests/ui` that had invalid directives, so this PR also fixes them. Finally, this PR adds human-readable ignore reasons when tests are ignored due to `ignore` or `only` directives, like *"only executed when the architecture is aarch64"* or *"ignored when the operative system is windows"*. This was the original reason why I started working on this PR and rust-lang#108659, as we need both of them for Ferrocene. The PR is a draft because the code is extremely inefficient: it calls `rustc --print=cfg --target $target` for every rustc target (to gather the list of allowed ignore values), which on my system takes between 4s and 5s, and performs a lot of allocations of constant values. I'll fix both of them in the coming days. r? `@ehuss`
This was referenced
Apr 11, 2023bors added a commit to rust-lang-ci/rust that referenced this pull request
Apr 14, 2023fix running Miri tests This partially reverts rust-lang#108659 to fix rust-lang#110102: the Miri test runner does not support any flags, they are interpreted as filters instead which leads to no tests being run. I have not checked any of the other test runners for whether they are having any trouble with these flags. Cc `@pietroalbini` `@Mark-Simulacrum` `@jyn514`
oli-obk pushed a commit to oli-obk/miri that referenced this pull request
Apr 17, 2023fix running Miri tests This partially reverts rust-lang/rust#108659 to fix rust-lang/rust#110102: the Miri test runner does not support any flags, they are interpreted as filters instead which leads to no tests being run. I have not checked any of the other test runners for whether they are having any trouble with these flags. Cc `@pietroalbini` `@Mark-Simulacrum` `@jyn514`
bors added a commit to rust-lang-ci/rust that referenced this pull request
May 2, 2023…ynchronization, r=pietroalbini Ensure test library issues json string line-by-line rust-lang#108659 introduces a custom test display implementation. It does so by using libtest to output json. The stdout is read line by line and parsed. The code trims the line read and checks whether it starts with a `{` and ends with a `}`. Unfortunately, there is a race condition in how json data is written to stdout. The `write_message` function calls `self.out.write_all` repeatedly to write a buffer that contains (partial) json data, or a new line. There is no lock around the `self.out.write_all` functions. Similarly, the `write_message` function itself is called with only partial json data. As these functions are called from concurrent threads, this may result in json data ending up on the same stdout line. This PR avoids this by buffering the complete json data before issuing a single `self.out.write_all`. (rust-lang#109484 implemented a partial fix for this issue; it only avoids that failed json parsing would result in a panic.) cc: `@jethrogb,` `@pietroalbini`
ehuss
mentioned this pull request
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request
Apr 8, 2025compiletest: Remove the `--logfile` flag This flag is deprecated in libtest (rust-lang#134283), and there's no evidence in-tree of this flag actually being passed to compiletest. For detailed information about test results, bootstrap parses JSON output from compiletest instead (rust-lang#108659). As part of my experimental work on removing the libtest dependency from compiletest, it's useful to be able to disconnect libtest functionality that isn't needed.
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request
Apr 9, 2025Rollup merge of rust-lang#139528 - Zalathar:no-logfile, r=jieyouxu compiletest: Remove the `--logfile` flag This flag is deprecated in libtest (rust-lang#134283), and there's no evidence in-tree of this flag actually being passed to compiletest. For detailed information about test results, bootstrap parses JSON output from compiletest instead (rust-lang#108659). As part of my experimental work on removing the libtest dependency from compiletest, it's useful to be able to disconnect libtest functionality that isn't needed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters