test_runner: add option to rerun only failed tests · nodejs/node@e076f78
@@ -153,6 +153,46 @@ test('skip() method with message', (t) => {
153153});
154154```
155155156+## Rerunning failed tests
157+158+The test runner supports persisting the state of the run to a file, allowing
159+the test runner to rerun failed tests without having to re-run the entire test suite.
160+Use the [`--test-rerun-failures`][] command-line option to specify a file path where the
161+state of the run is stored. if the state file does not exist, the test runner will
162+create it.
163+the state file is a JSON file that contains an array of run attempts.
164+Each run attempt is an object mapping successful tests to the attempt they have passed in.
165+The key identifying a test in this map is the test file path, with the line and column where the test is defined.
166+in a case where a test defined in a specific location is run multiple times,
167+for example within a function or a loop,
168+a counter will be appended to the key, to disambiguate the test runs.
169+note changing the order of test execution or the location of a test can lead the test runner
170+to consider tests as passed on a previous attempt,
171+meaning `--test-rerun-failures` should be used when tests run in a deterministic order.
172+173+example of a state file:
174+175+```json
176+[
177+ {
178+"test.js:10:5": { "passed_on_attempt": 0, "name": "test 1" },
179+ },
180+ {
181+"test.js:10:5": { "passed_on_attempt": 0, "name": "test 1" },
182+"test.js:20:5": { "passed_on_attempt": 1, "name": "test 2" }
183+ }
184+]
185+```
186+187+in this example, there are two run attempts, with two tests defined in `test.js`,
188+the first test succeeded on the first attempt, and the second test succeeded on the second attempt.
189+190+When the `--test-rerun-failures` option is used, the test runner will only run tests that have not yet passed.
191+192+```bash
193+node --test-rerun-failures /path/to/state/file
194+```
195+156196## TODO tests
157197158198Individual tests can be marked as flaky or incomplete by passing the `todo`
@@ -1342,6 +1382,9 @@ added:
13421382 - v18.9.0
13431383 - v16.19.0
13441384changes:
1385+ - version: REPLACEME
1386+ pr-url: https://github.com/nodejs/node/pull/59443
1387+ description: Added a rerunFailuresFilePath option.
13451388 - version: v23.0.0
13461389 pr-url: https://github.com/nodejs/node/pull/54705
13471390 description: Added the `cwd` option.
@@ -1432,6 +1475,10 @@ changes:
14321475 that specifies the index of the shard to run. This option is _required_.
14331476* `total` {number} is a positive integer that specifies the total number
14341477 of shards to split the test files to. This option is _required_.
1478+* `rerunFailuresFilePath` {string} A file path where the test runner will
1479+ store the state of the tests to allow rerunning only the failed tests on a next run.
1480+ see \[Rerunning failed tests]\[] for more information.
1481+**Default:** `undefined`.
14351482* `coverage` {boolean} enable [code coverage][] collection.
14361483**Default:** `false`.
14371484* `coverageExcludeGlobs` {string|Array} Excludes specific files from code coverage
@@ -3219,6 +3266,8 @@ Emitted when a test is enqueued for execution.
32193266* `cause` {Error} The actual error thrown by the test.
32203267* `type` {string|undefined} The type of the test, used to denote whether
32213268 this is a suite.
3269+* `attempt` {number|undefined} The attempt number of the test run,
3270+ present only when using the [`--test-rerun-failures`][] flag.
32223271* `file` {string|undefined} The path of the test file,
32233272`undefined` if test was run through the REPL.
32243273* `line` {number|undefined} The line number where the test is defined, or
@@ -3243,6 +3292,10 @@ The corresponding execution ordered event is `'test:complete'`.
32433292* `duration_ms` {number} The duration of the test in milliseconds.
32443293* `type` {string|undefined} The type of the test, used to denote whether
32453294 this is a suite.
3295+* `attempt` {number|undefined} The attempt number of the test run,
3296+ present only when using the [`--test-rerun-failures`][] flag.
3297+* `passed_on_attempt` {number|undefined} The attempt number the test passed on,
3298+ present only when using the [`--test-rerun-failures`][] flag.
32463299* `file` {string|undefined} The path of the test file,
32473300`undefined` if test was run through the REPL.
32483301* `line` {number|undefined} The line number where the test is defined, or
@@ -3946,6 +3999,7 @@ Can be used to abort test subtasks when the test has been aborted.
39463999[`--test-only`]: cli.md#--test-only
39474000[`--test-reporter-destination`]: cli.md#--test-reporter-destination
39484001[`--test-reporter`]: cli.md#--test-reporter
4002+[`--test-rerun-failures`]: cli.md#--test-rerun-failures
39494003[`--test-skip-pattern`]: cli.md#--test-skip-pattern
39504004[`--test-update-snapshots`]: cli.md#--test-update-snapshots
39514005[`--test`]: cli.md#--test