Remove `Lexer`'s dependency on `Parser`. by nnethercote · Pull Request #134192 · rust-lang/rust

@nnethercote marked this pull request as ready for review

December 12, 2024 01:05

@rustbot rustbot added S-waiting-on-review

Status: Awaiting review from the assignee but also interested parties.

T-compiler

Relevant to the compiler team, which will review and decide on the PR/issue.

labels

Dec 12, 2024

compiler-errors

@nnethercote

Lexing precedes parsing, as you'd expect: `Lexer` creates a
`TokenStream` and `Parser` then parses that `TokenStream`.

But, in a horrendous violation of layering abstractions and common
sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err`
method does some error recovery that relies on creating a `Parser` to do
some post-processing of the `TokenStream` that the `Lexer` just created.

This commit just removes `unclosed_delim_err`. This change removes
`Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s
return value can have a more typical form.

The cost is slightly worse error messages in two obscure cases, as shown
in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less
  explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff
  marker detection is no longer supported (because that detection is
  implemented in the parser).

In my opinion this cost is outweighed by the magnitude of the code
cleanup.

@bors bors added S-waiting-on-bors

Status: Waiting on bors to run and complete tests. Bors will change the label on completion.

and removed S-waiting-on-review

Status: Awaiting review from the assignee but also interested parties.

labels

Dec 12, 2024

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request

Dec 12, 2024
…=compiler-errors

Remove `Lexer`'s dependency on `Parser`.

Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.

But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.

This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.

The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).

In my opinion this cost is outweighed by the magnitude of the code cleanup.

r? `@chenyukang`

bors added a commit to rust-lang-ci/rust that referenced this pull request

Dec 13, 2024
…iaskrgr

Rollup of 7 pull requests

Successful merges:

 - rust-lang#130060 (Autodiff Upstreaming - rustc_codegen_llvm changes)
 - rust-lang#132038 (Add lint rule for `#[deprecated]` on re-exports)
 - rust-lang#133937 (Keep track of parse errors in `mod`s and don't emit resolve errors for paths involving them)
 - rust-lang#133942 (Clarify how to use `black_box()`)
 - rust-lang#134081 (Try to evaluate constants in legacy mangling)
 - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.)
 - rust-lang#134209 (validate `--skip` and `--exclude` paths)

Failed merges:

 - rust-lang#133099 (forbid toggling x87 and fpregs on hard-float targets)

r? `@ghost`
`@rustbot` modify labels: rollup

compiler-errors added a commit to compiler-errors/rust that referenced this pull request

Dec 13, 2024
…=compiler-errors

Remove `Lexer`'s dependency on `Parser`.

Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.

But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.

This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.

The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).

In my opinion this cost is outweighed by the magnitude of the code cleanup.

r? ``@chenyukang``

bors added a commit to rust-lang-ci/rust that referenced this pull request

Dec 13, 2024

Zalathar added a commit to Zalathar/rust that referenced this pull request

Dec 13, 2024
…=compiler-errors

Remove `Lexer`'s dependency on `Parser`.

Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.

But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.

This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.

The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).

In my opinion this cost is outweighed by the magnitude of the code cleanup.

r? `@chenyukang`

bors added a commit to rust-lang-ci/rust that referenced this pull request

Dec 13, 2024
Rollup of 9 pull requests

Successful merges:

 - rust-lang#130060 (Autodiff Upstreaming - rustc_codegen_llvm changes)
 - rust-lang#132038 (Add lint rule for `#[deprecated]` on re-exports)
 - rust-lang#132150 (Fix powerpc64 big-endian FreeBSD ABI)
 - rust-lang#133633 (don't show the full linker args unless `--verbose` is passed)
 - rust-lang#133942 (Clarify how to use `black_box()`)
 - rust-lang#134081 (Try to evaluate constants in legacy mangling)
 - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.)
 - rust-lang#134208 (coverage: Tidy up creation of covmap and covfun records)
 - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables)

Failed merges:

 - rust-lang#133099 (forbid toggling x87 and fpregs on hard-float targets)

r? `@ghost`
`@rustbot` modify labels: rollup

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request

Dec 13, 2024
…=compiler-errors

Remove `Lexer`'s dependency on `Parser`.

Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.

But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.

This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.

The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).

In my opinion this cost is outweighed by the magnitude of the code cleanup.

r? ``@chenyukang``

bors added a commit to rust-lang-ci/rust that referenced this pull request

Dec 13, 2024
…iaskrgr

Rollup of 8 pull requests

Successful merges:

 - rust-lang#132038 (Add lint rule for `#[deprecated]` on re-exports)
 - rust-lang#132150 (Fix powerpc64 big-endian FreeBSD ABI)
 - rust-lang#133633 (don't show the full linker args unless `--verbose` is passed)
 - rust-lang#133942 (Clarify how to use `black_box()`)
 - rust-lang#134081 (Try to evaluate constants in legacy mangling)
 - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.)
 - rust-lang#134208 (coverage: Tidy up creation of covmap and covfun records)
 - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables)

r? `@ghost`
`@rustbot` modify labels: rollup

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request

Dec 13, 2024
…=compiler-errors

Remove `Lexer`'s dependency on `Parser`.

Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.

But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.

This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.

The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).

In my opinion this cost is outweighed by the magnitude of the code cleanup.

r? ```@chenyukang```

bors added a commit to rust-lang-ci/rust that referenced this pull request

Dec 13, 2024
…iaskrgr

Rollup of 7 pull requests

Successful merges:

 - rust-lang#132038 (Add lint rule for `#[deprecated]` on re-exports)
 - rust-lang#132150 (Fix powerpc64 big-endian FreeBSD ABI)
 - rust-lang#133633 (don't show the full linker args unless `--verbose` is passed)
 - rust-lang#133942 (Clarify how to use `black_box()`)
 - rust-lang#134081 (Try to evaluate constants in legacy mangling)
 - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.)
 - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables)

r? `@ghost`
`@rustbot` modify labels: rollup

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request

Dec 13, 2024
…=compiler-errors

Remove `Lexer`'s dependency on `Parser`.

Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.

But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.

This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.

The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).

In my opinion this cost is outweighed by the magnitude of the code cleanup.

r? ````@chenyukang````

bors added a commit to rust-lang-ci/rust that referenced this pull request

Dec 13, 2024
Rollup of 3 pull requests

Successful merges:

 - rust-lang#134081 (Try to evaluate constants in legacy mangling)
 - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.)
 - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables)

r? `@ghost`
`@rustbot` modify labels: rollup

try-job: dist-x86_64-linux

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request

Dec 13, 2024
…=compiler-errors

Remove `Lexer`'s dependency on `Parser`.

Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.

But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.

This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.

The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).

In my opinion this cost is outweighed by the magnitude of the code cleanup.

r? `````@chenyukang`````

Zalathar added a commit to Zalathar/rust that referenced this pull request

Dec 14, 2024
…=compiler-errors

Remove `Lexer`'s dependency on `Parser`.

Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.

But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.

This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.

The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).

In my opinion this cost is outweighed by the magnitude of the code cleanup.

r? ``````@chenyukang``````

bors added a commit to rust-lang-ci/rust that referenced this pull request

Dec 14, 2024
…iaskrgr

Rollup of 7 pull requests

Successful merges:

 - rust-lang#132150 (Fix powerpc64 big-endian FreeBSD ABI)
 - rust-lang#133633 (don't show the full linker args unless `--verbose` is passed)
 - rust-lang#133942 (Clarify how to use `black_box()`)
 - rust-lang#134081 (Try to evaluate constants in legacy mangling)
 - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.)
 - rust-lang#134208 (coverage: Tidy up creation of covmap and covfun records)
 - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables)

r? `@ghost`
`@rustbot` modify labels: rollup

bors added a commit to rust-lang-ci/rust that referenced this pull request

Dec 14, 2024
…iaskrgr

Rollup of 6 pull requests

Successful merges:

 - rust-lang#132150 (Fix powerpc64 big-endian FreeBSD ABI)
 - rust-lang#133942 (Clarify how to use `black_box()`)
 - rust-lang#134081 (Try to evaluate constants in legacy mangling)
 - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.)
 - rust-lang#134208 (coverage: Tidy up creation of covmap and covfun records)
 - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables)

r? `@ghost`
`@rustbot` modify labels: rollup

bors added a commit to rust-lang-ci/rust that referenced this pull request

Dec 14, 2024
…iaskrgr

Rollup of 6 pull requests

Successful merges:

 - rust-lang#132150 (Fix powerpc64 big-endian FreeBSD ABI)
 - rust-lang#133942 (Clarify how to use `black_box()`)
 - rust-lang#134081 (Try to evaluate constants in legacy mangling)
 - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.)
 - rust-lang#134208 (coverage: Tidy up creation of covmap and covfun records)
 - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables)

r? `@ghost`
`@rustbot` modify labels: rollup

rust-timer added a commit to rust-lang-ci/rust that referenced this pull request

Dec 14, 2024
Rollup merge of rust-lang#134192 - nnethercote:rm-Lexer-Parser-dep, r=compiler-errors

Remove `Lexer`'s dependency on `Parser`.

Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.

But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.

This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.

The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).

In my opinion this cost is outweighed by the magnitude of the code cleanup.

r? ```````@chenyukang```````

github-actions bot pushed a commit to tautschnig/verify-rust-std that referenced this pull request

Mar 11, 2025
…iaskrgr

Rollup of 6 pull requests

Successful merges:

 - rust-lang#132150 (Fix powerpc64 big-endian FreeBSD ABI)
 - rust-lang#133942 (Clarify how to use `black_box()`)
 - rust-lang#134081 (Try to evaluate constants in legacy mangling)
 - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.)
 - rust-lang#134208 (coverage: Tidy up creation of covmap and covfun records)
 - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables)

r? `@ghost`
`@rustbot` modify labels: rollup