Remove `Lexer`'s dependency on `Parser`. by nnethercote · Pull Request #134192 · rust-lang/rust
marked this pull request as ready for review
rustbot
added
S-waiting-on-review
labels
Dec 12, 2024Lexing precedes parsing, as you'd expect: `Lexer` creates a
`TokenStream` and `Parser` then parses that `TokenStream`.
But, in a horrendous violation of layering abstractions and common
sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err`
method does some error recovery that relies on creating a `Parser` to do
some post-processing of the `TokenStream` that the `Lexer` just created.
This commit just removes `unclosed_delim_err`. This change removes
`Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s
return value can have a more typical form.
The cost is slightly worse error messages in two obscure cases, as shown
in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less
explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff
marker detection is no longer supported (because that detection is
implemented in the parser).
In my opinion this cost is outweighed by the magnitude of the code
cleanup.
bors
added
S-waiting-on-bors
and removed S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.labels
Dec 12, 2024matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request
Dec 12, 2024…=compiler-errors
Remove `Lexer`'s dependency on `Parser`.
Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.
But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.
This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.
The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).
In my opinion this cost is outweighed by the magnitude of the code cleanup.
r? `@chenyukang`
bors added a commit to rust-lang-ci/rust that referenced this pull request
Dec 13, 2024…iaskrgr Rollup of 7 pull requests Successful merges: - rust-lang#130060 (Autodiff Upstreaming - rustc_codegen_llvm changes) - rust-lang#132038 (Add lint rule for `#[deprecated]` on re-exports) - rust-lang#133937 (Keep track of parse errors in `mod`s and don't emit resolve errors for paths involving them) - rust-lang#133942 (Clarify how to use `black_box()`) - rust-lang#134081 (Try to evaluate constants in legacy mangling) - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.) - rust-lang#134209 (validate `--skip` and `--exclude` paths) Failed merges: - rust-lang#133099 (forbid toggling x87 and fpregs on hard-float targets) r? `@ghost` `@rustbot` modify labels: rollup
compiler-errors added a commit to compiler-errors/rust that referenced this pull request
Dec 13, 2024…=compiler-errors
Remove `Lexer`'s dependency on `Parser`.
Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.
But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.
This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.
The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).
In my opinion this cost is outweighed by the magnitude of the code cleanup.
r? ``@chenyukang``
Zalathar added a commit to Zalathar/rust that referenced this pull request
Dec 13, 2024…=compiler-errors
Remove `Lexer`'s dependency on `Parser`.
Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.
But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.
This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.
The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).
In my opinion this cost is outweighed by the magnitude of the code cleanup.
r? `@chenyukang`
bors added a commit to rust-lang-ci/rust that referenced this pull request
Dec 13, 2024Rollup of 9 pull requests Successful merges: - rust-lang#130060 (Autodiff Upstreaming - rustc_codegen_llvm changes) - rust-lang#132038 (Add lint rule for `#[deprecated]` on re-exports) - rust-lang#132150 (Fix powerpc64 big-endian FreeBSD ABI) - rust-lang#133633 (don't show the full linker args unless `--verbose` is passed) - rust-lang#133942 (Clarify how to use `black_box()`) - rust-lang#134081 (Try to evaluate constants in legacy mangling) - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.) - rust-lang#134208 (coverage: Tidy up creation of covmap and covfun records) - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables) Failed merges: - rust-lang#133099 (forbid toggling x87 and fpregs on hard-float targets) r? `@ghost` `@rustbot` modify labels: rollup
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request
Dec 13, 2024…=compiler-errors
Remove `Lexer`'s dependency on `Parser`.
Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.
But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.
This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.
The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).
In my opinion this cost is outweighed by the magnitude of the code cleanup.
r? ``@chenyukang``
bors added a commit to rust-lang-ci/rust that referenced this pull request
Dec 13, 2024…iaskrgr Rollup of 8 pull requests Successful merges: - rust-lang#132038 (Add lint rule for `#[deprecated]` on re-exports) - rust-lang#132150 (Fix powerpc64 big-endian FreeBSD ABI) - rust-lang#133633 (don't show the full linker args unless `--verbose` is passed) - rust-lang#133942 (Clarify how to use `black_box()`) - rust-lang#134081 (Try to evaluate constants in legacy mangling) - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.) - rust-lang#134208 (coverage: Tidy up creation of covmap and covfun records) - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables) r? `@ghost` `@rustbot` modify labels: rollup
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request
Dec 13, 2024…=compiler-errors
Remove `Lexer`'s dependency on `Parser`.
Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.
But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.
This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.
The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).
In my opinion this cost is outweighed by the magnitude of the code cleanup.
r? ```@chenyukang```
bors added a commit to rust-lang-ci/rust that referenced this pull request
Dec 13, 2024…iaskrgr Rollup of 7 pull requests Successful merges: - rust-lang#132038 (Add lint rule for `#[deprecated]` on re-exports) - rust-lang#132150 (Fix powerpc64 big-endian FreeBSD ABI) - rust-lang#133633 (don't show the full linker args unless `--verbose` is passed) - rust-lang#133942 (Clarify how to use `black_box()`) - rust-lang#134081 (Try to evaluate constants in legacy mangling) - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.) - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables) r? `@ghost` `@rustbot` modify labels: rollup
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request
Dec 13, 2024…=compiler-errors
Remove `Lexer`'s dependency on `Parser`.
Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.
But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.
This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.
The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).
In my opinion this cost is outweighed by the magnitude of the code cleanup.
r? ````@chenyukang````
bors added a commit to rust-lang-ci/rust that referenced this pull request
Dec 13, 2024Rollup of 3 pull requests Successful merges: - rust-lang#134081 (Try to evaluate constants in legacy mangling) - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.) - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables) r? `@ghost` `@rustbot` modify labels: rollup try-job: dist-x86_64-linux
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request
Dec 13, 2024…=compiler-errors
Remove `Lexer`'s dependency on `Parser`.
Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.
But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.
This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.
The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).
In my opinion this cost is outweighed by the magnitude of the code cleanup.
r? `````@chenyukang`````
Zalathar added a commit to Zalathar/rust that referenced this pull request
Dec 14, 2024…=compiler-errors
Remove `Lexer`'s dependency on `Parser`.
Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`.
But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created.
This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form.
The cost is slightly worse error messages in two obscure cases, as shown in these tests:
- tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`.
- tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser).
In my opinion this cost is outweighed by the magnitude of the code cleanup.
r? ``````@chenyukang``````
bors added a commit to rust-lang-ci/rust that referenced this pull request
Dec 14, 2024…iaskrgr Rollup of 7 pull requests Successful merges: - rust-lang#132150 (Fix powerpc64 big-endian FreeBSD ABI) - rust-lang#133633 (don't show the full linker args unless `--verbose` is passed) - rust-lang#133942 (Clarify how to use `black_box()`) - rust-lang#134081 (Try to evaluate constants in legacy mangling) - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.) - rust-lang#134208 (coverage: Tidy up creation of covmap and covfun records) - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables) r? `@ghost` `@rustbot` modify labels: rollup
bors added a commit to rust-lang-ci/rust that referenced this pull request
Dec 14, 2024…iaskrgr Rollup of 6 pull requests Successful merges: - rust-lang#132150 (Fix powerpc64 big-endian FreeBSD ABI) - rust-lang#133942 (Clarify how to use `black_box()`) - rust-lang#134081 (Try to evaluate constants in legacy mangling) - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.) - rust-lang#134208 (coverage: Tidy up creation of covmap and covfun records) - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables) r? `@ghost` `@rustbot` modify labels: rollup
bors added a commit to rust-lang-ci/rust that referenced this pull request
Dec 14, 2024…iaskrgr Rollup of 6 pull requests Successful merges: - rust-lang#132150 (Fix powerpc64 big-endian FreeBSD ABI) - rust-lang#133942 (Clarify how to use `black_box()`) - rust-lang#134081 (Try to evaluate constants in legacy mangling) - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.) - rust-lang#134208 (coverage: Tidy up creation of covmap and covfun records) - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables) r? `@ghost` `@rustbot` modify labels: rollup
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request
Dec 14, 2024Rollup merge of rust-lang#134192 - nnethercote:rm-Lexer-Parser-dep, r=compiler-errors Remove `Lexer`'s dependency on `Parser`. Lexing precedes parsing, as you'd expect: `Lexer` creates a `TokenStream` and `Parser` then parses that `TokenStream`. But, in a horrendous violation of layering abstractions and common sense, `Lexer` depends on `Parser`! The `Lexer::unclosed_delim_err` method does some error recovery that relies on creating a `Parser` to do some post-processing of the `TokenStream` that the `Lexer` just created. This commit just removes `unclosed_delim_err`. This change removes `Lexer`'s dependency on `Parser`, and also means that `lex_token_tree`'s return value can have a more typical form. The cost is slightly worse error messages in two obscure cases, as shown in these tests: - tests/ui/parser/brace-in-let-chain.rs: there is slightly less explanation in this case involving an extra `{`. - tests/ui/parser/diff-markers/unclosed-delims{,-in-macro}.rs: the diff marker detection is no longer supported (because that detection is implemented in the parser). In my opinion this cost is outweighed by the magnitude of the code cleanup. r? ```````@chenyukang```````
github-actions bot pushed a commit to tautschnig/verify-rust-std that referenced this pull request
Mar 11, 2025…iaskrgr Rollup of 6 pull requests Successful merges: - rust-lang#132150 (Fix powerpc64 big-endian FreeBSD ABI) - rust-lang#133942 (Clarify how to use `black_box()`) - rust-lang#134081 (Try to evaluate constants in legacy mangling) - rust-lang#134192 (Remove `Lexer`'s dependency on `Parser`.) - rust-lang#134208 (coverage: Tidy up creation of covmap and covfun records) - rust-lang#134211 (On Neutrino QNX, reduce the need to set archiver via environment variables) r? `@ghost` `@rustbot` modify labels: rollup
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters