reduce the amount of panics in `{TokenStream, Literal}::from_str` calls #147859

cyrgani · 2025-10-18T20:13:31Z

Before this PR, calling TokenStream::from_str or Literal::from_str with an invalid argument would always cause a compile error, even if the TokenStream is not used afterwards at all.
This PR changes this so it returns a LexError instead in some cases.

This is very theoretically a breaking change, but the doc comment on the impl already says

/// NOTE: some errors may cause panics instead of returning `LexError`. We reserve the right to
/// change these errors into `LexError`s later.

Fixes some cases of #58736.

rustbot · 2025-10-18T20:13:34Z

rust-analyzer is developed in its own repository. If possible, consider making this change to rust-lang/rust-analyzer instead.

cc @rust-lang/rust-analyzer

rustbot · 2025-10-18T20:13:36Z

r? @davidtwco

rustbot has assigned @davidtwco.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

cyrgani · 2025-10-18T21:47:52Z

@rustbot author

rustbot · 2025-10-18T21:47:56Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

cyrgani · 2025-10-19T10:12:43Z

@rustbot ready

tests/ui/proc-macro/invalid-punct-ident-4.rs

cyrgani · 2025-10-23T14:03:39Z

This does not fix #58736 entirely yet:

TokenStream::from_str("0b2");

is still a compile error because it internally calls literal_from_str, which then emits the diagnostic.

davidtwco · 2025-10-25T11:33:55Z

I'm not familiar enough with this part of the compiler

r? compiler

add larger test for `proc_macro` `FromStr` implementations Currently, there are only few tests that check the output of `TokenStream::from_str` and `Literal::from_str` (which is somewhat understandable as the rustc implementation just delegates these calls to the parser). In preparation for both the standalone backend (rust-lang#130856) which will probably need to reimplement this logic as well as for removing panics from these functions (rust-lang#58736), this PR adds a test which shows the various messy ways of how these functions report errors and the return values for successful parses. Followup PRs such as rust-lang#147859 will change more and more of these "diagnostic + error"s into `LexErrors`. The test structure with the extra module is used to allow reusing it later easily for the standalone backend.

bors · 2025-12-01T18:24:22Z

☔ The latest upstream changes (presumably #149499) made this pull request unmergeable. Please resolve the merge conflicts.

JonathanBrouwer · 2025-12-15T13:33:20Z

I think there is still work to do in improving the error messages, but this can happen in future PRs, this is already a big improvement.
One final concern I have: Are there any stability guarantees for the Display implementation of LexError? I'm assuming there wouldn't (or at least shouldn't) be but not exactly sure if/where this is documented.
This PR changes the Display implementation of the type & I'd like to leave the option of improving these messages later open.

JonathanBrouwer · 2025-12-15T17:44:35Z

@rustbot author

JonathanBrouwer · 2025-12-22T20:56:11Z

Started a thread here to discuss this #t-libs > Questions for a PR I'm reviewing @ 💬

JonathanBrouwer

From a T-compiler perspective this code looks fine, but I'm not super familiar with this code and feel like T-libs should also have a say about it, so will re-assign.
r? libs

View changes since this review

JonathanBrouwer · 2026-01-27T16:27:06Z

compiler/rustc_errors/src/diagnostic.rs

@@ -1378,6 +1378,13 @@ impl<'a, G: EmissionGuarantee> Diag<'a, G> {
        drop(self);


Will put the text from my comment above in a discussion so it is not missed:

One final concern I have: Are there any stability guarantees for the Display implementation of LexError? I'm assuming there wouldn't (or at least shouldn't) be but not exactly sure if/where this is documented.
This PR changes the Display implementation of the type & I'd like to leave the option of improving these messages later open.

joboet · 2026-01-27T19:20:51Z

I'm not very familiar with proc_macro.

r? @petrochenkov

rustbot · 2026-01-27T21:29:16Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

rustbot assigned davidtwco Oct 18, 2025

This comment has been minimized.

Sign in to view

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 18, 2025

cyrgani force-pushed the nonfatal-tokenstream-parse branch from e2c0009 to 80b67bb Compare October 19, 2025 09:03

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 19, 2025

bjorn3 reviewed Oct 20, 2025

View reviewed changes

tests/ui/proc-macro/invalid-punct-ident-4.rs Show resolved Hide resolved

cyrgani force-pushed the nonfatal-tokenstream-parse branch from ba098bc to 0274d78 Compare October 21, 2025 14:28

rustbot assigned wesleywiser and unassigned davidtwco Oct 25, 2025

cyrgani force-pushed the nonfatal-tokenstream-parse branch from 0274d78 to 13d3d06 Compare October 29, 2025 22:19

This comment has been minimized.

Sign in to view

cyrgani changed the title ~~replace TokenStream::from_str panics with LexErrors~~ reduce the amount of panics in {TokenStream, Literal}::from_str calls Oct 29, 2025

cyrgani mentioned this pull request Nov 4, 2025

add larger test for proc_macro FromStr implementations #148505

Merged

rustbot added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Nov 29, 2025

rust-cloud-vms bot force-pushed the nonfatal-tokenstream-parse branch from 91a861c to 24b96f1 Compare December 14, 2025 22:01

This comment has been minimized.

Sign in to view

cyrgani added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Dec 14, 2025

rust-cloud-vms bot force-pushed the nonfatal-tokenstream-parse branch from 24b96f1 to d847df3 Compare December 14, 2025 22:12

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 15, 2025

This comment has been minimized.

Sign in to view

rust-cloud-vms bot force-pushed the nonfatal-tokenstream-parse branch from d847df3 to e21dfa8 Compare January 25, 2026 22:23

This comment has been minimized.

Sign in to view

rust-cloud-vms bot force-pushed the nonfatal-tokenstream-parse branch 2 times, most recently from 6f82492 to 33ff2a3 Compare January 26, 2026 10:56

JonathanBrouwer approved these changes Jan 27, 2026

View reviewed changes

rustbot added the T-libs Relevant to the library team, which will review and decide on the PR/issue. label Jan 27, 2026

rustbot assigned joboet and unassigned JonathanBrouwer Jan 27, 2026

JonathanBrouwer reviewed Jan 27, 2026

View reviewed changes

rustbot assigned petrochenkov and unassigned joboet Jan 27, 2026

This comment has been minimized.

Sign in to view

reduce the amount of panics in {TokenStream, Literal}::from_str calls

519a3f7

rust-cloud-vms bot force-pushed the nonfatal-tokenstream-parse branch from 33ff2a3 to 519a3f7 Compare January 27, 2026 21:29

		@@ -1378,6 +1378,13 @@ impl<'a, G: EmissionGuarantee> Diag<'a, G> {
		drop(self);

Uh oh!

reduce the amount of panics in {TokenStream, Literal}::from_str calls #147859

Are you sure you want to change the base?

reduce the amount of panics in {TokenStream, Literal}::from_str calls #147859

Uh oh!

Conversation

cyrgani commented Oct 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Oct 18, 2025

Uh oh!

rustbot commented Oct 18, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

cyrgani commented Oct 18, 2025

Uh oh!

rustbot commented Oct 18, 2025

Uh oh!

cyrgani commented Oct 19, 2025

Uh oh!

Uh oh!

cyrgani commented Oct 23, 2025

Uh oh!

davidtwco commented Oct 25, 2025

Uh oh!

This comment has been minimized.

bors commented Dec 1, 2025

Uh oh!

This comment has been minimized.

JonathanBrouwer commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JonathanBrouwer commented Dec 15, 2025

Uh oh!

JonathanBrouwer commented Dec 22, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

JonathanBrouwer left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JonathanBrouwer Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

joboet commented Jan 27, 2026

Uh oh!

This comment has been minimized.

rustbot commented Jan 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

reduce the amount of panics in `{TokenStream, Literal}::from_str` calls #147859

reduce the amount of panics in `{TokenStream, Literal}::from_str` calls #147859

cyrgani commented Oct 18, 2025 •

edited

Loading

JonathanBrouwer commented Dec 15, 2025 •

edited

Loading

JonathanBrouwer left a comment •

edited by rustbot

Loading