Raw lifetimes #1603

compiler-errors · 2024-09-08T03:21:30Z

src/tokens.md

mattheww · 2024-09-09T22:06:15Z

The

(not immediately followed by ')

bit is there in LIFETIME_OR_LABEL as a way to say that 'xxx'yyy is rejected (rather than being interpreted as two lifetimes).

But (after rust-lang/rust#126452) 'r#xxx'yyy is interpreted as two lifetimes. So I think that bit should be left out of the RAW_LIFETIME rule.

(There's no need to say anything special to indicate that 'r#kw' is rejected: that's true because ' on its own isn't a token, and character literals can't have more than one character between the quotes.)

Alternatively, it might be worth considering changing the implementation to reject 'r#xxx'yyy for consistency with the non-raw case.

Note that allowing that form would close the door to delaying the rejection of overlong character literals to post-expansion, which was being considered late last year at rust-lang/rust#118699 .

ehuss

Can you update https://doc.rust-lang.org/nightly/reference/tokens.html#reserved-prefixes to include a new rule for the reserved prefix lifetime? I would assume it is something like

`'` (IDENTIFIER_OR_KEYWORD | _) `#`

And update the examples in the "Edition differences" in that section to include reserved lifetime prefixes.

Can you add a "Raw lifetime or label" section similar to https://doc.rust-lang.org/nightly/reference/identifiers.html#raw-identifiers that explains what the raw lifetime means?

Can you add an "Edition differences" block in the "Lifetimes and loop labels" section that explains that raw lifetimes are not supported before 2021? There are some examples within the tokens.md chapter of "Edition differences" blocks for the kind of wording to use.

src/tokens.md

ehuss · 2024-10-22T21:40:28Z

@compiler-errors Just checking if you'll be able to look at the requested changes?

compiler-errors · 2024-10-30T00:49:46Z

I believe I addressed the comments, but please look closely because markdown is not my thing.

compiler-errors · 2024-10-30T00:50:08Z

Well, except for:

Alternatively, it might be worth considering changing the implementation to reject 'r#xxx'yyy for consistency with the non-raw case.

Which I think I will open a rustc PR to implement.

compiler-errors · 2024-10-30T03:07:25Z

@rustbot ready

ehuss · 2024-10-30T14:11:10Z

src/tokens.md

+> &nbsp;&nbsp; | RAW_LIFETIME
+>
+> RAW_LIFETIME :\
+> &nbsp;&nbsp; `'r#` [IDENTIFIER_OR_KEYWORD][identifier]


I believe this should also include _, right? That is, 'r#_ is valid, correct?

I also just want to double-check, it is intentional that 'r#_ is allowed as a loop label? (I only ask because '_ is explicitly not allowed).

Suggested change

>    `'r#` [IDENTIFIER_OR_KEYWORD][identifier]

>       `'r#` [IDENTIFIER_OR_KEYWORD][identifier]

> _(not immediately followed by `'`)_\

>    | `'r#_`

Actually this is likely something else we want to deny here. I feel like 'r#_ is invalid, or at least not something we need to support initially.

rust-lang/rust#132363

Hopefully resolved by 912a6d5

mattheww · 2024-10-30T15:18:13Z

Do we need changes outside the Lexical structure chapter?

As far as I can see nothing is saying that r#a and 'a are treated as equivalent when used as a lifetime or label.

That is, I don't think the Reference is saying that these are now accepted:

fn foo<'r#a>(s: &'a str) {}

    'r#a: { break 'a; }

mattheww · 2024-10-30T15:21:43Z

src/tokens.md

@@ -851,7 +866,8 @@ r[lex.token.reserved-prefix.syntax]
 > **<sup>Lexer 2021+</sup>**\
 > RESERVED_TOKEN_DOUBLE_QUOTE : ( IDENTIFIER_OR_KEYWORD <sub>_Except `b` or `c` or `r` or `br` or `cr`_</sub> | `_` ) `"`\
 > RESERVED_TOKEN_SINGLE_QUOTE : ( IDENTIFIER_OR_KEYWORD <sub>_Except `b`_</sub> | `_` ) `'`\
-> RESERVED_TOKEN_POUND : ( IDENTIFIER_OR_KEYWORD <sub>_Except `r` or `br` or `cr`_</sub> | `_` ) `#`
+> RESERVED_TOKEN_POUND : ( IDENTIFIER_OR_KEYWORD <sub>_Except `r` or `br` or `cr`_</sub> | `_` ) `#`\
+> RESERVED_TOKEN_LIFETIME : `'` (IDENTIFIER_OR_KEYWORD <sub>_Except `r`_</sub> | _) `#`


I think this is isn't reserving 'r# on its own, which rustc rejects but could lex as 'r followed by #.

But I'm not sure it's worth trying to get all the RESERVED_ rules exactly right at this stage.

ehuss · 2024-10-30T17:21:01Z

Posted rust-lang/edition-guide#330 as the companion for the edition.

…r=chenyukang Reject raw lifetime followed by `'`, like regular lifetimes do See comment. We want to reject cases like `'r#long'id`, which currently gets interpreted as a raw lifetime (`'r#long`) followed by a lifetime (`'id`). This could have alternative lexes, such as an overlong char literal (`'r#long'`) followed by an identifier (`id`). To avoid committing to this in any case, let's reject the whole thing. `@mattheww,` is this what you were looking for in rust-lang/reference#1603 (comment)? I'd say ignore the details about the specific error message (the fact that this gets reinterpreted as a char literal is 🤷), just that because this causes a lexer error we're effectively saving syntactical space like you wanted.

ehuss · 2024-11-05T22:59:54Z

@rustbot author

…r=chenyukang Reject raw lifetime followed by `'`, like regular lifetimes do See comment. We want to reject cases like `'r#long'id`, which currently gets interpreted as a raw lifetime (`'r#long`) followed by a lifetime (`'id`). This could have alternative lexes, such as an overlong char literal (`'r#long'`) followed by an identifier (`id`). To avoid committing to this in any case, let's reject the whole thing. `@mattheww,` is this what you were looking for in rust-lang/reference#1603 (comment)? I'd say ignore the details about the specific error message (the fact that this gets reinterpreted as a char literal is 🤷), just that because this causes a lexer error we're effectively saving syntactical space like you wanted.

…=wesleywiser Enforce that raw lifetimes must be valid raw identifiers Make sure that the identifier part of a raw lifetime is a valid raw identifier. This precludes `'r#_` and all module segment paths for now. I don't believe this is compelling to support. This was raised by `@ehuss` in rust-lang/reference#1603 (comment) (well, specifically the `'r#_` case), but I don't see why we shouldn't just make it consistent with raw identifiers.

Rollup merge of rust-lang#132363 - compiler-errors:raw-lt-id-valid, r=wesleywiser Enforce that raw lifetimes must be valid raw identifiers Make sure that the identifier part of a raw lifetime is a valid raw identifier. This precludes `'r#_` and all module segment paths for now. I don't believe this is compelling to support. This was raised by `@ehuss` in rust-lang/reference#1603 (comment) (well, specifically the `'r#_` case), but I don't see why we shouldn't just make it consistent with raw identifiers.

Rollup merge of rust-lang#132341 - compiler-errors:raw-lt-prefix-id, r=chenyukang Reject raw lifetime followed by `'`, like regular lifetimes do See comment. We want to reject cases like `'r#long'id`, which currently gets interpreted as a raw lifetime (`'r#long`) followed by a lifetime (`'id`). This could have alternative lexes, such as an overlong char literal (`'r#long'`) followed by an identifier (`id`). To avoid committing to this in any case, let's reject the whole thing. `@mattheww,` is this what you were looking for in rust-lang/reference#1603 (comment)? I'd say ignore the details about the specific error message (the fact that this gets reinterpreted as a char literal is 🤷), just that because this causes a lexer error we're effectively saving syntactical space like you wanted.

…r=chenyukang Reject raw lifetime followed by `'`, like regular lifetimes do See comment. We want to reject cases like `'r#long'id`, which currently gets interpreted as a raw lifetime (`'r#long`) followed by a lifetime (`'id`). This could have alternative lexes, such as an overlong char literal (`'r#long'`) followed by an identifier (`id`). To avoid committing to this in any case, let's reject the whole thing. `@mattheww,` is this what you were looking for in rust-lang/reference#1603 (comment)? I'd say ignore the details about the specific error message (the fact that this gets reinterpreted as a char literal is 🤷), just that because this causes a lexer error we're effectively saving syntactical space like you wanted.

…=wesleywiser Enforce that raw lifetimes must be valid raw identifiers Make sure that the identifier part of a raw lifetime is a valid raw identifier. This precludes `'r#_` and all module segment paths for now. I don't believe this is compelling to support. This was raised by `@ehuss` in rust-lang/reference#1603 (comment) (well, specifically the `'r#_` case), but I don't see why we shouldn't just make it consistent with raw identifiers.

ehuss · 2024-11-14T17:16:11Z

@compiler-errors I pushed a change for the validation, and also rebased since we made a slight change in the definition of the lifetime token in #1668.

Please let me know if you think the current version looks good.

Update for rust-lang/rust#132363

These are rejected by the lexer.

ehuss · 2024-11-15T04:01:17Z

I pushed a commit specifying that 'r#_ will generate an error (and similarly for r#_).

This is a bit of an awkward issue around whether or not _ is an identifier. Currently the reference does not define it that way (intentionally), even though internally rustc allows that (but quickly rejects it). IIRC, there are some issues around that and proc-macros, but I did not look into that.

I did not want to define an identifier token that allowed _, because then I would have to define separate identifier tokens that exclude it, and this _ identifier would only be used by these raw tokens.

compiler-errors · 2024-11-20T04:28:45Z

LGTM!

compiler-errors commented Sep 8, 2024

View reviewed changes

src/tokens.md Outdated Show resolved Hide resolved

ehuss reviewed Sep 10, 2024

View reviewed changes

src/tokens.md Show resolved Hide resolved

src/tokens.md Show resolved Hide resolved

src/tokens.md Outdated Show resolved Hide resolved

ehuss added the S-waiting-on-author Status: The marked PR is awaiting some action (such as code changes) from the PR author. label Sep 10, 2024

compiler-errors force-pushed the patch-1 branch from e718896 to 0b6ce17 Compare October 30, 2024 00:49

compiler-errors mentioned this pull request Oct 30, 2024

Reject raw lifetime followed by ', like regular lifetimes do rust-lang/rust#132341

Merged

rustbot added S-waiting-on-review Status: The marked PR is awaiting review from the PR author. and removed S-waiting-on-author Status: The marked PR is awaiting some action (such as code changes) from the PR author. labels Oct 30, 2024

ehuss reviewed Oct 30, 2024

View reviewed changes

compiler-errors mentioned this pull request Oct 30, 2024

Enforce that raw lifetimes must be valid raw identifiers rust-lang/rust#132363

Merged

mattheww reviewed Oct 30, 2024

View reviewed changes

rustbot added S-waiting-on-author Status: The marked PR is awaiting some action (such as code changes) from the PR author. and removed S-waiting-on-review Status: The marked PR is awaiting review from the PR author. labels Nov 5, 2024

compiler-errors added 3 commits November 14, 2024 08:13

Raw lifetimes

ef87ed6

Don't allow 'r#kw'

df53c5e

update

5c0c306

ehuss force-pushed the patch-1 branch from 0b6ce17 to 912a6d5 Compare November 14, 2024 17:13

ehuss added 2 commits November 14, 2024 19:16

Clarify raw lifetime cannot be certain keywords

f04cd63

Update for rust-lang/rust#132363

Add rule name for raw lifetime edition differences

f490b7c

ehuss force-pushed the patch-1 branch from 912a6d5 to f490b7c Compare November 15, 2024 03:17

Document r#_ and 'r#_ as reserved

baa3d73

These are rejected by the lexer.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Raw lifetimes #1603

Raw lifetimes #1603

compiler-errors commented Sep 8, 2024

mattheww commented Sep 9, 2024

ehuss left a comment

ehuss commented Oct 22, 2024

compiler-errors commented Oct 30, 2024

compiler-errors commented Oct 30, 2024

compiler-errors commented Oct 30, 2024

ehuss Oct 30, 2024

compiler-errors Oct 30, 2024

ehuss Nov 14, 2024

mattheww commented Oct 30, 2024

mattheww Oct 30, 2024

ehuss commented Oct 30, 2024

ehuss commented Nov 5, 2024

ehuss commented Nov 14, 2024

ehuss commented Nov 15, 2024

compiler-errors commented Nov 20, 2024

-> &nbsp;&nbsp; `'r#` [IDENTIFIER_OR_KEYWORD][identifier]
+> &nbsp;&nbsp; &nbsp;&nbsp; `'r#` [IDENTIFIER_OR_KEYWORD][identifier]
+>   _(not immediately followed by `'`)_\
+> &nbsp;&nbsp; | `'r#_`

Raw lifetimes #1603

Are you sure you want to change the base?

Raw lifetimes #1603

Conversation

compiler-errors commented Sep 8, 2024

mattheww commented Sep 9, 2024

ehuss left a comment

Choose a reason for hiding this comment

ehuss commented Oct 22, 2024

compiler-errors commented Oct 30, 2024

compiler-errors commented Oct 30, 2024

compiler-errors commented Oct 30, 2024

ehuss Oct 30, 2024

Choose a reason for hiding this comment

compiler-errors Oct 30, 2024

Choose a reason for hiding this comment

ehuss Nov 14, 2024

Choose a reason for hiding this comment

mattheww commented Oct 30, 2024

mattheww Oct 30, 2024

Choose a reason for hiding this comment

ehuss commented Oct 30, 2024

ehuss commented Nov 5, 2024

ehuss commented Nov 14, 2024

ehuss commented Nov 15, 2024

compiler-errors commented Nov 20, 2024