[draft] tweak MaskErrors to handle pre_execution_errors and execution errors

[dextermb]

Description

An initial attempt at updating the processing for MaskErrors to handle pre-execution errors and execution errors properly.

Types of Changes

• Core
• Bugfix
• New feature
• Enhancement/optimization
• Documentation

Issues Fixed or Closed by This PR

• Flaky error handling with MaskErrors #3844
• MaskErrors cannot deal with PreExecutionResult #3959

Checklist

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• I have tested the changes and verified that they work and don't break anything (as well as I can manage).

Summary by Sourcery

Improve MaskErrors extension to separately process and anonymize pre-execution and execution errors

Bug Fixes:

• Ensure pre-execution errors are properly masked
• Handle execution errors directly on the result without relying on initial_result

Enhancements:

• Extract common error masking logic into a new _process_errors method

Documentation:

• Add RELEASE.md entry for minor release update

[sourcery-ai]

Reviewer's Guide

Refactor the MaskErrors extension by extracting common error masking logic into a new helper and applying it consistently to both pre-execution and execution errors in on_operation, along with adding a corresponding release note.

File-Level Changes

Change	Details	Files
Extract and unify error masking logic into a helper	Removed obsolete _process_result method Added _process_errors(list) returning processed error list Reworked loop to accept any error list and return masked/unmasked errors	strawberry/extensions/mask_errors.py
Apply unified error masking in on_operation	Process and replace pre_execution_errors via _process_errors Simplify execution result handling by checking result.errors and masking them Removed ExecutionResult type check and initial_result branch	strawberry/extensions/mask_errors.py
Add release note for minor update	Created RELEASE.md with minor release entry for MaskErrors changes	RELEASE.md

Possibly linked issues

• Flaky error handling with MaskErrors #3844: The PR updates MaskErrors to correctly process pre-execution errors, resolving the 'initial_result' attribute error described in the issue.
• Flaky error handling with MaskErrors #3844: The PR updates MaskErrors to handle pre-execution and execution errors, directly addressing the issue's problem with validation errors.
• Flaky error handling with MaskErrors #3844: The PR fixes the MaskErrors extension to handle pre-execution and execution errors, directly addressing the bug where MaskErrors was not working.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[dextermb]

I've still got to properly test this, but I thought I'd open a draft pull request to get some initial feedback

[patrick91]

Look alright! But I'll do a proper review once we get tests :D

[codecov]

Codecov Report

❌ Patch coverage is 87.50000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 94.41%. Comparing base (1bbe8db) to head (446a9a1).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3968      +/-   ##
==========================================
+ Coverage   94.40%   94.41%   +0.01%     
==========================================
  Files         528      528              
  Lines       34371    34368       -3     
  Branches     1803     1801       -2     
==========================================
+ Hits        32449    32450       +1     
+ Misses       1630     1627       -3     
+ Partials      292      291       -1     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[codspeed-hq]

CodSpeed Performance Report

Merging #3968 will not alter performance

_{Comparing dextermb:fix/mask-errors (446a9a1) with main (1bbe8db)}

Summary

✅ 26 untouched benchmarks

[dextermb]

Look alright! But I'll do a proper review once we get tests :D

Yeah, need to look into how to set them up for the extension as there weren't any originally

[patrick91]

there's some here: https://github.com/strawberry-graphql/strawberry/blob/main/tests/schema/extensions/test_mask_errors.py

[Speedy1991]

Hey @dextermb are you still working on this PR? I can provide a test if needed - I'm just not sure how i can contribute to another author branch :)

Test:

@pytest.mark.asyncio
async def test_mask_errors_with_validation_error_async():

    @strawberry.type
    class Query:
        @strawberry.field
        def test_field(self) -> str:
            return 'TestField'

    schema = strawberry.Schema(query=Query, extensions=[MaskErrors()])

    query = "query { testField( }" # missing closing brace should raise a PreExecutionError (validation)

    # Use async execution to ensure we get StrawberryExecutionResult path
    result = await schema.execute(query)
    assert isinstance(result, PreExecutionError) # <<<< TODO SMELLY?
    assert result.errors is not None
    formatted_errors = [err.formatted for err in result.errors]
    assert formatted_errors == [
        {
            "locations": [{"column": 20, "line": 1}],
            "message": "Unexpected error.",
        }
    ]

But i found another code smell in https://github.com/strawberry-graphql/strawberry/blob/main/strawberry/schema/schema.py#L602. This returns an Optional[PreExecutionError] which returns immediately an PreExecutionError but async def execute...) -> ExecutionResult should be an execution result and not an PreExeuctionError - not sure how we should handle this @patrick91?