Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delay snapshotting various flakey UI tests #3667

Merged
merged 1 commit into from
Jan 8, 2025

Conversation

stefanceriu
Copy link
Member

We've been getting quite a few failures on AppLockSetupUITests but there's nothing obviously wrong with them. Running them repeatedly locally though eventually results in random snapshotting failures, which might very well be the real reason.

As such I went and added delays to all the UI tests that have been reported as flakey on Codecov 🤞

@stefanceriu stefanceriu added the pr-misc for other changes label Jan 8, 2025
@stefanceriu stefanceriu requested a review from a team as a code owner January 8, 2025 08:58
@stefanceriu stefanceriu requested review from Velin92 and pixlwave and removed request for a team and Velin92 January 8, 2025 08:58
Copy link
Member

@pixlwave pixlwave left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh, I didn't realise/forgot that second parameter was even there 🙈

Copy link

sonarqubecloud bot commented Jan 8, 2025

@pixlwave
Copy link
Member

pixlwave commented Jan 8, 2025

Note: I don't think this is going to fix the AppLock tests though. They're not failing with bad snapshots, they're failing with weird lock-ups that cause the test to time-out:

Screenshot 2025-01-08 at 9 06 09 am

Copy link

codecov bot commented Jan 8, 2025

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
908 1 907 0
View the full list of 1 ❄️ flaky tests
PreviewTests test_roomMemberDetailsScreen()

Flake rate in main: 13.04% (Passed 120 times, Failed 18 times)

Stack Traces | 1.97s run time
failed - Snapshot "Ignored User" does not match reference.

@−
"file:.../__Snapshots__/PreviewTests/test_roomMemberDetailsScreen-iPhone-16-pseudo.Ignored-User.png"
@+
"file:.../tmp/PreviewTests/test_roomMemberDetailsScreen-iPhone-16-pseudo.Ignored-User.png"

To configure output for a custom diff tool, use 'withSnapshotTesting'. For example:

    withSnapshotTesting(diffTool: .ksdiff) {
      // ...
    }

The percentage of pixels that match 0.8309326 is less than required 1.0
The lowest perceptual color precision 0.39937502 is less than required 0.98 (PreviewTests/Sources/PreviewTests.swift:85)

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

@stefanceriu stefanceriu merged commit fc719d9 into develop Jan 8, 2025
10 checks passed
@stefanceriu stefanceriu deleted the stefan/uiTestFailures branch January 8, 2025 10:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-misc for other changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants