[BUG]: Tales of Rebirth software rendering regression

[macrosscape]

Describe the Bug

Tales of Rebirth software rendering broke with PR #11324 in PCSX2 v1.7.5865 (June 4th 2024)
It's been broken for half a year and causes the backgrounds to tear when using software rendering.

DRAGONBLEAPIECE asked me to make this report.

Reproduction Steps

Walk around Sulz, the first town, or any town, with software rendering on.

Expected Behavior

No response

PCSX2 Revision

Latest

Operating System

Windows 10 (64bit)

If Linux - Specify Distro

No response

CPU

Ryzen 5 3600

GPU

rx570

GS Settings

No response

Emulation Settings

No response

GS Window Screenshots

No response

Logs & Dumps

No response

[Mrlinkwii]

can i have a memcard if possible

[Mrlinkwii]

heres a memcard for testing memcard_issue_12100.zip

[Fis-Ura]

This happends in Hardware Rending as well
Steps to reproduce the bug.
fight a battle on the overworld
open the main menu
close the main menu
open the main menu again

This will stay broken until another battle is triggered

[badspott]

https://files.catbox.moe/xo3s3k.mp4
Footage of the bug in question if it is needed.

[lightningterror]

@macrosscape Can you post a gs dump of the issue?

[DRAGONBLE3API3C3]

https://drive.google.com/file/d/1TMqnqOPaDy20YjU1SW1eHkkMjkkgEbt8/view?usp=sharing

[DRAGONBLE3API3C3]

there's a gs dump. while I'm at it, here's a memory card. walk into the town with it set to software renderer.
https://drive.google.com/file/d/1zf_GBr3c1NG3YKeVtwupBwyHObtD49Gx/view?usp=drive_link

[lightningterror]

Just posting the debug progress so I/we don't forget what's happening:

Reverting this gets around the issue:
https://github.com/PCSX2/pcsx2/pull/11324/files#diff-b7112f98608190fe3ac7cf4e147bbc393fb36953955c2c97c563d213904ee47bR3980

I compared the draws, first draw indicated in the log says 68, however when I compare the dumped draws it starts from 3 which is itex alpha

draw 3 info:
itex alpha:
master:

revert:

draw:
00003_vertex.txt
00003_context.txt

draw 68 info:
draw:
00068_vertex.txt
00068_context.txt

Currently on master it breaks because inclusive_y_req is 0 in these cases

TODO: Find out why inclusive_y_req is 0 in the conditions and if there's something we can do about it.

[TJnotJT]

Hi Lightning, I have also been looking at this bug, hope that is ok!

Thanks for sharing the above, here is what I have found also: The glitch seems to be caused by bilinear filtering and goes away when bilinear filtering is disabled. Particularly, it seems to caused by some of the 4 UV coordinates used in the bilinear interpolation going outside the texture by 1 pixel and becoming pure black. This causes the alpha of the final color to be slightly less than 128, which causes it to fail the alpha test (the alpha test is set to GEQUAL with AREF=128). The blue streaks are from the background not being covered up.

It is strange that this seems to only affect software rendering though.

[TJnotJT]

I did some tests with homebrew and it seems like the issue above affects all the renders (SW, VK, OGL, DX11) and both for triangles and sprites. Assuming this is a problem with bilinear filtering sampling outside the range of texture, is this already a well-known issue? I wonder if it is a limitation of PS2 that is being emulated accurately or a problem with the emulation. If anyone is able to test on a PS2, here are the ELFs. Also the GS dumps and a video showing the glitch. Hope this is useful.

https://drive.google.com/file/d/1FjAbwvuCv462o1YSR7SB01AlJIjCIJWy/view?usp=sharing
https://drive.google.com/file/d/1N_EIrJrmrYUlrKGU6uW1wIlkdS00nbaJ/view?usp=sharing
https://drive.google.com/file/d/1hr3I3VWaOykZTY-mfZpddYug-H0VNiCD/view?usp=sharing
https://drive.google.com/file/d/1j6GjMu1WeX3QPQRMUEVbJakl4Lxng9iU/view?usp=sharing
https://drive.google.com/file/d/1lZ8f7sVI4G8iRRJvLgF9ahdbMRCGsq-f/view?usp=sharing

EDIT: These file can be ignored. I made a mistake in my homebrew code that caused the screen tearing, and was not related to problem in this game.

[TJnotJT]

I submitted as PR #12127 that may address the issue in this game. Though I haven't tested it very well so potentially it could break other things. Much appreciated if anyone is able to test it.