Update json cache to eliminate race condition

[ubaskota]

Description of changes:
Use os.replace() for atomic file operations to eliminate race conditions during concurrent access to the file cache. This ensures that writes are either fully completed or not applied at all, preventing partial writes that could leave the JSON file in an invalid state when multiple processes access the cache simultaneously.

Tests:
I verified that all tests including the the newly added unit tests pass.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.70%. Comparing base (bbed2c0) to head (527e25d).
⚠️ Report is 375 commits behind head on develop.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3597      +/-   ##
===========================================
- Coverage    93.13%   92.70%   -0.44%     
===========================================
  Files           68       68              
  Lines        15336    15563     +227     
===========================================
+ Hits         14283    14427     +144     
- Misses        1053     1136      +83     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[SamRemis]

What would be the tradeoffs of making sure the cache key is always unique to avoid this race condition entirely?

[SamRemis]

So @ubaskota and I chatted about we are removing the nested os.open(full_key, os.O_WRONLY | os.O_CREAT, 0o600), but I want to record it here for posterity:

The inner os.open that was formerly on line 3582 was necessary because Python has less restrictive default permissions for a newly created file. Now that we are using tempfile.mkstemp, the default permissions are already owner read/write for owner only, so we no longer need the nested open call.

[SamRemis]

What specific errors are we trying to protect against by setting temp_fd and temp_path to None?

If we don't have a clear idea of the new failure modes we are accounting for by adding this logic, we should remove it. If we do and these None checks are necessary/valuable, we should document that here so it's easier to maintain going forward.

[ubaskota]

We can remove the lines setting temp_fd and temp_path to None along with the exception handler, and the code will work fine in normal operation. However, it may introduce issues in error scenarios as described below:

1. File descriptor leak: If an exception occurs between mkstemp() and os.fdopen(), the file descriptor remains open
2. Orphaned temp files: If errors occur during write/flush/fsync or if os.replace() fails, temporary files will remain on disk forever

These are edge cases that handle those rare scenarios. Since we haven't encountered these problems historically and they haven't caused issues in our environment, I'm okay with removing the cleanup code.