fix: enforce Redis-backed upload queue in production

[Namraa310806]

Summary

This PR improves the reliability of the upload job queue by preventing unsafe in-memory queue usage in production environments.

Previously, UploadJobQueue silently fell back to an in-memory queue whenever Redis was unavailable or REDIS_URL was not configured. While suitable for local development, this behavior could lead to lost upload jobs in multi-worker deployments and after application restarts.

This PR introduces a fail-fast policy for production while preserving the existing development workflow.

Changes

• Added centralized environment and Redis configuration helpers.
• Enforced fail-fast initialization in production and staging when Redis is unavailable or REDIS_URL is missing.
• Restricted the in-memory queue fallback to development, testing, and local environments.
• Improved startup logging with environment-aware diagnostics.
• Added clear documentation explaining queue behavior across environments.
• Preserved the existing queue API (enqueue, reserve, retry scheduling, dead-letter handling, and UploadJob schema).
• Added comprehensive reliability tests covering Redis availability, environment detection, backend selection, and queue behavior.

Why

The previous implementation could silently enqueue upload jobs into a process-local Python list when Redis was unavailable.

This could result in:

• Upload jobs being isolated to a single worker process.
• Jobs being lost after process or container restarts.
• Unreliable background document processing in production deployments.

The new behavior guarantees that production deployments require a durable queue backend while maintaining a frictionless development experience.

Testing

Executed the upload queue test suites:

python -m pytest \
backend/tests/test_upload_queue_reliability.py \
backend/tests/test_upload_background_worker.py \
backend/tests/test_upload_failure_scenarios.py \
backend/tests/test_upload_integration.py \
backend/tests/test_upload_regression.py \
backend/tests/test_upload_task_storage.py

Result:

• ✅ 126 tests passed

Impact

• ✅ Improves production reliability.
• ✅ Prevents silent upload job loss.
• ✅ Preserves backward compatibility.
• ✅ No public API changes.
• ✅ Development and testing workflows continue using the in-memory fallback.

Fixes: #587

[vercel]

@Namraa310806 is attempting to deploy a commit to the WizardKing's projects Team on Vercel.

A member of the Team first needs to authorize it.

[AnuranjanJain]

@Namraa310806 Can you look for the faling test cases in this PR ?