Add retry/backoff around block submission to avoid noise from transient fullnode failures

[luislhl]

Summary

Add retry/backoff around block submission in _push_block_job() so that transient fullnode failures don't immediately surface as Error when submitting block log errors (which generate OpsGenie alert noise).

Background

_push_block_job() in txstratum/manager.py logs an error on the first PushTxFailed:

try:
    await self.backend.push_tx_or_block(job.get_data())
except PushTxFailed:
    self.log.error("Error when submitting block", job=job)

PushTxFailed is raised by python-hathorlib's push_tx_or_block() on a non-2xx fullnode response. In practice these are usually transient fullnode conditions — rate-limit (429), short out-of-sync windows (503/504), or a now-stale block — i.e. the same fullnode-dependency chain behind the wider tx-mining block-template alert family.

This was triaged in on-call (on-call-incidents#260) and is currently silenced via a rate-limited log-manager whitelist (5/1h). The whitelist stops the noise but the underlying behavior (no retry) remains; a short fullnode blip still produces an error log on the first failure.

Related analysis: ops-tools/docs/on-call/reports/20260514-open-alerts-review/30-group-3-tx-mining.md. Related past incidents: on-call-incidents #1, #10, #42, #160, #203.

Proposed change

• Add a bounded retry with backoff around push_tx_or_block() for block submission (a few attempts, short exponential backoff), only logging error once all retries are exhausted.
• Consider distinguishing retryable statuses (429/502/503/504, connection/DNS errors) from genuinely non-retryable rejections, and log the HTTP status / backend host in the failure message to aid future triage.
• Same reasoning likely applies to the other emitters in this chain (update_block_template() / block-template fetch), so consider a shared retry policy.

Acceptance criteria

• Transient single-failure block submissions are retried and do not emit an Error when submitting block error log unless retries are exhausted.
• The error log, when it does fire, includes enough context (status code, backend) to triage.

Notes

This complements (does not replace) the log-manager whitelist. Once retries land, the whitelist rate limit can be revisited/removed.