Skip to content

Adds metric for disconnects during upload - #1117

Open
aagbsn wants to merge 8 commits into
masterfrom
fix_1116_uncaught_exc
Open

aagbsn wants to merge 8 commits into
masterfrom
fix_1116_uncaught_exc

Conversation

@aagbsn

@aagbsn aagbsn commented Mar 12, 2026 •

Copy link
Copy Markdown
Contributor

catch exception raised and keep track of this metric #1116

hellais

This comment was marked as outdated.

except ClientDisconnect:
log.info(f"Client disconnected mid-upload")
Metrics.CLIENT_DISCONNECT.inc()
return empty_measurement

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should verify how clients behave when we return empty_measurement instead of raising an exception, since this might affect changes in the client behavior.

I would actually suggest we preserve the original behavior by re-rasing the exception

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we re-raise the exception it pollutes the logs with a Traceback; but if the client has disconnected - how would it receive anything?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think for the ClientDisconnect you are right. In the case of the catch-all exception, we ought to though return something that's an error to the client, either a 5xx error (by just re-raising the exception) or a specific 5xx/4xx status code.

return empty_measurement
except Exception as e:
log.error(f"Uncaught exception {e}")
return empty_measurement

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should return something that indicates that this was not a successful submission; verify what repsonse should be sent. IE, should re-raise as a wrapped HTTPException?

aagbsn added 5 commits March 24, 2026 16:11
on ClientDisconnect this won't be returned, but will suppress a noisy
traceback in logs.
# Conflicts:
#	ooniapi/services/ooniprobe/src/ooniprobe/routers/reports.py
receive_measurement catches ClientDisconnect without importing it, so
any exception while reading the body raised a NameError there instead
of reaching either handler.
client_upload_disconnect now counts by the probe_cc in the report_id.
Nothing has checked that cc when the body is read, so anything that
isn't a known country code is counted as XX. Tests cover a disconnect,
for each kind of cc, and any other error reading the body.
@aagbsn
aagbsn requested a review from hellais October 8, 2026 09:20
@aagbsn

aagbsn commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

added a label with probe_cc so we can see which countries are disconnecting during upload

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants