Anymail Mailgun: How to block a suspicious recipient?

[Eraldo]

I double and triple checked my settings:

MAILGUN_API_KEY
MAILGUN_API_URL
MAILGUN_DOMAIN
ANYMAIL_WEBHOOK_SECRET
DJANGO_SERVER_EMAIL
MAILGUN_WEBHOOK_SIGNING_KEY

The production server perfectly delivers emails as expected.

However I regularly (every couple of hours) get an email informing me of the following error:

Missing or invalid basic auth in Anymail Mailgun webhook

Upon further inspection, I learned that in the body of the request there is always the same recipient, which is completely unknown to me and seems "spammy".

First I thought I had misconfigured something, however now I rather believe that these requests are actually missing the required auth and are rightfully rejected.
This challenge on my side is now that I get an email every hour or so informing me of that error and all of these emails also count toward my quote.

I will happily supply more information.
I wonder what my options are to resolve this issue?
I already tried blocking the recipient on the mailgun web interface, creating a drop rule, however that did nothing. :|
Any ideas what I can do or try next? :D

Big thanks to the community. I love how we support each other! 🙌

[medmunds]

Is your production web server correctly receiving webhook tracking events for other messages you send? (If it had been working fine before and this just started recently, what else changed around that time?)

If all webhooks are failing, the problem is with your configuration: your Django Anymail settings don't match what's in Mailgun's webhook dashboard:

• If you're setting Anymail's WEBHOOK_SECRET then you must include that random:random secret in every webhook URL in Mailgun's dashboard.
• Maybe something in your server config strips basic auth from the request before forwarding it to Django? (See the note about WSGIPassAuthorization.)

If tracking webhooks are mostly working, and you're just getting a few that trigger the auth error, then you'll want to figure out what is posting those mis-authed webhooks:

• Did you maybe set a webhook for your Mailgun sandbox domain, with different or missing webhook secret in the url? (There's an easy-to-miss Domain menu at the top of Mailgun's webhooks dashboard. Is the "spammy" recipient email something @sandbox___.mailgun.org?)
• Do you have some test code that posts to your production /anymail/mailgun/tracking/ webhook url?
• Did you use Mailgun's "Test webhook" feature without including the webhook secret in the url? (Is the "spammy" recipient "[email protected]" or similar?)
• What IP address is posting to the webhook that's failing auth? Is it one of your machines? Is it Mailgun? (They run on Google Cloud.) Or it's possible someone is trying to post maliciously to your /anymail/mailgun/tracking/ webhook url. (But I'd look at everything else first.)

I'd recommend figuring out where the mis-authed webhooks are coming from and solving the problem at the source. But if you just want to stop the email notifications, you could adjust your Django logging configuration to remove AdminEmailHandler (disables all error emails) or to filter out AnymailWebhookValidationFailure errors.

[Eraldo]

Hello Mike and thank you for getting back to me with such a wonderfully verbose response!

Is your production web server correctly receiving webhook tracking events for other messages you send? (If it had been working fine before and this just started recently, what else changed around that time?)

The error emails started on Apr 6, 2023, 7:11 PM UTC.

If all webhooks are failing, the problem is with your configuration: your Django Anymail settings don't match what's in Mailgun's webhook dashboard:

I am not noticing any issues other than these error emails - all with the same recipient. (more on this later)

• If you're setting Anymail's WEBHOOK_SECRET then you must include that random:random secret in every webhook URL in Mailgun's dashboard.

I tried removing the WEBHOOK_SECRET as it is supposed to fallback on signing with similar security as far as I can tell from the docs.
Removing the WEBHOOK_SECRET did not change anything about the error emails.

• Maybe something in your server config strips basic auth from the request before forwarding it to Django? (See the note about WSGIPassAuthorization.)

I checked my caddy (production webserver) which is set to transparent by default and should not drop or mask any such headers.
But to be sure I can still manually test that. If that were an issue, then if I'm correct I would expect that other things would also not work as expected anymore - but they do work fine.

If tracking webhooks are mostly working, and you're just getting a few that trigger the auth error, then you'll want to figure out what is posting those mis-authed webhooks:

• Did you maybe set a webhook for your Mailgun sandbox domain, with different or missing webhook secret in the url? (There's an easy-to-miss Domain menu at the top of Mailgun's webhooks dashboard. Is the "spammy" recipient email something @sandbox___.mailgun.org?)

I did not use the sandbox domain. Thanks for thinking about and checking that one as well. 👏

• Do you have some test code that posts to your production /anymail/mailgun/tracking/ webhook url?

No, I don't.

• Did you use Mailgun's "Test webhook" feature without including the webhook secret in the url? (Is the "spammy" recipient "[email protected]" or similar?)

No, the recipient is: "info@phuk****clinic.com" in every such email and it triggers daily. I can very confidently say that this email is unrelated to my project.

• What IP address is posting to the webhook that's failing auth? Is it one of your machines? Is it Mailgun? (They run on Google Cloud.) Or it's possible someone is trying to post maliciously to your /anymail/mailgun/tracking/ webhook url. (But I'd look at everything else first.)

I sadly don't see or rather don't know where/how I would be able to see the IP address of the poster in the sentry error stack trace. 🤷‍♂️

I'd recommend figuring out where the mis-authed webhooks are coming from and solving the problem at the source. But if you just want to stop the email notifications, you could adjust your Django logging configuration to remove AdminEmailHandler (disables all error emails) or to filter out AnymailWebhookValidationFailure errors.

Thanks for the pro-active suggestions. It even feels good to have an option like that on my mind.

I will try to further find check through your suggestions. The next step I will undergo is manually testing the tracking endpoint and observing if I get the same (or similar) error message. I also find it quite confusing as I did setup everything on mailgun so that I can add tracking, however I am not sending a single email with tracking info in it, so it doubly does not make sense to me that I get these emails. I'll do the manual testing and report back.

Thank you so much Mike for the steps I can go through once more. 🙏

[Eraldo]

This IP is located in Belgium and belongs to Google LLC

HTTP_USER_AGENT = 'Go-http-client/2.0'
HTTP_X_FORWARDED_FOR = '35.210.207.249'

I don't have any webhooks setup on mailgun. Neither new nor legacy ones.
I also disabled all tracking options:

Click tracking    Off
Open tracking    Off
Unsubscribes    Off

[medmunds]

That IP is probably Mailgun (which uses GCP), so this is probably a legitimate webhook post from Mailgun to your server.

Is it possible you have Mailgun inbound routing set up? (That would generate the same error message, if the webhook secret was missing from the "receiving route forward url". Mailgun inbound is configured on a different dashboard page from their delivery tracking webhooks. Inbound spam with a spoofed recipient seems consistent with what you've described.)

If you remove the ANYMAIL_WEBHOOK_SECRET setting (and/or ANYMAIL = {..., "WEBHOOK_SECRET": ...} setting), it should not be possible to get "Missing or invalid basic auth" errors. Anymail doesn't even look at the HTTP_AUTHORIZATION header unless that setting is present. (You might still get other validation errors, just not the one about basic auth.)

If you're able capture the webhook payload, it might help to take a look at that. (You can redact the signature and any private domains before posting here.)

[medmunds]

Also, since you're using Sentry…

You can figure out whether the auth is "missing" or "invalid": in the Sentry stack trace, look for AnymailBasicAuthMixin.validate_request in anymail/webhooks/base.py (should be at the top). Check the value of the request_auth local variable: None means it was missing from the request url; a string value must match your Anymail WEBHOOK_SECRET setting (in self.allowed_auth) to be valid.

And if you've set up Sentry to notify you about Django errors, there's really no need to also have Django's default AdminEmailHandler in your production logging config.

[Eraldo]

@medmunds You were absolutely right!
I had inbound routes defined and that was the source of the missing auth warnings I was getting. ✅

Sorry for taking that much time for reporting back.
I simply removed the inbound routes and voila, the warnings are gone.
I waited a couple of days as I was overly skeptical it could be that simple.
(Well it seems the complexity lies in understanding what the problem is, as once that is understood the solution can come quite easily.)

Thank you so much for pointing me in that direction. I don't now how to better express that than to repeat myself and appreciate how methodical and thorough and kind you have been taking the time to helping me save hours of further clueless debugging in the wrong direction. I learned from the interaction with you not just when it comes to this bug, but regarding your approach as well. 🙏🏽

I wish you a wonderful start of July! ☀️

[medmunds]

Great! Glad you figured it out, and happy I could help.