Add setting to disable the broken website alert

[surli]

Describe the issue

We're getting alerts on pretty much all components in https://l10n.xwiki.org related to Broken project website, with the following error:

(403 Client Error: Forbidden for url: https://www.xwiki.org/xwiki/bin/view/Main/WebHome)

I think the issue is that our sysadmins configured www.xwiki.org to prevent some bots to access it and so it's blocking also Weblate checks. Now for us this check is completely useless as we control the components we create, so I set WEBSITE_REQUIRED=False in settings and I manually triggered the tasks repository_alerts and component_alerts, but I'm still having the alert displayed. And I don't find a way to globally dismiss it.

I already tried

• I've read and searched the documentation.
• I've searched for similar filed issues in this repository.

Steps to reproduce the behavior

1. Create a project and indicate the URL https://www.xwiki.org
2. See that an alert is created for broken project website
3. Check that the website is actually available

Expected behavior

Website availability should be checked with some specific documented headers so that it's possible to filter them out to not consider them as bots.
Also it should be possible to entirely ignore those alerts with some configuration, and if possible to be able to bulk dismiss alerts.

Screenshots

Exception traceback

No response

How do you run Weblate?

PyPI module

Weblate versions

Weblate: 5.9.2
Django: 5.1.4
siphashc: 2.3
translate-toolkit: 3.14.5
lxml: 5.3.0
pillow: 11.1.0
nh3: 0.2.15
python-dateutil: 2.8.2
social-auth-core: 4.5.1
social-auth-app-django: 5.4.2
django-crispy-forms: 2.3
oauthlib: 3.2.2
django-compressor: 4.4
djangorestframework: 3.15.2
django-filter: 24.3
django-appconf: 1.0.6
user-agents: 2.2.0
filelock: 3.16.1
RapidFuzz: 3.11.0
openpyxl: 3.1.2
celery: 5.4.0
django-celery-beat: 2.7.0
kombu: 5.3.4
translation-finder: 2.19
weblate-language-data: 2024.16
html2text: 2024.2.26
pycairo: 1.25.1
PyGObject: 3.46.0
diff-match-patch: 20241021
requests: 2.32.3
django-redis: 5.4.0
hiredis: 2.2.3
sentry-sdk: 2.19.2
Cython: 3.0.11
mistletoe: 1.4.0
GitPython: 3.1.40
borgbackup: 1.2.7
pyparsing: 3.1.1
ahocorasick_rs: 0.22.1
python-redis-lock: 4.0.0
charset-normalizer: 3.3.2
cyrtranslit: 1.1.1
drf-spectacular: 0.28.0
Python: 3.11.2
Git: 2.39.5
psycopg: 3.1.17
psycopg-binary: 3.1.17
phply: 1.2.6
ruamel.yaml: 0.17.40
tesserocr: 2.6.2
boto3: 1.28.85
aeidon: 1.15
iniparse: 0.5
google-cloud-translate: 3.19.0
openai: 1.59.7
Mercurial: 6.9
git-review: 2.4.0
PostgreSQL server: 15.10
Database backends: django.db.backends.postgresql
PostgreSQL implementation: psycopg3 (binary)
Cache backends: default:RedisCache, avatar:FileBasedCache
Email setup: django.core.mail.backends.smtp.EmailBackend: localhost
OS encoding: filesystem=utf-8, default=utf-8
Celery: redis://localhost:6379, redis://localhost:6379, regular
Platform: Linux 6.1.0-17-amd64 (x86_64)

Weblate deploy checks

No response

Additional context

No response

[nijel]

Weblate always uses Weblate/version (for example Weblate/5.9.2) as a User-Agent, you can use it to allow it.

WEBSITE_REQUIRED=False makes the website field optional so that you can leave it empty. It has no effect on URL validation.

[surli]

Weblate always uses Weblate/version (for example Weblate/5.9.2) as a User-Agent, you can use it to allow it.

Thanks I'll inform our sysadmins about that.

WEBSITE_REQUIRED=False makes the website field optional so that you can leave it empty. It has no effect on URL validation.

Ok I wasn't sure. Would it be possible to have some configuration options for being able to disable such check?

[github-actions]

This issue seems to be a good fit for newbie contributors. You are welcome to contribute to Weblate! Don't hesitate to ask any questions you would have while implementing this.

You can learn about how to get started in our contributors documentation.

[nijel]

Patches are welcome to implement this feature.

Weblate will always be issuing requests to third-party servers because of its Git integration, and these URLs can be manipulated by the same users as project URLs. So this really doesn't increase the exposure.

I doubt that checking a server once in a day will have any impact on your IP reputation.