Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add additional requested bots #40

Open
wants to merge 4 commits into
base: eoin/refactor-pattern-instances
Choose a base branch
from
+100 −6
Open

feat: add additional requested bots #40

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
47b4535
chore: add imessage preview
e-moran Mar 12, 2025
96206da
chore: add facebot to imessage preview
e-moran Mar 12, 2025
31d5a2d
chore: add applebot to apple category
e-moran Mar 12, 2025
f0364c0
chore: add stripe and postman
e-moran Mar 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 100 additions & 6 deletions well-known-bots.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2803,7 +2803,10 @@
"accepted": [
"facebookexternalhit"
],
"forbidden": []
"forbidden": [
"Twitterbot",
"Facebot"
]
},
"addition_date": "2012/05/07",
"verification": [],
Expand All @@ -2813,7 +2816,9 @@
"facebookexternalhit/1.1",
"facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
],
"rejected": []
"rejected": [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0"
]
},
"url": "https://developers.facebook.com/docs/sharing/webmasters/crawler/",
"aliases": [
Expand Down Expand Up @@ -3760,7 +3765,10 @@
"accepted": [
"Twitterbot"
],
"forbidden": []
"forbidden": [
"facebookexternalhit",
"Facebot"
]
},
"addition_date": "2014/09/12",
"url": "https://dev.twitter.com/cards/getting-started",
Expand All @@ -3770,7 +3778,9 @@
"Twitterbot/0.1",
"Twitterbot/1.0"
],
"rejected": []
"rejected": [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0"
]
},
"aliases": [
"TwitterBot"
Expand Down Expand Up @@ -3897,7 +3907,10 @@
"accepted": [
"Face(book){0,1}[Bb]ot"
],
"forbidden": []
"forbidden": [
"facebookexternalhit",
"Twitterbot"
]
},
"url": "https://developers.facebook.com/docs/sharing/bot",
"addition_date": "2014/12/30",
Expand All @@ -3907,7 +3920,9 @@
"Facebot/1.0",
"Mozilla/5.0 (compatible; FacebookBot/1.0; +https://developers.facebook.com/docs/sharing/webmasters/facebookbot/)"
],
"rejected": []
"rejected": [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0"
]
},
"aliases": [
"FacebookBot"
Expand Down Expand Up @@ -4167,6 +4182,7 @@
{
"id": "apple-crawler",
"categories": [
"apple",
"search-engine"
],
"pattern": {
Expand Down Expand Up @@ -13355,5 +13371,83 @@
"rejected": []
},
"url": "https://docs.hydrozen.io/overview/misc/user-agent-and-ip-list"
},
{
"id": "imessage-preview",
"categories": [
"apple",
"preview",
"social"
],
"pattern": {
"accepted": [
"facebookexternalhit",
"Twitterbot",
"Facebot"
],
"forbidden": []
},
"addition_date": "2025/03/12",
"verification": [],
"instances": {
"accepted": [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0"
],
"rejected": []
},
"url": "https://docs.hydrozen.io/overview/misc/user-agent-and-ip-list"
},
{
"id": "stripe-webhook",
"categories": [
"webhook"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing to consider here is that we don't currently have a webhook category. This doesn't neatly fit in to any of the other categories though, so what do you think of introducing one?

],
"pattern": {
"accepted": [
"Stripe\\/"
],
"forbidden": []
},
"addition_date": "2025/03/13",
"verification": [
{
"type": "cidr",
"sources": [
{
"type": "http-json",
"url": "https://stripe.com/files/ips/ips_webhooks.json",
"selector": "$.WEBHOOKS[*]"
}
]
}
],
"instances": {
"accepted": [
"Stripe/1.0 (+https://stripe.com/docs/webhooks)"
],
"rejected": []
},
"url": "https://github.com/fnando/browser/issues/258"
},
{
"id": "postman",
"categories": [
"tool"
],
"pattern": {
"accepted": [
"PostmanRuntime\\/"
],
"forbidden": []
},
"addition_date": "2025/03/13",
"verification": [],
"instances": {
"accepted": [
"PostmanRuntime/7.28.4"
],
"rejected": []
},
"url": "https://user-agents.net/string/postmanruntime-7-28-4"
}
]