You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I would expect the following user agents to be matched by the spiders catch all.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.2210.133 VirusTotalBot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; https://openai.com/bot
I'm not a regex expert, but I suspect that they aren't because with each one, the match (Bot, facebookexternalhit, and bot respectively) doesn't occur within the first 100 characters. Would it make sense to check the first 200 characters for the spiders catch all, similar to the general matcher for bots?
The text was updated successfully, but these errors were encountered:
I would expect the following user agents to be matched by the spiders catch all.
I'm not a regex expert, but I suspect that they aren't because with each one, the match (Bot, facebookexternalhit, and bot respectively) doesn't occur within the first 100 characters. Would it make sense to check the first 200 characters for the spiders catch all, similar to the general matcher for bots?
The text was updated successfully, but these errors were encountered: