[5.x]: ChatGPT can't access internal pages

[tyssen]

After upgrading to Craft 5 accessing internal pages of the site with ChatGPT no longer works, i.e. asking "Please analyse … " causes ChatGPT to report a 500 internal server error. ChatGPT can still access the site's home page though. And other models like Gemini and Claude can still access pages.

The client has reported that before the Craft 5 upgrade ChatGPT was able to access all pages fine.

Server logs show the following for ChatGPT requests:

HTTP_USER_AGENT: ChatGPT-Browser/1.0
REQUEST_METHOD: HEAD
web.ERROR: yii\web\BadRequestHttpException: Unable to verify your data submission
Trace:
#0 /vendor/craftcms/cms/src/web/Controller.php(221): yii\web\Controller->beforeAction()
#1 ...

Terminal curl tests from my local machine succeed though, e.g.

curl -I https://site-domain.com.au/internal-page -A "ChatGPT-Browser/1.0"

HTTP/2 200 
referrer-policy: no-referrer-when-downgrade
strict-transport-security: max-age=15552000
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
cache-control: no-store
x-robots-tag: all
link: <https://site-domain.com.au/internal-page>; rel='canonical'
x-powered-by: SEOmatic
content-type: text/html; charset=UTF-8
date: Tue, 19 Aug 2025 04:27:33 GMT
server: LiteSpeed
alt-svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46"

Has something changed between Craft 4 and 5 (or one of the installed plugins) which would cause this?

Craft CMS version

5.8.12

PHP version

8.2.29

Operating system and version

No response

Database type and version

No response

Image driver and version

No response

Installed plugins and versions

ckeditor
cp-nav
craft-blitz
craft-freeform
craft-imageoptimize
craft-position-fieldtype
craft-retour
craft-seomatic
craft-sherlock
craft-sprig
environment-label
feed-me
field-manager
navigation
oembed
phpdotenv

[linear]

CMS-1550 [5.x]: ChatGPT can't access internal pages

[brandonkelly]

The one line of the stack trace you posted doesn’t make sense if you’re actually running Craft 5.8.12. Line 221 of src/web/Controller.php in Craft 5.8.12 is:

cms/src/web/Controller.php

Line 221 in 43bd419

if (!$isLive) {

Are you modifying Craft’s files in any way?

[tyssen]

Sorry, logging output's got a bit mixed up there somehow.

2025-08-19 14:04:00 [web.ERROR] [yii\web\HttpException:400] Unable to verify your data submission. {"trace":["#0 /vendor/craftcms/cms/src/web/Controller.php(176): yii\\web\\Controller->beforeAction()","#1 /vendor/yiisoft/yii2/base/Controller.php(176): craft\\web\\Controller->beforeAction()","#2 /vendor/yiisoft/yii2/base/Module.php(552): yii\\base\\Controller->runAction()","#3 /vendor/craftcms/cms/src/web/Application.php(360): yii\\base\\Module->runAction()","#4 /vendor/craftcms/cms/src/web/Application.php(659): craft\\web\\Application->runAction()","#5 /vendor/craftcms/cms/src/web/Application.php(322): craft\\web\\Application->_processActionRequest()","#6 /vendor/yiisoft/yii2/base/Application.php(384): craft\\web\\Application->handleRequest()","#7 /public_html/index.php(26): yii\\base\\Application->run()","#8 {main}"],"memory":4204544,"exception":"[object] (yii\\web\\BadRequestHttpException(code: 0): Unable to verify your data submission. at /vendor/yiisoft/yii2/web/Controller.php:221)"}

[brandonkelly]

Thanks, that makes more sense.

Nothing changed in Craft 5 related to CSRF validation.

Check config/app.php (and config/app.web.php if it exists) for validateCsrfHeaderOnly or csrfTokenSafeMethods – are you setting either of those anywhere?

[tyssen]

Neither of those is set. enableCsrfProtection is set to true in config/general.php though.

[brandonkelly]

Your original error log snippet showed that this was happening on a HEAD request. Was that error log generated from AI, or copied directly from Craft’s logs? The second error report you posted looks completely different, and doesn’t make any mention of it being a HEAD request.

It seems like, most likely, ChatGPT is making POST requests to Craft for some reason, which is where CSRF validation gets involved, and that error would be expected.