statamic:static:warm 504s

[stephenmeehanuk]

Hi

Anyone got any tips for static caching?

On my staging website, I'm experiencing 504 errors when trying to run php artisan statamic:static:warm. It only stops 504ing when I run - and allow to finish - php please assets:generate-presets --queue

Weird thing is though, all the image transformations have already been done before running statamic:static:warm

The problem is this 'solution' takes nearly two hours to run.

Is this normal?

Image transformations are located in /public/img/containers/assets. The assets directory is 2.75GB.

How come running assets:generate-presets improves the speed of statamic:static:warm, it doesn't 504 and it's super fast.

statamic:static:warm also runs faster when doing the same locally.

I'm just concerned this might be an issue when the website finally goes live. If it takes so long to rebuild the cache, with the possibility of 504ing the website in the process.

Statamic 3.3.16 Pro
Laravel 9.12.2
PHP 8.0.14
simonhamp/statamic-heroicons 1.0.1
statamic/collaboration 0.4.0

[joshuablum]

Hey @stephenmeehanuk,

Can you provide some more information?

• Which static caching strategy are you using? Half or full measure?
• How are your assets set up? Are the assets between multiple containers or all in one?
• How many assets are we talking about?
• Anything about this in the logs? laravel.log, php-fpm.log or from nginx etc.
• You mention it only happening on staging. Locally it works fine without running assets:generate-presets first, right?

Looks like a resource limit or configuration problem of your server to me. What's the max_execution_time on your server set to? If it's too low this can result in the 504s. You can run php --info on your server to find out. Try to increase the amount to something like 300 and see if this helps :)

[stephenmeehanuk]

Hi, thanks for the reply.

Which static caching strategy are you using? Half or full measure?

Half measure

How are your assets set up? Are the assets between multiple containers or all in one?

A single container

How many assets are we talking about?

/public/assets 374Mb
public/img/containers/assets 2.83 GB

Anything about this in the logs? laravel.log, php-fpm.log or from nginx etc.

laravel.log is clean/empty.
php-fpm.log was showing this, but I've fixed it, increased it to 10

[27-Jun-2022 09:43:00] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

You mention it only happening on staging. Locally it works fine without running assets:generate-presets first, right?

• On staging it'll 504 unless I run assets:generate-presets first.
• Locally, it'll work without running assets:generate-presets first, it doesn't 504.

I've since tried to run statamic:static:warm after changing the pm.max_children to 10, but I'm still seeing the same error.

Compiling URLs...
[✔] Entries
[✔] Taxonomies
[✔] Taxonomy terms
[✔] Custom routes

Visiting 356 URLs...
[✓] /about
[✓] /about/corporate-values
[✓] /about/business-system
[✓] /about/investor-relations
[✓] /about/leadership
[✓] /about/sustainability
[✓] /about/the-business-system
[✓] /about/vendor-relations
[✓] /about/what-we-do
[✗] /case-studies/birmingham-city-council-amey-security-cctv-and-communications → 504 Gateway Time-out
[✗] /case-studies/bridgestrike-remote-contract → 504 Gateway Time-out
[✗] /case-studies/calendar-club-chooses-conveyor-solution-from-lb-foster → 504 Gateway Time-out

NGINX error log

2022/06/27 19:26:42 [error] 23477#23477: *114477 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 143.XXX.168.XXX, server: preview.website.com, request: "GET /case-studies/automated-engine-transfer HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.1-fpm.sock", host: "preview.website.com"

/var/log/php8.1-fpm.log

[27-Jun-2022 19:44:00] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it

I think you're right,

Looks like a resource limit or configuration problem of your server to me

running statamic:static:warm triggers this error in php8.1-fpm. I'll keep increasing the pm.max_children setting and see how I get on.

[jasonvarga]

How come running assets:generate-presets improves the speed of statamic:static:warm,

More than likely the bottlenecks are from images being generated when the static:warm command visits the pages.
If the images were already generated by the generate-presets command, then it makes sense that visiting the pages would be fast.

Weird thing is though, all the image transformations have already been done before running statamic:static:warm
The problem is this 'solution' takes nearly two hours to run.

It feels a bit strange that it takes a long time, every time. Can you provide your deploy script? Do you ever clear the glide cache?

[stephenmeehanuk]

This is my deploy script

php artisan cache:clear
php artisan config:cache
php artisan route:cache
php artisan statamic:stache:warm
php artisan statamic:search:update --all
php artisan statamic:static:clear
php artisan horizon:terminate

[stephenmeehanuk]

I've been asking in the Ploi Discord group for help too. That's a work in progress. Ploi has a useful 'insights' feature

When running the warm command, I see this insight message

Your server load seems excessive for what your CPU can handle. We don't have a clear solution for this, but we recommend to check htop & top in SSH to discover any high demanding services.

Screenshot of htop and top when the server isn't running the warm command

Same reports when warm command causes a 504

[stephenmeehanuk]

Would warming 356 URLs, with assets pre generated be considered excessive?

Is there anything else I can check to figure out why this is causing a 504?

[jasonvarga]

Are your glide tags actually using the presets? Like {{ glide:whatever preset="something" }}?

Generating presets wouldn't be helpful unless you actually use them. If your glide tags are doing separate transforms, then they will be generated as your static:warm command runs, which will take CPU.

[stephenmeehanuk]

Yup. Glide tags are using my presets. I've just double checked.

But could the way I'm using glide be causing an issue? I'm using the new runtime parser.

I'm using a few partials to manage different image formats (1:1, 4:3, 16:9).

Depending on the what format I need, I can use a specific partial and pass in the glide handle as a partial variable, like this:

{{ partial:_components/glide/ratio/16-9 glide_handle="thumbnail" }}

Inside the partial I use the glide_handle like this:

 {{ glide:{glide_handle ?? "url"} fit="crop_focal" preset="w1280_h720_wp" }} 1280w

There's more to the partial than this, but this bit shows how the glide variable is being used.

Can you think of anything else that might be causing the 504s?

[stephenmeehanuk]

I'm still having issues with this.

I don't need the warm command to be fast, I can set a cron job to do it, but running it must not 504 the website.

Is there a command to cache certain pages?

It'd be helpful to try and cache certain URLs, I'm wondering if the 504s are happening on certain templates. Running a full cache is taking a long time.

[stephenmeehanuk]

Been looking into this some more...

Working and testing locally. I thought if I picked the template apart I might be able to identify the source of the problem. The problem being static:warm 504s on certain pages.

The end point of removing partial after partial was a blank view template. I went a step further and deleted the contents of layout as well.

Still working locally, I ran cache:clear.

Using Google Dev Tools. The first request/response is 2.07s. A second load (refresh), request/response goes down to 138ms. But the page is blank. I'm not using any caching (half or full) locally.

I thought it perhaps it had something to do with using a collection as a structure? So I tried the same thing on the home page (empty view and layout, then cache:clear). Seeing a similar first request/response time of 2.12s.

These tests, show my local installation of Statamic is taking just over 2 seconds to load a blank view and layout?

A fresh local installation of Statamic. Same test, blank view and layout, clear app cache, first request/response time 107ms.

Staging

It's worth mentioning, when I visit a page marked as 504 Gateway Time-out on my staging site it loads. I clear the application cache first. I'm seeing a request/response time of 5.95s. Which is slow. I'm using half measure caching on the staging site. A page refresh dramitically improves this with a request/response time of 247ms! This is great.

But the first visit is how the warm script sees the site, and this is causing problems.

If my installation of Statamic is taking just over 2 seconds to load a blank view and layout locally, perhaps this is being replicated on the staging server?

Is there anything else I can try?
Is the test of running cache:clear a valid one?

I'm kinda running out of ideas, I'd really appreciate some suggestions.

Statamic 3.3.17 Pro
Laravel 9.19.0
PHP 8.0.8
simonhamp/statamic-heroicons 1.0.1
statamic/collaboration 0.4.0

[stephenmeehanuk]

Ok, think I've got to the bottom of this!

My site is on a staging server, it's not gone live yet.

The client is making regular content updates, while I work on the templates. It's a really good process.

My deploy script includes a stache:warm ...

php artisan cache:clear
php artisan config:cache
php artisan route:cache
php artisan statamic:stache:warm
php artisan statamic:search:update --all
php artisan statamic:static:clear
php artisan horizon:terminate

... but not a static:warm command, I didn't want to cache the staging site on every deploy.

During development, I'd occasionally ssh into the sever and run these two commands

php artisan cache:clear

Followed by

php artisan statamic:static:warm

I was missing the php artisan statamic:stache:warm, it appears this might be the reason so many pages where 504ing - is this likely?

To cache the staging website via ssh I now run these commands

• php artisan cache:clear
• php please stache:refresh
• php artisan statamic:static:warm

This significantly reduces the number of pages 504ing, from many to only a handful.

Running the statamic:static:warm a second time, without clearing the app cache or refreshing the stache successfully caches all the pages without any 504 messages.

[Assiente]

I also once had a similar problem, which was caused by cache not being connected on the intermediate site. I often forgot about it and for a long time could not understand what the problem was. Therefore, I recommend using suitable site monitoring: this is relevant for the intermediate stage and for the final version of the website.