Rough idea of RAM and max_execution_time for 1600 meeting feed? Or general import throughput?

[patmaddox]

Hi there, we are currently having trouble importing the https://www.seattleaa.org/ feed using the WP plugin. Our host currently assigns us 768 MB of RAM, and has max_execution_time of 120s which is the limit with our plan. We are typically seeing the import fail at 2 minutes with the host's error page, although an interesting thing is that I've seen 500, 502, and 504 errors. We don't ever see the progress bar.

We could just bump up to a larger plan that permits longer max_execution_time... but before I do that, I am hoping to get a sense of how long it should take. 1600 in 120s is ~ 13 meetings per second. Not sure what the importer's expected throughput might be. And I don't want to just run into the same problem in a few months.

I've done some testing and found 4-6 meetings/sec for feeds exported from various Intergroup wordpress sites. Interestingly, google sheets API feeds appear to be processed at around 11 meetings/sec. It's possible this is a total red herring, and there's some other check going on that makes that particular feed faster... but I have imported each of these feeds multiple times and am getting pretty consistent results on the times.

Anyway assuming 4 meetings/sec, that puts our required max_execution_time at 400s. I'll try to get some more info from the guy who has been running the importer, to see if the number of meetings made a big jump recently. But in the mean time, I'm hoping to get some info on what to expect, or suggestions on configurations for a feed with 1600 meetings in it.

Also, it looks like ~250 of the meetings have "attendance_option": "inactive". I wonder if it would be possible to skip importing them to save time?

[patmaddox]

Also, it looks like ~250 of the meetings have "attendance_option": "inactive". I wonder if it would be possible to skip importing them to save time?

Turns out yes. I added this to functions.php (based on this comment):

function tsml_import_filter($meeting) {
	return $meeting['attendance_option'] != 'inactive';
}

and now we're able to complete the import.

It's a bit of a stop-gap, but we get at least one more day! :)

[anchovie91471]

Hey @patmaddox, have you checked your error log for any errors/warnings?

[patmaddox]

So far I've only been able to find the regular access logs, which has lines like

"POST /wp-admin/edit.php?post_type=tsml_meeting&page=import HTTP/1.1" 500

but no other info.

Do you know where I might find additional logs, or are there particular settings to enable to get additional logs from the importer?

[anchovie91471]

The php error logs will contain more info. Where those are is dependent on how your host has things setup. You can contact them to find out how to view them.