Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Regression: Matomo ignores --concurrent-archivers=3 option on core:archive #21985

Closed
4 tasks done
atom-box opened this issue Mar 1, 2024 · 5 comments · Fixed by #22515
Closed
4 tasks done

[Bug] Regression: Matomo ignores --concurrent-archivers=3 option on core:archive #21985

atom-box opened this issue Mar 1, 2024 · 5 comments · Fixed by #22515
Labels
duplicate For issues that already existed in our issue tracker and were reported previously.

Comments

@atom-box
Copy link
Contributor

atom-box commented Mar 1, 2024 •
edited
Loading

What happened?

You can run 4 or more crontabs of the following command, simultaneously, in parallel, Matomo doesn't try to stop you:
core:archive --no-ansi --no-interaction --concurrent-archivers=3 --concurrent-requests-per-website=4 --max-archives-to-process=200 --disable-scheduled-tasks --force-idsites=11

What should happen?

When attempting to start too many archivers, here is the expected message:

INFO [20:39:22] 21625 Archiving will stop now because 14 archivers are already running and max 3 are supposed to run at once.
INFO [20:39:22] 21625 Reached maximum concurrent archivers allowed (3), aborting run.

How can this be reproduced?

For core:archive:

  • Try to start concurrently an excessive number of the below command from the CLI. (Matomo will disallow it) 🙂
  • Try to start concurrently an excessive number of the below command from the Cron. (Matomo will allow it) 🙁

Here is the syntax used by the person who reported this problem:
core:archive --no-ansi --no-interaction --concurrent-archivers=3 --concurrent-requests-per-website=4 --max-archives-to-process=200 --disable-scheduled-tasks --force-idsites=11

Notice the log output below. You can see that four archivers start up, each reporting that 0 of 3 are running.

(In the private Jira version of this ticket I have included more information. The user says that before upgrading to Matomo 5, this bug did not occur in Matomo 4)

Matomo version

5.0.1

Server operating system

Linux

Relevant log output

THIS IS LOG FROM RUNNING IN CRON.
NOTICE THAT WE CAN GET TOO MANY ARCHIVERS TO RUN

INFO      [20:26:01] 109953  ---------------------------
INFO      [20:26:01] 109953  INIT
INFO      [20:26:01] 109953  Running Matomo 5.0.1 as Super User
INFO      [20:26:01] 109953  ---------------------------
INFO      [20:26:01] 109953  NOTES
INFO      [20:26:01] 109953  - Async process archiving supported, using CliMulti.
INFO      [20:26:01] 109953  - Reports for today will be processed at most every 14400 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO      [20:26:01] 109953  - Reports for the current week will be processed at most every 28800 seconds. You can change this value in config/config.ini.php by editing 'time_before_week_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:01] 109953  - Reports for the current month will be processed at most every 43200 seconds. You can change this value in config/config.ini.php by editing 'time_before_month_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:01] 109953  - Reports for the current year will be processed at most every 86400 seconds. You can change this value in config/config.ini.php by editing 'time_before_year_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:01] 109953  - Reports for the current range will be processed at most every 2700 seconds. You can change this value in config/config.ini.php by editing 'time_before_range_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:01] 109953  - Maximum 200 archives will be processed (soft limit).
INFO      [20:26:01] 109953  - Archiving was last executed without error 3 min 36s ago.
INFO      [20:26:01] 109953  - Will process 1 websites (--force-idsites)
INFO      [20:26:01] 109953  - Will process specified sites: 11
INFO      [20:26:01] 109953  ---------------------------
INFO      [20:26:01] 109953  START
INFO      [20:26:01] 109953  Starting Matomo reports archiving...
INFO      [20:26:01] 109953  0 out of 3 archivers running currently
INFO      [20:26:01] 109953  Start processing archives for site 11.
INFO      [20:26:02] 109953    Will invalidate archived reports for today in site ID = 11's timezone (2024-03-01 00:00:00).
INFO      [20:26:02] 109953    Will invalidate archived reports for yesterday in site ID = 11's timezone (2024-02-29 00:00:00).
nohup: les entr�es sont ignor�es
INFO      [20:26:34] 110390  ---------------------------
INFO      [20:26:34] 110390  INIT
INFO      [20:26:34] 110390  Running Matomo 5.0.1 as Super User
INFO      [20:26:34] 110390  ---------------------------
INFO      [20:26:34] 110390  NOTES
INFO      [20:26:34] 110390  - Async process archiving supported, using CliMulti.
INFO      [20:26:34] 110390  - Reports for todaycore:archive --no-ansi --no-interaction --concurrent-archivers=3 --concurrent-requests-per-website=4 --max-archives-to- will be processed at most every 14400 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO      [20:26:34] 110390  - Reports for the current week will be processed at most every 28800 seconds. You can change this value in config/config.ini.php by editing 'time_before_week_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:34] 110390  - Reports for the current month will be processed at most every 43200 seconds. You can change this value in config/config.ini.php by editing 'time_before_month_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:34] 110390  - Reports for the current year will be processed at most every 86400 seconds. You can change this value in config/config.ini.php by editing 'time_before_year_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:34] 110390  - Reports for the current range will be processed at most every 2700 seconds. You can change this value in config/config.ini.php by editing 'time_before_range_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:34] 110390  - Maximum 200 archives will be processed (soft limit).
INFO      [20:26:34] 110390  - Archiving was last executed without error 4 min 9s ago.
INFO      [20:26:34] 110390  - Will process 1 websites (--force-idsites)
INFO      [20:26:34] 110390  - Will process specified sites: 11
INFO      [20:26:34] 110390  ---------------------------
INFO      [20:26:34] 110390  START
INFO      [20:26:34] 110390  Starting Matomo reports archiving...
INFO      [20:26:34] 110390  0 out of 3 archivers running currently
INFO      [20:26:34] 110390  Start processing archives for site 11.
INFO      [20:26:34] 110390    Will invalidate archived reports for today in site ID = 11's timezone (2024-03-01 00:00:00).
INFO      [20:26:35] 110390    Will invalidate archived reports for yesterday in site ID = 11's timezone (2024-02-29 00:00:00).
nohup: les entr�es sont ignor�es
INFO      [20:26:40] 110575  ---------------------------
INFO      [20:26:40] 110575  INIT
INFO      [20:26:40] 110575  Running Matomo 5.0.1 as Super User
INFO      [20:26:40] 110575  ---------------------------
INFO      [20:26:40] 110575  NOTES
INFO      [20:26:40] 110575  - Async process archiving supported, using CliMulti.
INFO      [20:26:40] 110575  - Reports for today will be processed at most every 14400 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO      [20:26:40] 110575  - Reports for the current week will be processed at most every 28800 seconds. You can change this value in config/config.ini.php by editing 'time_before_week_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:40] 110575  - Reports for the current month will be processed at most every 43200 seconds. You can change this value in config/config.ini.php by editing 'time_before_month_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:40] 110575  - Reports for the current year will be processed at most every 86400 seconds. You can change this value in config/config.ini.php by editing 'time_before_year_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:40] 110575  - Reports for the current range will be processed at most every 2700 seconds. You can change this value in config/config.ini.php by editing 'time_before_range_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:40] 110575  - Maximum 200 archives will be processed (soft limit).
INFO      [20:26:40] 110575  - Archiving was last executed without error 4 min 15s ago.
INFO      [20:26:40] 110575  - Will process 1 websites (--force-idsites)
INFO      [20:26:40] 110575  - Will process specified sites: 11
INFO      [20:26:40] 110575  ---------------------------
INFO      [20:26:40] 110575  START
INFO      [20:26:40] 110575  Starting Matomo reports archiving...
INFO      [20:26:40] 110575  0 out of 3 archivers running currently
INFO      [20:26:40] 110575  Start processing archives for site 11.
INFO      [20:26:40] 110575    Will invalidate archived reports for today in site ID = 11's timezone (2024-03-01 00:00:00).
nohup: les entr�es sont ignor�es
INFO      [20:26:41] 110575    Will invalidate archived reports for yesterday in site ID = 11's timezone (2024-02-29 00:00:00).
INFO      [20:26:42] 111165  ---------------------------
INFO      [20:26:42] 111165  INIT
INFO      [20:26:42] 111165  Running Matomo 5.0.1 as Super User
INFO      [20:26:42] 111165  ---------------------------
INFO      [20:26:42] 111165  NOTES
INFO      [20:26:42] 111165  - Async process archiving supported, using CliMulti.
INFO      [20:26:42] 111165  - Reports for today will be processed at most every 14400 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO      [20:26:42] 111165  - Reports for the current week will be processed at most every 28800 seconds. You can change this value in config/config.ini.php by editing 'time_before_week_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:42] 111165  - Reports for the current month will be processed at most every 43200 seconds. You can change this value in config/config.ini.php by editing 'time_before_month_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:42] 111165  - Reports for the current year will be processed at most every 86400 seconds. You can change this value in config/config.ini.php by editing 'time_before_year_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:42] 111165  - Reports for the current range will be processed at most every 2700 seconds. You can change this value in config/config.ini.php by editing 'time_before_range_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:42] 111165  - Maximum 200 archives will be processed (soft limit).
INFO      [20:26:42] 111165  - Archiving was last executed without error 4 min 17s ago.
INFO      [20:26:42] 111165  - Will process 1 websites (--force-idsites)
INFO      [20:26:42] 111165  - Will process specified sites: 11
INFO      [20:26:42] 111165  ---------------------------
INFO      [20:26:42] 111165  START
INFO      [20:26:42] 111165  Starting Matomo reports archiving...
INFO      [20:26:42] 111165  0 out of 3 archivers running currently
INFO      [20:26:42] 111165  Start processing archives for site 11.
INFO      [20:26:42] 111165    Will invalidate archived reports for today in site ID = 11's timezone (2024-03-01 00:00:00).
nohup: les entr�es sont ignor�es
INFO      [20:26:43] 111165    Will invalidate archived reports for yesterday in site ID = 11's timezone (2024-02-29 00:00:00).
INFO      [20:26:43] 111707  ---------------------------
INFO      [20:26:43] 111707  INIT
INFO      [20:26:43] 111707  Running Matomo 5.0.1 as Super User
INFO      [20:26:43] 111707  ---------------------------
INFO      [20:26:43] 111707  NOTES
INFO      [20:26:43] 111707  - Async process archiving supported, using CliMulti.
INFO      [20:26:43] 111707  - Reports for today will be processed at most every 14400 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO      [20:26:43] 111707  - Reports for the current week will be processed at most every 28800 seconds. You can change this value in config/config.ini.php by editing 'time_before_week_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:43] 111707  - Reports for the current month will be processed at most every 43200 seconds. You can change this value in config/config.ini.php by editing 'time_before_month_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:43] 111707  - Reports for the current year will be processed at most every 86400 seconds. You can change this value in config/config.ini.php by editing 'time_before_year_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:43] 111707  - Reports for the current range will be processed at most every 2700 seconds. You can change this value in config/config.ini.php by editing 'time_before_range_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:43] 111707  - Maximum 200 archives will be processed (soft limit).
INFO      [20:26:43] 111707  - Archiving was last executed without error 4 min 18s ago.
INFO      [20:26:43] 111707  - Will process 1 websites (--force-idsites)
INFO      [20:26:43] 111707  - Will process specified sites: 11
INFO      [20:26:43] 111707  ---------------------------
INFO      [20:26:43] 111707  START
INFO      [20:26:43] 111707  Starting Matomo reports archiving...
INFO      [20:26:43] 111707  0 out of 3 archivers running currently
INFO      [20:26:43] 111707  Start processing archives for site 11.
INFO      [20:26:43] 111707    Will invalidate archived reports for today in site ID = 11's timezone (2024-03-01 00:00:00).
INFO      [20:26:44] 111707    Will invalidate archived reports for yesterday in site ID = 11's timezone (2024-02-29 00:00:00).



THIS IS LOG FROM RUNNING IN COMMAND LINE CLI.
NOTICE THAT IT QUICKLY SHUTS DOWN - TOO MANY ARCHIVERS TO RUN


INFO [20:39:21] 21625 ---------------------------
INFO [20:39:21] 21625 INIT
INFO [20:39:21] 21625 Running Matomo 5.0.1 as Super User
INFO [20:39:21] 21625 ---------------------------
INFO [20:39:21] 21625 NOTES
INFO [20:39:21] 21625 - Async process archiving supported, using CliMulti.
INFO [20:39:21] 21625 - Reports for today will be processed at most every 14400 seconds. You can hange this value in Matomo UI > Settings > General Settings.
INFO [20:39:21] 21625 - Reports for the current week will be processed at most every 28800 seconds. You can change this value in config/config.ini.php by editing 'time_before_week_archive_considered_outdated' in the '[General]' section.
INFO [20:39:21] 21625 - Reports for the current month will be processed at most every 43200 seconds. You can change this value in config/config.ini.php by editing 'time_before_month_archive_considered_outdated' in the '[General]' section.
INFO [20:39:21] 21625 - Reports for the current year will be processed at most every 86400 seconds. You can change this value in config/config.ini.php by editing 'time_before_year_archive_considered_outdated' in the '[General]' section.
INFO [20:39:21] 21625 - Reports for the current range will be processed at most every 2700 seconds. You can change this value in config/config.ini.php by editing 'time_before_range_archive_considered_outdated' in the '[General]' section.
INFO [20:39:21] 21625 - Maximum 200 archives will be processed (soft limit).
INFO [20:39:21] 21625 - Archiving was last executed without error 16 min 56s ago.
INFO [20:39:21] 21625 - Will process 1 websites (--force-idsites)
INFO [20:39:21] 21625 - Will process specified sites: 11
INFO [20:39:21] 21625 ---------------------------
INFO [20:39:21] 21625 START
INFO [20:39:21] 21625 Starting Matomo reports archiving...
INFO [20:39:22] 21625 Archiving will stop now because 14 archivers are already running and max 3 are supposed to run at once.
INFO [20:39:22] 21625 Reached maximum concurrent archivers allowed (3), aborting run.
INFO [20:39:22] 21625 ---------------------------

Validations
@atom-box atom-box added Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. To Triage An issue awaiting triage by a Matomo core team member labels Mar 1, 2024
@atom-box
Copy link
Contributor Author

atom-box commented Mar 4, 2024

The user had a second Matomo instance, on Matomo 4. But now they report this:

We've had the same behavior on the 2nd instance since the matomo 5 update on Friday night.

@mneudert
Copy link
Member

As already mentioned internally we need more details to reproduce the issue.

Just using nohup or running the archiving via crontab should not output "0 out of x archivers". It should always at least find itself and display a positive number (at least 1).

Without more knowledge of the system it may be impossible to reproduce (and fix) it.

@mneudert mneudert added Waiting for user feedback Indicates the Matomo team is waiting for feedback from the author or other users. and removed To Triage An issue awaiting triage by a Matomo core team member labels Mar 20, 2024
@atom-box
Copy link
Contributor Author

atom-box commented Jun 28, 2024 •
edited
Loading

I sent a message to the person who discovered this issue.

@atom-box
Copy link
Contributor Author

atom-box commented Jul 1, 2024

The user found a workaround as follows:

We found a « solution » for this issue : no more problem since we use php-fpm (fpm-fcgi) and no Apache php module (mod_php).

The problem is still there in matomo 5.1 with mod_php

We didn't have this problem before matomo 5 and with mod_php

@BoyanH BoyanH mentioned this issue Aug 15, 2024
Closed
4 tasks
sgiehl added a commit that referenced this issue Aug 19, 2024
@sgiehl
Ensure process list isn't cut of when looking for running processes
7da5454 
fixes #22508
fixes #21985
@sgiehl sgiehl mentioned this issue Aug 19, 2024
Merged
11 tasks
@sgiehl
Copy link
Member

sgiehl commented Aug 19, 2024

Closing as duplicate of #22508, as the other issue contains more detailed information about the actual problem.

@sgiehl sgiehl closed this as not planned Won't fix, can't repro, duplicate, stale Aug 19, 2024
@sgiehl sgiehl added duplicate For issues that already existed in our issue tracker and were reported previously. and removed Waiting for user feedback Indicates the Matomo team is waiting for feedback from the author or other users. Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. labels Aug 19, 2024
mneudert pushed a commit that referenced this issue Aug 19, 2024
@sgiehl
Ensure process list isn't cut of when looking for running processes (#…
8e74ba8 
…22515)

fixes #22508
fixes #21985
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
duplicate For issues that already existed in our issue tracker and were reported previously.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Ensure process list isn't cut of when looking for running processes matomo-org/matomo
3 participants
@mneudert @sgiehl @atom-box