Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Setting a very high value for "Delete logs older than (days)" causes archiving issues #20927

Open
Starker3 opened this issue Jun 26, 2023 · 4 comments
Open

Setting a very high value for "Delete logs older than (days)" causes archiving issues #20927

Starker3 opened this issue Jun 26, 2023 · 4 comments
Labels
Bug For errors / faults / flaws / inconsistencies etc.
Milestone
For Prioritization

Comments

@Starker3
Copy link
Contributor

It seems that setting an incredibly high value for the option Delete logs older than (days)
Results in the archiver not being able to invalidate and process reports. For example setting this to 4000000000000 results in:

image


INFO [2023-06-23 10:51:01] 11536 Warning: the following Dates have not been invalidated, because they are earlier than your Log Deletion limit: 2023-06-01, 2023-06-02, 2023-06-03, 2023-06-04, 2023-06-05, 2023-06-06, 2023-06-07, 2023-06-08, 2023-06-09, 2023-06-10, 2023-06-11, 2023-06-12, 2023-06-13, 2023-06-14, 2023-06-15, 2023-06-16, 2023-06-17, 2023-06-18, 2023-06-19, 2023-06-20, 2023-06-21, 2023-06-22, 2023-06-23

The last day with logs is 2023-06-23.

Please disable 'Delete old Logs' or set it to a higher deletion threshold (eg. 180 days or 365 years).'.

INFO [2023-06-23 10:51:01] 11536 Success. The following dates were invalidated successfully:
@Starker3 Starker3 added Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. To Triage An issue awaiting triage by a Matomo core team member labels Jun 26, 2023
@sgiehl
Copy link
Member

sgiehl commented Jun 26, 2023

@Starker3 What exactly doesn't work in this case? The output above seems correct. It didn't invalidate any newer data and also didn't invalidate any old data. Did you run core:archive command?
I'd like to be able to reproduce that in order to find a proper fix. But anyway it might make sense to have an upper limit for that configuration in the UI. Guess it doesn't make sense to allow numbers that would be above like e.g. 5 years or so.

@sgiehl sgiehl added Waiting for user feedback Indicates the Matomo team is waiting for feedback from the author or other users. and removed To Triage An issue awaiting triage by a Matomo core team member labels Jun 26, 2023
@Starker3
Copy link
Contributor Author

@sgiehl The issue is that with such a high value for some reason it never invalidates any data. In this specific case the user didn't get any new reports generated for several weeks since setting this. So i.e. there was no report data since the beginning of the month and the above output was shown in the core:archive command. The same was output for each ID Site that was tried to process:

C:\Windows\system32>D:\PHP\php.exe "D:\inetpub\wwwroot\matomo\console" core:invalidate-report-data --dates=2023-06-01,today -vvv

PHP Warning: Module "mysqli" is already loaded in Unknown on line 0

INFO [2023-06-23 10:51:01] 11536 Invalidating day periods in 2023-06-01,today [segment = ]...

INFO [2023-06-23 10:51:01] 11536 Warning: the following Dates have not been invalidated, because they are earlier than your Log Deletion limit: 2023-06-01, 2023-06-02, 2023-06-03, 2023-06-04, 2023-06-05, 2023-06-06, 2023-06-07, 2023-06-08, 2023-06-09, 2023-06-10, 2023-06-11, 2023-06-12, 2023-06-13, 2023-06-14, 2023-06-15, 2023-06-16, 2023-06-17, 2023-06-18, 2023-06-19, 2023-06-20, 2023-06-21, 2023-06-22, 2023-06-23

The last day with logs is 2023-06-23.

Please disable 'Delete old Logs' or set it to a higher deletion threshold (eg. 180 days or 365 years).'.

INFO [2023-06-23 10:51:01] 11536 Success. The following dates were invalidated successfully:

INFO [2023-06-23 10:51:01] 11536 Invalidating week periods in 2023-06-01,today [segment = ]...

INFO [2023-06-23 10:51:01] 11536 Warning: the following Dates have not been invalidated, because they are earlier than your Log Deletion limit: 2023-05-29, 2023-06-05, 2023-06-12, 2023-06-19

The last day with logs is 2023-06-23.

Please disable 'Delete old Logs' or set it to a higher deletion threshold (eg. 180 days or 365 years).'.

INFO [2023-06-23 10:51:01] 11536 Success. The following dates were invalidated successfully:

INFO [2023-06-23 10:51:01] 11536 Invalidating month periods in 2023-06-01,today [segment = ]...

INFO [2023-06-23 10:51:01] 11536 Warning: the following Dates have not been invalidated, because they are earlier than your Log Deletion limit: 2023-06-01

The last day with logs is 2023-06-23.

Please disable 'Delete old Logs' or set it to a higher deletion threshold (eg. 180 days or 365 years).'.

INFO [2023-06-23 10:51:01] 11536 Success. The following dates were invalidated successfully:

INFO [2023-06-23 10:51:01] 11536 Invalidating year periods in 2023-06-01,today [segment = ]...

INFO [2023-06-23 10:51:01] 11536 Warning: the following Dates have not been invalidated, because they are earlier than your Log Deletion limit: 2023-01-01

The last day with logs is 2023-06-23.

Please disable 'Delete old Logs' or set it to a higher deletion threshold (eg. 180 days or 365 years).'.

INFO [2023-06-23 10:51:01] 11536 Success. The following dates were invalidated successfully:

C:\Windows\system32>D:\PHP\php.exe "D:\inetpub\wwwroot\matomo\console" core:archive --force-all-websites --force-date-range=2023-06-01,today -vvv

PHP Warning: Module "mysqli" is already loaded in Unknown on line 0

INFO [2023-06-23 10:51:40] 19948 ---------------------------

INFO [2023-06-23 10:51:40] 19948 INIT

INFO [2023-06-23 10:51:40] 19948 Running Matomo 4.14.2 as Super User

INFO [2023-06-23 10:51:40] 19948 ---------------------------

INFO [2023-06-23 10:51:40] 19948 NOTES

INFO [2023-06-23 10:51:40] 19948 - Async process archiving not supported, using curl requests.

INFO [2023-06-23 10:51:40] 19948 - Reports for today will be processed at most every 900 seconds. You can change this value in Matomo UI > Settings > General Settings.

INFO [2023-06-23 10:51:40] 19948 - Archiving was last executed without error 26 min 46s ago.

INFO [2023-06-23 10:51:40] 19948 - Will process all 13 websites

INFO [2023-06-23 10:51:40] 19948 ---------------------------

INFO [2023-06-23 10:51:40] 19948 START

INFO [2023-06-23 10:51:40] 19948 Starting Matomo reports archiving...

DEBUG [2023-06-23 10:51:40] 19948 Applying queued rearchiving...

INFO [2023-06-23 10:51:40] 19948 Start processing archives for site 1.

DEBUG [2023-06-23 10:51:40] 19948 Checking for queued invalidations...

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for today in site ID = 1's timezone (2023-06-23 00:00:00).

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for yesterday in site ID = 1's timezone (2023-06-22 00:00:00).

DEBUG [2023-06-23 10:51:40] 19948 Done invalidating

DEBUG [2023-06-23 10:51:40] 19948 No next invalidated archive.

INFO [2023-06-23 10:51:40] 19948 Finished archiving for site 1, 0 API requests, Time elapsed: 0.089s [1 / 13 done]

INFO [2023-06-23 10:51:40] 19948 Start processing archives for site 2.

DEBUG [2023-06-23 10:51:40] 19948 Checking for queued invalidations...

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for today in site ID = 2's timezone (2023-06-23 00:00:00).

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for yesterday in site ID = 2's timezone (2023-06-22 00:00:00).

DEBUG [2023-06-23 10:51:40] 19948 Done invalidating

DEBUG [2023-06-23 10:51:40] 19948 No next invalidated archive.

INFO [2023-06-23 10:51:40] 19948 Finished archiving for site 2, 0 API requests, Time elapsed: 0.046s [2 / 13 done]

INFO [2023-06-23 10:51:40] 19948 Start processing archives for site 3.

DEBUG [2023-06-23 10:51:40] 19948 Checking for queued invalidations...

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for today in site ID = 3's timezone (2023-06-23 00:00:00).

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for yesterday in site ID = 3's timezone (2023-06-22 00:00:00).

DEBUG [2023-06-23 10:51:40] 19948 Done invalidating

DEBUG [2023-06-23 10:51:40] 19948 No next invalidated archive.

INFO [2023-06-23 10:51:40] 19948 Finished archiving for site 3, 0 API requests, Time elapsed: 0.034s [3 / 13 done]

INFO [2023-06-23 10:51:40] 19948 Start processing archives for site 4.

DEBUG [2023-06-23 10:51:40] 19948 Checking for queued invalidations...

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for today in site ID = 4's timezone (2023-06-23 00:00:00).

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for yesterday in site ID = 4's timezone (2023-06-22 00:00:00).

DEBUG [2023-06-23 10:51:40] 19948 Done invalidating

DEBUG [2023-06-23 10:51:40] 19948 No next invalidated archive.

INFO [2023-06-23 10:51:40] 19948 Finished archiving for site 4, 0 API requests, Time elapsed: 0.041s [4 / 13 done]

INFO [2023-06-23 10:51:40] 19948 Start processing archives for site 5.

DEBUG [2023-06-23 10:51:40] 19948 Checking for queued invalidations...

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for today in site ID = 5's timezone (2023-06-23 00:00:00).

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for yesterday in site ID = 5's timezone (2023-06-22 00:00:00).

DEBUG [2023-06-23 10:51:40] 19948 Done invalidating

DEBUG [2023-06-23 10:51:40] 19948 No next invalidated archive.

INFO [2023-06-23 10:51:40] 19948 Finished archiving for site 5, 0 API requests, Time elapsed: 0.034s [5 / 13 done]

INFO [2023-06-23 10:51:40] 19948 Start processing archives for site 6.

DEBUG [2023-06-23 10:51:40] 19948 Checking for queued invalidations...

DEBUG [2023-06-23 10:51:40] 19948 Today archive can be skipped due to no visits for idSite = 6, skipping invalidation...

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for yesterday in site ID = 6's timezone (2023-06-22 00:00:00).

DEBUG [2023-06-23 10:51:40] 19948 Done invalidating

DEBUG [2023-06-23 10:51:40] 19948 No next invalidated archive.

INFO [2023-06-23 10:51:40] 19948 Finished archiving for site 6, 0 API requests, Time elapsed: 0.030s [6 / 13 done]

INFO [2023-06-23 10:51:40] 19948 Start processing archives for site 7.

DEBUG [2023-06-23 10:51:40] 19948 Checking for queued invalidations...

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for today in site ID = 7's timezone (2023-06-23 00:00:00).

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for yesterday in site ID = 7's timezone (2023-06-22 00:00:00).

DEBUG [2023-06-23 10:51:40] 19948 Done invalidating

DEBUG [2023-06-23 10:51:40] 19948 No next invalidated archive.

INFO [2023-06-23 10:51:40] 19948 Finished archiving for site 7, 0 API requests, Time elapsed: 0.032s [7 / 13 done]

INFO [2023-06-23 10:51:40] 19948 Start processing archives for site 8.

DEBUG [2023-06-23 10:51:40] 19948 Checking for queued invalidations...

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for today in site ID = 8's timezone (2023-06-23 00:00:00).

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for yesterday in site ID = 8's timezone (2023-06-22 00:00:00).

DEBUG [2023-06-23 10:51:40] 19948 Done invalidating

DEBUG [2023-06-23 10:51:40] 19948 No next invalidated archive.

INFO [2023-06-23 10:51:40] 19948 Finished archiving for site 8, 0 API requests, Time elapsed: 0.036s [8 / 13 done]

INFO [2023-06-23 10:51:40] 19948 Start processing archives for site 9.

DEBUG [2023-06-23 10:51:40] 19948 Checking for queued invalidations...

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for today in site ID = 9's timezone (2023-06-23 00:00:00).

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for yesterday in site ID = 9's timezone (2023-06-22 00:00:00).

DEBUG [2023-06-23 10:51:40] 19948 Done invalidating

DEBUG [2023-06-23 10:51:40] 19948 No next invalidated archive.

INFO [2023-06-23 10:51:40] 19948 Finished archiving for site 9, 0 API requests, Time elapsed: 0.037s [9 / 13 done]

INFO [2023-06-23 10:51:40] 19948 Start processing archives for site 10.

DEBUG [2023-06-23 10:51:40] 19948 Checking for queued invalidations...

DEBUG [2023-06-23 10:51:40] 19948 Today archive can be skipped due to no visits for idSite = 10, skipping invalidation...

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for yesterday in site ID = 10's timezone (2023-06-22 00:00:00).

DEBUG [2023-06-23 10:51:40] 19948 Done invalidating

DEBUG [2023-06-23 10:51:40] 19948 No next invalidated archive.

INFO [2023-06-23 10:51:40] 19948 Finished archiving for site 10, 0 API requests, Time elapsed: 0.030s [10 / 13 done]

INFO [2023-06-23 10:51:40] 19948 Start processing archives for site 11.

DEBUG [2023-06-23 10:51:40] 19948 Checking for queued invalidations...

INFO [2023-06-23 10:51:40] 19948 Will invalidate archived reports for today in site ID = 11's timezone (2023-06-23 00:00:00).

DEBUG [2023-06-23 10:51:40] 19948 Yesterday archive can be skipped due to no visits for idSite = 11, skipping invalidation...

DEBUG [2023-06-23 10:51:40] 19948 Done invalidating

DEBUG [2023-06-23 10:51:40] 19948 No next invalidated archive.

INFO [2023-06-23 10:51:40] 19948 Finished archiving for site 11, 0 API requests, Time elapsed: 0.027s [11 / 13 done]

INFO [2023-06-23 10:51:40] 19948 Start processing archives for site 12.

DEBUG [2023-06-23 10:51:40] 19948 Checking for queued invalidations...

DEBUG [2023-06-23 10:51:40] 19948 Today archive can be skipped due to no visits for idSite = 12, skipping invalidation...

DEBUG [2023-06-23 10:51:40] 19948 Yesterday archive can be skipped due to no visits for idSite = 12, skipping invalidation...

DEBUG [2023-06-23 10:51:40] 19948 Done invalidating

DEBUG [2023-06-23 10:51:40] 19948 No next invalidated archive.

INFO [2023-06-23 10:51:40] 19948 Finished archiving for site 12, 0 API requests, Time elapsed: 0.020s [12 / 13 done]

INFO [2023-06-23 10:51:40] 19948 Start processing archives for site 15.

DEBUG [2023-06-23 10:51:40] 19948 Checking for queued invalidations...

DEBUG [2023-06-23 10:51:40] 19948 Today archive can be skipped due to no visits for idSite = 15, skipping invalidation...

DEBUG [2023-06-23 10:51:40] 19948 Yesterday archive can be skipped due to no visits for idSite = 15, skipping invalidation...

DEBUG [2023-06-23 10:51:40] 19948 Done invalidating

DEBUG [2023-06-23 10:51:40] 19948 No next invalidated archive.

DEBUG [2023-06-23 10:51:40] 19948 Archiving for imported site was finished, but date environment variable not set. Cannot mark day as complete.

INFO [2023-06-23 10:51:40] 19948 Finished archiving for site 15, 0 API requests, Time elapsed: 0.020s [13 / 13 done]

DEBUG [2023-06-23 10:51:40] 19948 No more sites left to archive, stopping.

INFO [2023-06-23 10:51:40] 19948 Done archiving!

INFO [2023-06-23 10:51:40] 19948 ---------------------------

INFO [2023-06-23 10:51:40] 19948 SUMMARY

INFO [2023-06-23 10:51:40] 19948 Processed 0 archives.

INFO [2023-06-23 10:51:40] 19948 Total API requests: 0

INFO [2023-06-23 10:51:40] 19948 done: 0 req, 510 ms, no error

INFO [2023-06-23 10:51:40] 19948 Time elapsed: 0.514s

INFO [2023-06-23 10:51:40] 19948 ---------------------------

@michalkleiner
Copy link
Contributor

A guess would be that such high number overflows somewhere or the date conversion goes back to unsupported date range such as before the start of the unix epoch, causing some (possibly silent) error somewhere, preventing the archiving from running correctly.

Another thing I see weird is the "180 days or 365 years" suggestion, probably should be both days.

@sgiehl sgiehl added Bug For errors / faults / flaws / inconsistencies etc. and removed Waiting for user feedback Indicates the Matomo team is waiting for feedback from the author or other users. Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. labels Jul 3, 2023
@sgiehl
Copy link
Member

sgiehl commented Jul 3, 2023

Let's simply disallow all values that would be bigger than e.g. 5 years or so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug For errors / faults / flaws / inconsistencies etc.
Projects
None yet
Milestone
For Prioritization
Development

No branches or pull requests
4 participants
@michalkleiner @sgiehl @Starker3 @innocraft-automation