Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use RecordBuilders in VisitTime plugin #20766

Merged
merged 102 commits into from
Jul 4, 2023
Merged
Show file tree
Hide file tree
Changes from 101 commits
Commits
Show all changes
102 commits
Select commit Hold shift + click to select a range
6a388a0
introduce RecordBuilder concept and re-organize Goals archiving code …
diosmosis Feb 23, 2023
5f42975
fix loop iteration bug
diosmosis Feb 23, 2023
a9c3168
Merge branch '5.x-dev' into record-builders-poc
diosmosis Feb 25, 2023
5921f9d
split ecommerce records recordbuilder into 3 separate records
diosmosis Feb 26, 2023
b2d8e10
make sure Goals::getRecordMetadata() behaves like old archiver code
diosmosis Feb 26, 2023
2be26f2
make sure recordbuilder archive processor is restored after being use…
diosmosis Feb 26, 2023
0db7cb5
just make ArchiveProcessor a parameter
diosmosis Feb 26, 2023
fda7ee7
check for plugin before calling buildMultiplePeriod()
diosmosis Feb 26, 2023
b2ea252
do not invoke record builders if archiver has no plugin (happens duri…
diosmosis Feb 26, 2023
6cfe3ee
insert empty DataTables (as this appears to be the existing behavior …
diosmosis Feb 26, 2023
43bd346
add RecordBuilder class name to aggregation query hint
diosmosis Feb 26, 2023
c24d3aa
clear up in-source todo
diosmosis Feb 26, 2023
c2de813
attempt only archiving requested report if range archive and the reco…
diosmosis Mar 3, 2023
0139de3
refactor ArchiveSelector::getArchiveIds() to provide result with stri…
diosmosis Mar 4, 2023
f7e3831
when all found archives are partial archives, check that requested da…
diosmosis Mar 4, 2023
70b5bab
return correct value in Model::getRecordsContainedInArchives()
diosmosis Mar 4, 2023
d05f243
fix if formatting
diosmosis Mar 4, 2023
1609d31
existingArchives can be falsy
diosmosis Mar 4, 2023
234467a
existing archives can be null if the check is not relevant to the cur…
diosmosis Mar 5, 2023
cb1cf08
do not archive dependent segments if only processing the specific req…
diosmosis Mar 5, 2023
21b62d8
fix more tests
diosmosis Mar 5, 2023
99599e9
fix LoaderTest
diosmosis Mar 5, 2023
46abe79
make sure if archiving specific reports for a single plugin that arch…
diosmosis Mar 5, 2023
4c2701b
add filterRecordBuilders event
diosmosis Mar 15, 2023
c817f79
if it looks like the requested records are numeric, prioritize the nu…
diosmosis Mar 15, 2023
d18b61e
fix copy-paste error
diosmosis Mar 15, 2023
f6714dc
add dummy test for numeric values
diosmosis Mar 16, 2023
52c4e93
add test for partial archiving of numeric records for ranges and fix …
diosmosis Mar 16, 2023
e1938de
Merge branch '5.x-dev' into record-builders-poc
diosmosis Apr 20, 2023
ce8f7a0
lessen code redundancy in Archive.php, use Piwik\\Request and do not …
diosmosis Apr 20, 2023
5b719b1
fix type hint
diosmosis Apr 20, 2023
8b48227
fix php-cs errors
diosmosis Apr 20, 2023
16c5857
fix failing tests
diosmosis Apr 20, 2023
0d5bd3d
fix failing tests (really)
diosmosis Apr 20, 2023
a30c89e
Merge branch '5.x-dev' into record-builders-poc
diosmosis May 7, 2023
e6b3ca2
move sitesearch category archiving to record builder
diosmosis May 9, 2023
a5c2685
unfinished
diosmosis May 9, 2023
1d01f73
move rest of actions archiving to ActionReports + add option for blob…
diosmosis May 11, 2023
bab6124
simplify, add typehints and get simple tests to pass
diosmosis May 11, 2023
991979d
only do recursive row count in RecordBuilder if there is a numeric re…
diosmosis May 11, 2023
e7f60b0
Merge branch '5.x-dev' into record-builders-poc
diosmosis May 11, 2023
f2dbac8
Merge branch 'record-builders-poc' into actions-record-builder
diosmosis May 11, 2023
81e9ead
fix isEnabled calls
diosmosis May 11, 2023
1496f99
Merge branch 'record-builders-poc' into actions-record-builder
diosmosis May 11, 2023
94b98da
only add idarchive to Archive.php idarchive cache if it is not alread…
diosmosis May 11, 2023
63146a3
remove unneeded TODO
diosmosis May 11, 2023
6b09b25
when forcing new archive because timestamp is too old, do not report …
diosmosis May 11, 2023
23e2e37
report no existing archives if done flag is different + add tests
diosmosis May 12, 2023
111d15e
Merge branch 'record-builders-poc' into actions-record-builder
diosmosis May 12, 2023
369071a
array intersect instead of array dif
diosmosis May 13, 2023
d092a5c
ArchivingHelper::updateActionsTableWithRowQuery() needs access to eve…
diosmosis May 13, 2023
03f1d36
should never insert datatable maps
diosmosis May 13, 2023
2465be9
change ordering of row deletion
diosmosis May 13, 2023
e770d71
use correct re-introduced variable
diosmosis May 13, 2023
e900f40
fix ranking query override
diosmosis May 13, 2023
a7b9cde
fix OneVisitorOneWebsiteSeveralDaysDateRangeArchivingTest
diosmosis May 13, 2023
fd73433
one more test failure fix
diosmosis May 13, 2023
e07f08f
fix one more test
diosmosis May 13, 2023
f50ece3
move content archiving code to RecordBuilder
diosmosis May 14, 2023
f0e9ca4
start on using record builders for custom dimensions
diosmosis May 14, 2023
04a4f41
remove comment
diosmosis May 14, 2023
c8f7972
remove unused imports
diosmosis May 14, 2023
602d4a2
Merge branch 'actions-record-builder' into contents-record-builder
diosmosis May 14, 2023
b8c7c6a
remove unused imports
diosmosis May 14, 2023
386b234
make sure when numeric records that depend on blob records are archiv…
diosmosis May 14, 2023
eea3211
remove unneeded unset
diosmosis May 14, 2023
42a1763
Merge branch 'actions-record-builder' into contents-record-builder
diosmosis May 14, 2023
d4e4e89
handle ranking query summary row
diosmosis May 14, 2023
c40dd02
ranking query can be null
diosmosis May 14, 2023
cf7ffc6
ignore interactions that do not have an impression
diosmosis May 14, 2023
875b497
fix analyze archive table test
diosmosis May 14, 2023
5ddcf72
Merge branch 'actions-record-builder' into custom-dimensions-record-b…
diosmosis May 14, 2023
8c4afbc
handle rankingquery summary row + convert values to float since they …
diosmosis May 15, 2023
ce74a8c
accidentally fixing BlobReportLimitingTest. maybe.
diosmosis May 15, 2023
4db5d0a
remove no longer needed DataArray subclass
diosmosis May 15, 2023
8e780aa
Merge branch '5.x-dev' into record-builders-poc
sgiehl May 15, 2023
6c610fc
fix phpcs
sgiehl May 15, 2023
5eea5a1
remove unneeded newline
diosmosis May 15, 2023
6a8b108
Merge branch 'record-builders-poc' into actions-record-builder
diosmosis May 15, 2023
a84d759
add clarifying comment
diosmosis May 15, 2023
62bcc75
move datatable modification methods to DataTable & Row since it will …
diosmosis May 15, 2023
637a559
Merge branch 'contents-record-builder' into custom-dimensions-record-…
diosmosis May 15, 2023
7751e5c
use new methods in DataTable, Row
diosmosis May 15, 2023
ee17a00
fix unrefactored function calls
diosmosis May 15, 2023
ff44892
Merge branch 'contents-record-builder' into custom-dimensions-record-…
diosmosis May 15, 2023
0ff27c0
fix copy paste errors
diosmosis May 16, 2023
5d7eda5
Merge branch 'contents-record-builder' into custom-dimensions-record-…
diosmosis May 16, 2023
fa3dd1e
allow using custom column aggregation ops when calling new DataTable …
diosmosis May 16, 2023
57e61c3
Merge branch 'contents-record-builder' into custom-dimensions-record-…
diosmosis May 16, 2023
c3c48b4
add newline for ContentRecords
diosmosis May 16, 2023
eddd536
Merge branch 'record-builders-poc' into contents-record-builder
diosmosis May 16, 2023
384075d
newline
diosmosis May 16, 2023
c94238f
Merge branch 'actions-record-builder' into contents-record-builder
diosmosis May 16, 2023
1a530e5
Merge branch 'actions-record-builder' into custom-dimensions-record-b…
diosmosis May 16, 2023
1a26904
Merge branch 'contents-record-builder' into custom-dimensions-record-…
diosmosis May 17, 2023
7508d41
move VisitTime archiving code to RecordBuilders
diosmosis May 19, 2023
0eee66e
Merge branch '5.x-dev' into visit-time-record-builder
diosmosis Jun 28, 2023
776e269
add missing typehints
diosmosis Jun 28, 2023
a565ea3
ensure same column ordering as before
diosmosis Jun 28, 2023
62ee874
fix record count in test
diosmosis Jun 28, 2023
8b64152
Merge branch '5.x-dev' into visit-time-record-builder
sgiehl Jul 4, 2023
70f408a
convert to float in Metrics::makeGoalColumnsRow()
diosmosis Jul 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 0 additions & 77 deletions plugins/VisitTime/Archiver.php
Original file line number Diff line number Diff line change
Expand Up @@ -9,85 +9,8 @@

namespace Piwik\Plugins\VisitTime;

use Piwik\DataArray;
use Piwik\Date;

class Archiver extends \Piwik\Plugin\Archiver
{
const SERVER_TIME_RECORD_NAME = 'VisitTime_serverTime';
const LOCAL_TIME_RECORD_NAME = 'VisitTime_localTime';

public function aggregateDayReport()
{
$this->aggregateByLocalTime();
$this->aggregateByServerTime();
}

public function aggregateMultipleReports()
{
$dataTableRecords = array(
self::LOCAL_TIME_RECORD_NAME,
self::SERVER_TIME_RECORD_NAME,
);
$columnsAggregationOperation = null;
$this->getProcessor()->aggregateDataTableRecords(
$dataTableRecords,
$maximumRowsInDataTableLevelZero = null,
$maximumRowsInSubDataTable = null,
$columnToSortByBeforeTruncation = null,
$columnsAggregationOperation,
$columnsToRenameAfterAggregation = null,
$countRowsRecursive = array());
}

protected function aggregateByServerTime()
{
$dataArray = $this->getLogAggregator()->getMetricsFromVisitByDimension(array("label" => "HOUR(log_visit.visit_first_action_time)"));
$query = $this->getLogAggregator()->queryConversionsByDimension(array("label" => "HOUR(log_conversion.server_time)"));
if ($query === false) {
return;
}

while ($conversionRow = $query->fetch()) {
$dataArray->sumMetricsGoals($conversionRow['label'], $conversionRow);
}
$dataArray->enrichMetricsWithConversions();
$dataArray = $this->convertTimeToLocalTimezone($dataArray);
$this->ensureAllHoursAreSet($dataArray);
$report = $dataArray->asDataTable()->getSerialized();
$this->getProcessor()->insertBlobRecord(self::SERVER_TIME_RECORD_NAME, $report);
}

protected function aggregateByLocalTime()
{
$array = $this->getLogAggregator()->getMetricsFromVisitByDimension("HOUR(log_visit.visitor_localtime)");
$this->ensureAllHoursAreSet($array);
$report = $array->asDataTable()->getSerialized();
$this->getProcessor()->insertBlobRecord(self::LOCAL_TIME_RECORD_NAME, $report);
}

protected function convertTimeToLocalTimezone(DataArray &$array)
{
$date = Date::factory($this->getProcessor()->getParams()->getDateStart()->getDateStartUTC())->toString();
$timezone = $this->getProcessor()->getParams()->getSite()->getTimezone();

$converted = array();
foreach ($array->getDataArray() as $hour => $stats) {
$datetime = $date . ' ' . $hour . ':00:00';
$hourInTz = (int)Date::factory($datetime, $timezone)->toString('H');
$converted[$hourInTz] = $stats;
}
return new DataArray($converted);
}

private function ensureAllHoursAreSet(DataArray &$array)
{
$data = $array->getDataArray();
for ($i = 0; $i <= 23; $i++) {
if (empty($data[$i])) {
$array->sumMetricsVisits($i, DataArray::makeEmptyRow());
}
}
}

}
37 changes: 37 additions & 0 deletions plugins/VisitTime/RecordBuilders/Base.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/

namespace Piwik\Plugins\VisitTime\RecordBuilders;

use Piwik\ArchiveProcessor\RecordBuilder;
use Piwik\DataTable;
use Piwik\Metrics;

abstract class Base extends RecordBuilder
{
protected function ensureAllHoursAreSet(DataTable $record): void
{
$emptyRow = [
Metrics::INDEX_NB_UNIQ_VISITORS => 0,
Metrics::INDEX_NB_VISITS => 0,
Metrics::INDEX_NB_ACTIONS => 0,
Metrics::INDEX_NB_USERS => 0,
Metrics::INDEX_MAX_ACTIONS => 0,
Metrics::INDEX_SUM_VISIT_LENGTH => 0,
Metrics::INDEX_BOUNCE_COUNT => 0,
Metrics::INDEX_NB_VISITS_CONVERTED => 0,
];

for ($i = 0; $i <= 23; $i++) {
if (!$record->getRowFromLabel($i)) {
$record->addRowFromSimpleArray(['label' => $i] + $emptyRow);
}
}
}
}
58 changes: 58 additions & 0 deletions plugins/VisitTime/RecordBuilders/LocalTime.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/

namespace Piwik\Plugins\VisitTime\RecordBuilders;

use Piwik\ArchiveProcessor;
use Piwik\ArchiveProcessor\Record;
use Piwik\DataTable;
use Piwik\DataTable\Filter\Sort;
use Piwik\Metrics;
use Piwik\Plugins\VisitTime\Archiver;

class LocalTime extends Base
{
public function getRecordMetadata(ArchiveProcessor $archiveProcessor): array
{
return [
Record::make(Record::TYPE_BLOB, Archiver::LOCAL_TIME_RECORD_NAME),
];
}

protected function aggregate(ArchiveProcessor $archiveProcessor): array
{
$logAggregator = $archiveProcessor->getLogAggregator();

$record = new DataTable();

$query = $logAggregator->queryVisitsByDimension(["label" => "HOUR(log_visit.visitor_localtime)"]);
while ($row = $query->fetch()) {
$columns = [
Metrics::INDEX_NB_UNIQ_VISITORS => $row[Metrics::INDEX_NB_UNIQ_VISITORS],
Metrics::INDEX_NB_VISITS => $row[Metrics::INDEX_NB_VISITS],
Metrics::INDEX_NB_ACTIONS => $row[Metrics::INDEX_NB_ACTIONS],
Metrics::INDEX_NB_USERS => $row[Metrics::INDEX_NB_USERS],
Metrics::INDEX_MAX_ACTIONS => $row[Metrics::INDEX_MAX_ACTIONS],
Metrics::INDEX_SUM_VISIT_LENGTH => $row[Metrics::INDEX_SUM_VISIT_LENGTH],
Metrics::INDEX_BOUNCE_COUNT => $row[Metrics::INDEX_BOUNCE_COUNT],
Metrics::INDEX_NB_VISITS_CONVERTED => $row[Metrics::INDEX_NB_VISITS_CONVERTED],
];

$record->sumRowWithLabel($row['label'], $columns);
}

$this->ensureAllHoursAreSet($record);

$record->filter(Sort::class, ['label', 'asc']);

return [
Archiver::LOCAL_TIME_RECORD_NAME => $record,
];
}
}
86 changes: 86 additions & 0 deletions plugins/VisitTime/RecordBuilders/ServerTime.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/

namespace Piwik\Plugins\VisitTime\RecordBuilders;

use Piwik\ArchiveProcessor;
use Piwik\ArchiveProcessor\Record;
use Piwik\DataTable;
use Piwik\DataTable\Filter\Sort;
use Piwik\Date;
use Piwik\Metrics;
use Piwik\Plugins\VisitTime\Archiver;

class ServerTime extends Base
{
public function getRecordMetadata(ArchiveProcessor $archiveProcessor): array
{
return [
Record::make(Record::TYPE_BLOB, Archiver::SERVER_TIME_RECORD_NAME),
];
}

protected function aggregate(ArchiveProcessor $archiveProcessor): array
{
$logAggregator = $archiveProcessor->getLogAggregator();

$record = new DataTable();

$query = $logAggregator->queryVisitsByDimension(["label" => "HOUR(log_visit.visit_first_action_time)"]);
while ($row = $query->fetch()) {
$row['label'] = $this->convertTimeToLocalTimezone($row['label'], $archiveProcessor);

$columns = [
Metrics::INDEX_NB_UNIQ_VISITORS => $row[Metrics::INDEX_NB_UNIQ_VISITORS],
Metrics::INDEX_NB_VISITS => $row[Metrics::INDEX_NB_VISITS],
Metrics::INDEX_NB_ACTIONS => $row[Metrics::INDEX_NB_ACTIONS],
Metrics::INDEX_NB_USERS => $row[Metrics::INDEX_NB_USERS],
Metrics::INDEX_MAX_ACTIONS => $row[Metrics::INDEX_MAX_ACTIONS],
Metrics::INDEX_SUM_VISIT_LENGTH => $row[Metrics::INDEX_SUM_VISIT_LENGTH],
Metrics::INDEX_BOUNCE_COUNT => $row[Metrics::INDEX_BOUNCE_COUNT],
Metrics::INDEX_NB_VISITS_CONVERTED => $row[Metrics::INDEX_NB_VISITS_CONVERTED],
];

$record->sumRowWithLabel($row['label'], $columns);
}

$query = $logAggregator->queryConversionsByDimension(["label" => "HOUR(log_conversion.server_time)"]);
while ($conversionRow = $query->fetch()) {
$idGoal = $conversionRow['idgoal'];
$columns = [
Metrics::INDEX_GOALS => [
$idGoal => Metrics::makeGoalColumnsRow($idGoal, $conversionRow),
],
];

$conversionRow['label'] = $this->convertTimeToLocalTimezone($conversionRow['label'], $archiveProcessor);
$record->sumRowWithLabel($conversionRow['label'], $columns);
}

$record->filter(DataTable\Filter\EnrichRecordWithGoalMetricSums::class);

$this->ensureAllHoursAreSet($record);

$record->filter(Sort::class, ['label', 'asc']);

return [
Archiver::SERVER_TIME_RECORD_NAME => $record,
];
}

protected function convertTimeToLocalTimezone(int $hour, ArchiveProcessor $archiveProcessor): int
{
$date = Date::factory($archiveProcessor->getParams()->getDateStart()->getDateStartUTC())->toString();
$timezone = $archiveProcessor->getParams()->getSite()->getTimezone();

$datetime = $date . ' ' . $hour . ':00:00';
$hourInTz = (int)Date::factory($datetime, $timezone)->toString('H');
return $hourInTz;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ public function test_checkArchiveRecords_whenPeriodIsRange()
$tests = array(
'archive_blob_2010_12' => ( ($expectedActionsBlobs+1) /*Actions*/
+ 1 /* Resolution */
+ 2 /* VisitTime */) * 3,
+ 1 /* VisitTime */) * 3,

/**
* segments: 9 (including all visits)
Expand Down
Loading