Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

APIv2: TimeSeries using QueryBuilder, release experimental_session_count #4305

Merged
merged 43 commits into from
Jul 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
a6b2cd3
Move fragments module under Plausible.Stats.SQL
macobo Jun 28, 2024
bf795d6
Introduce select_merge_as macro
macobo Jun 28, 2024
57fee06
Simplify select_join_fields
macobo Jun 28, 2024
34a6025
Remove a needless dynamic
macobo Jun 28, 2024
588a5cc
wrap_select_columns macro
macobo Jun 28, 2024
a91501a
Move metrics from base.ex to expression.ex
macobo Jun 28, 2024
3dbdad1
Move WhereBuilder under Plausible.Stats.SQL
macobo Jun 28, 2024
dcfa7a6
Moduledoc
macobo Jun 28, 2024
1336940
Improved macros
macobo Jun 28, 2024
1698ead
Wrap more code
macobo Jun 28, 2024
4548461
select_merge_as more
macobo Jun 28, 2024
def9e26
Move defp to the end
macobo Jun 28, 2024
ffbf21d
include.time_labels parsing
macobo Jun 28, 2024
2b4c13c
include.time_labels in result
macobo Jul 1, 2024
69ece9a
Apply consistent function in imports and timeseries.ex
macobo Jul 1, 2024
1968d3d
Remove boilerplate
macobo Jul 1, 2024
5e9a2c6
WIP: Limited support for timeseries-with-querybuilder
macobo Jul 1, 2024
ab3cf05
time:week dimension
macobo Jul 1, 2024
0a43348
cleanup: property -> dimension
macobo Jul 1, 2024
7609750
Make querying with time series work
macobo Jul 1, 2024
73c79e7
Refactor: Move special metrics (percentage, conversion rate) to own m…
macobo Jul 2, 2024
2e24a85
Explicitly format datetimes
macobo Jul 2, 2024
5d355be
Consistent include_imported in special metrics
macobo Jul 2, 2024
87edab1
Solve week-related crash
macobo Jul 2, 2024
5f22239
conversion_rate hacking
macobo Jul 2, 2024
618525f
Keep include_imported consistent after splitting the query
macobo Jul 2, 2024
1d2bd2f
Simplify do_decide_tables
macobo Jul 2, 2024
01a99eb
Handle time dimensions in imports cleaner
macobo Jul 2, 2024
794aed1
Allow time dimensions in custom property queries
macobo Jul 2, 2024
44d2b0f
time:week handling continued
macobo Jul 2, 2024
ece6f32
cast_revenue_metrics_to_money
macobo Jul 3, 2024
4036103
fix `full_intervals` support
macobo Jul 3, 2024
400386f
Handle minute/realtime graphs
macobo Jul 3, 2024
18162b7
experimental_session_count? with timeseries
macobo Jul 3, 2024
333d93a
Support hourly data in imports
macobo Jul 3, 2024
4536b35
Update bounce_rate in more csv tests
macobo Jul 3, 2024
e4deb1d
Update some time-series query tests
macobo Jul 3, 2024
95ff561
Fix for meta.warning being included incorrectly
macobo Jul 3, 2024
53c8c6b
Simplify imported.ex
macobo Jul 3, 2024
7dd2eff
experimental_session_count flag removal
macobo Jul 3, 2024
54310b2
Merge remote-tracking branch 'origin/master' into apiv2-timeseries-re…
macobo Jul 3, 2024
61ff043
moduledoc
macobo Jul 3, 2024
3ed9b90
Split interval and time modules
macobo Jul 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ All notable changes to this project will be documented in this file.

### Changed

- Realtime and hourly graphs now show visits lasting their whole duration instead when specific events occur
- Increase hourly request limit for API keys in CE from 600 to 1000000 (practically removing the limit) plausible/analytics#4200
- Make TCP connections try IPv6 first with IPv4 fallback in CE plausible/analytics#4245
- `is` and `is not` filters in dashboard no longer support wildcards. Use contains/does not contain filter instead.
Expand Down
1 change: 0 additions & 1 deletion assets/js/dashboard/api.js
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ export function serializeQuery(query, extraQuery = []) {
if (query.from) { queryObj.from = formatISO(query.from) }
if (query.to) { queryObj.to = formatISO(query.to) }
if (query.filters) { queryObj.filters = serializeApiFilters(query.filters) }
if (query.experimental_session_count) { queryObj.experimental_session_count = query.experimental_session_count }
if (query.with_imported) { queryObj.with_imported = query.with_imported }
if (SHARED_LINK_AUTH) { queryObj.auth = SHARED_LINK_AUTH }

Expand Down
1 change: 0 additions & 1 deletion assets/js/dashboard/query.js
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@ export function parseQuery(querystring, site) {
to: q.get('to') ? dayjs.utc(q.get('to')) : undefined,
match_day_of_week: matchDayOfWeek == 'true',
with_imported: q.get('with_imported') ? q.get('with_imported') === 'true' : true,
experimental_session_count: q.get('experimental_session_count'),
filters: parseJsonUrl(q.get('filters'), []),
labels: parseJsonUrl(q.get('labels'), {})
}
Expand Down
80 changes: 0 additions & 80 deletions lib/plausible/stats/base.ex
Original file line number Diff line number Diff line change
Expand Up @@ -139,84 +139,4 @@ defmodule Plausible.Stats.Base do

"^#{escaped}$"
end

defp total_visitors(site, query) do
base_event_query(site, query)
|> select([e],
total_visitors: fragment("toUInt64(round(uniq(?) * any(_sample_factor)))", e.user_id)
)
end

# `total_visitors_subquery` returns a subquery which selects `total_visitors` -
# the number used as the denominator in the calculation of `conversion_rate` and
# `percentage` metrics.

# Usually, when calculating the totals, a new query is passed into this function,
# where certain filters (e.g. goal, props) are removed. That might make the query
# able to include imported data. However, we always want to include imported data
# only if it's included in the base query - otherwise the total will be based on
# a different data set, making the metric inaccurate. This is why we're using an
# explicit `include_imported` argument here.
def total_visitors_subquery(site, query, include_imported)

def total_visitors_subquery(site, query, true = _include_imported) do
wrap_alias([], %{
total_visitors:
subquery(total_visitors(site, query)) +
subquery(Plausible.Stats.Imported.total_imported_visitors(site, query))
})
end

def total_visitors_subquery(site, query, false = _include_imported) do
wrap_alias([], %{
total_visitors: subquery(total_visitors(site, query))
})
end

def add_percentage_metric(q, site, query, metrics) do
if :percentage in metrics do
total_query = Query.set_dimensions(query, [])

q
|> select_merge_as([], total_visitors_subquery(site, total_query, query.include_imported))
|> select_merge_as([], %{
percentage:
fragment(
"if(? > 0, round(? / ? * 100, 1), null)",
selected_as(:total_visitors),
selected_as(:visitors),
selected_as(:total_visitors)
)
})
else
q
end
end

# Adds conversion_rate metric to query, calculated as
# X / Y where Y is the same breakdown value without goal or props
# filters.
def maybe_add_conversion_rate(q, site, query, metrics) do
if :conversion_rate in metrics do
total_query =
query
|> Query.remove_filters(["event:goal", "event:props"])
|> Query.set_dimensions([])

# :TRICKY: Subquery is used due to event:goal breakdown above doing an UNION ALL
subquery(q)
|> select_merge_as([], total_visitors_subquery(site, total_query, query.include_imported))
|> select_merge_as([e], %{
conversion_rate:
fragment(
"if(? > 0, round(? / ? * 100, 1), 0)",
selected_as(:total_visitors),
e.visitors,
selected_as(:total_visitors)
)
})
else
q
end
end
end
10 changes: 5 additions & 5 deletions lib/plausible/stats/comparisons.ex
Original file line number Diff line number Diff line change
Expand Up @@ -163,21 +163,21 @@ defmodule Plausible.Stats.Comparisons do
end

defp maybe_include_imported(query, source_query) do
requested? = source_query.imported_data_requested
requested? = source_query.include.imports

case Query.ensure_include_imported(query, requested?) do
:ok ->
struct!(query,
imported_data_requested: true,
include_imported: true,
skip_imported_reason: nil
skip_imported_reason: nil,
include: Map.put(query.include, :imports, true)
)

{:error, reason} ->
struct!(query,
imported_data_requested: requested?,
include_imported: false,
skip_imported_reason: reason
skip_imported_reason: reason,
include: Map.put(query.include, :imports, requested?)
)
end
end
Expand Down
4 changes: 2 additions & 2 deletions lib/plausible/stats/filters/filters.ex
Original file line number Diff line number Diff line change
Expand Up @@ -83,8 +83,8 @@ defmodule Plausible.Stats.Filters do

def parse(_), do: []

def without_prefix(property) do
property
def without_prefix(dimension) do
dimension
|> String.split(":")
|> List.last()
|> String.to_existing_atom()
Expand Down
46 changes: 39 additions & 7 deletions lib/plausible/stats/filters/query_parser.ex
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@ defmodule Plausible.Stats.Filters.QueryParser do
alias Plausible.Stats.Query
alias Plausible.Stats.Metrics

@default_include %{
imports: false,
time_labels: false
}

def parse(site, params, now \\ nil) when is_map(params) do
with {:ok, metrics} <- parse_metrics(Map.get(params, "metrics", [])),
{:ok, filters} <- parse_filters(Map.get(params, "filters", [])),
Expand All @@ -22,13 +27,14 @@ defmodule Plausible.Stats.Filters.QueryParser do
dimensions: dimensions,
order_by: order_by,
timezone: site.timezone,
imported_data_requested: Map.get(include, :imports, false),
preloaded_goals: preloaded_goals
preloaded_goals: preloaded_goals,
include: include
},
:ok <- validate_order_by(query),
:ok <- validate_goal_filters(query),
:ok <- validate_custom_props_access(site, query),
:ok <- validate_metrics(query) do
:ok <- validate_metrics(query),
:ok <- validate_include(query) do
{:ok, query}
end
end
Expand Down Expand Up @@ -219,16 +225,32 @@ defmodule Plausible.Stats.Filters.QueryParser do
defp parse_time("time"), do: {:ok, "time"}
defp parse_time("time:hour"), do: {:ok, "time:hour"}
defp parse_time("time:day"), do: {:ok, "time:day"}
defp parse_time("time:week"), do: {:ok, "time:week"}
defp parse_time("time:month"), do: {:ok, "time:month"}
defp parse_time(_), do: :error

defp parse_order_direction([_, "asc"]), do: {:ok, :asc}
defp parse_order_direction([_, "desc"]), do: {:ok, :desc}
defp parse_order_direction(entry), do: {:error, "Invalid order_by entry '#{inspect(entry)}'"}

defp parse_include(%{"imports" => value}) when is_boolean(value), do: {:ok, %{imports: value}}
defp parse_include(%{}), do: {:ok, %{}}
defp parse_include(include), do: {:error, "Invalid include passed '#{inspect(include)}'"}
defp parse_include(include) when is_map(include) do
with {:ok, parsed_include_list} <- parse_list(include, &parse_include_value/1) do
include = Map.merge(@default_include, Enum.into(parsed_include_list, %{}))

{:ok, include}
end
end

defp parse_include(entry), do: {:error, "Invalid include passed '#{inspect(entry)}'"}

defp parse_include_value({"imports", value}) when is_boolean(value),
do: {:ok, {:imports, value}}

defp parse_include_value({"time_labels", value}) when is_boolean(value),
do: {:ok, {:time_labels, value}}

defp parse_include_value({key, value}),
do: {:error, "Invalid include entry '#{inspect(%{key => value})}'"}

defp parse_filter_key_string(filter_key, error_message \\ "") do
case filter_key do
Expand Down Expand Up @@ -386,14 +408,24 @@ defmodule Plausible.Stats.Filters.QueryParser do
end
end

def event_dimensions_not_allowing_session_metrics?(dimensions) do
defp event_dimensions_not_allowing_session_metrics?(dimensions) do
Enum.any?(dimensions, fn
"event:page" -> false
"event:" <> _ -> true
_ -> false
end)
end

defp validate_include(query) do
time_dimension? = Enum.any?(query.dimensions, &String.starts_with?(&1, "time"))

if query.include.time_labels and not time_dimension? do
{:error, "Invalid include.time_labels: requires a time dimension"}
else
:ok
end
end

defp parse_list(list, parser_function) do
Enum.reduce_while(list, {:ok, []}, fn value, {:ok, results} ->
case parser_function.(value) do
Expand Down
31 changes: 17 additions & 14 deletions lib/plausible/stats/imported/base.ex
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,19 @@ defmodule Plausible.Stats.Imported.Base do
"event:page" => "imported_pages",
"event:name" => "imported_custom_events",

# NOTE: these properties can be only filtered by
# NOTE: these dimensions can be only filtered by
"visit:screen" => "imported_devices",
"event:hostname" => "imported_pages"
"event:hostname" => "imported_pages",

# NOTE: These dimensions are only used in group by
"time:month" => "imported_visitors",
"time:week" => "imported_visitors",
"time:day" => "imported_visitors",
"time:hour" => "imported_visitors"
}

@queriable_time_dimensions ["time:month", "time:week", "time:day", "time:hour"]

@imported_custom_props Imported.imported_custom_props()

@db_field_mappings %{
Expand Down Expand Up @@ -121,9 +129,10 @@ defmodule Plausible.Stats.Imported.Base do
do_decide_custom_prop_table(query, dimension)
end

@queriable_custom_prop_dimensions ["event:goal", "event:name"] ++ @queriable_time_dimensions
defp do_decide_custom_prop_table(%{dimensions: dimensions} = query) do
if dimensions == [] or
(length(dimensions) == 1 and hd(dimensions) in ["event:goal", "event:name"]) do
(length(dimensions) == 1 and hd(dimensions) in @queriable_custom_prop_dimensions) do
custom_prop_filters =
query.filters
|> Enum.map(&Enum.at(&1, 1))
Expand Down Expand Up @@ -169,14 +178,6 @@ defmodule Plausible.Stats.Imported.Base do
["imported_pages", "imported_custom_events"]
end

defp do_decide_tables(%Query{filters: [], dimensions: [dimension]}) do
if Map.has_key?(@property_to_table_mappings, dimension) do
[@property_to_table_mappings[dimension]]
else
[]
end
end

defp do_decide_tables(%Query{filters: filters, dimensions: ["event:goal"]}) do
filter_props = Enum.map(filters, &Enum.at(&1, 1))

Expand All @@ -197,13 +198,15 @@ defmodule Plausible.Stats.Imported.Base do
filters
|> Enum.map(fn [_, filter_key | _] -> filter_key end)
|> Enum.concat(dimensions)
|> Enum.map(fn
"visit:screen" -> "visit:device"
dimension -> dimension
|> Enum.reject(&(&1 in @queriable_time_dimensions))
|> Enum.flat_map(fn
"visit:screen" -> ["visit:device"]
dimension -> [dimension]
end)
|> Enum.map(&@property_to_table_mappings[&1])

case Enum.uniq(table_candidates) do
[] -> ["imported_visitors"]
[nil] -> []
[candidate] -> [candidate]
_ -> []
Expand Down
Loading