-
-
Notifications
You must be signed in to change notification settings - Fork 1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Snippet integration verification (#4106)
* Allow running browserless.io locally * Compile tailwind classes based on extra/ too * Add browserless runtime configuration * Ignore verification events on ingestion * Improve extracting HTML text in tests * Update dependencies - Floki will be used on production to parse site contents - Req will be used to handle redundant stuff like retrying etc. * Add shuttle SVG to generic components Later on we'll use it to indicate verification errors * Connect live socket & allow skipping awaiting the first pageview * Connect live socket in general settings * Implement verification checks & diagnostics * Stub remote services with Req for testing * Change snippet screen copy * Update tracker script, so that: 1. headless browsers aren't ignored if `window.__plausible` is defined 2. callback optionally supplies the event response HTTP status This will be later used to check whether the server acknowledged the verification event. * Implement LiveView verification UI * Embed the verification UIs into settings and onboarding * Implement browserless puppeteer verification script It: - tries to visit the site - defines window.__plausible, so the tracker doesn't ignore test events - sends a verification event and instruments the callback - awaits the callback to fire and returns the result * Improve diagnostics for CSP Only report CSP error if the snippet is already found * Put verification behind a feature flag/env setting * Contact Us hint only for Enterprise Edition * For headless code, use JS context instead of EEx interpolation * Update diagnostics test with WordPress scenarios * Shorten exception/throw interception * Rename test * Tidy up * Bust URL always on headless check * Update moduledoc * Detect official Plausible WordPress Plugin and act accordingly on diagnostics interoperation * Stop using 'rating' in favour of 'interpretation' * Only report CSP error if no proxy is likely * Update CHANGELOG * Allow event-* attributes on snippet elements * Improve naive GTM detection, not to confuse it with GA4 * Update lib/plausible/verification.ex Co-authored-by: Adrian Gruntkowski <[email protected]> * Update test/plausible/site/verification/checks_test.exs Co-authored-by: Adrian Gruntkowski <[email protected]> * s/perform_wrapped/perform_safe * Update lib/plausible/verification/checks/installation.ex Co-authored-by: Adrian Gruntkowski <[email protected]> * Remove garbage --------- Co-authored-by: Adrian Gruntkowski <[email protected]>
- Loading branch information
Showing
44 changed files
with
2,838 additions
and
34 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
defmodule Plausible.Verification do | ||
@moduledoc """ | ||
Module defining the user-agent used for site verification. | ||
""" | ||
use Plausible | ||
|
||
@feature_flag :verification | ||
|
||
def enabled?(user) do | ||
enabled_via_config? = | ||
:plausible |> Application.fetch_env!(__MODULE__) |> Keyword.fetch!(:enabled?) | ||
|
||
enabled_for_user? = not is_nil(user) and FunWithFlags.enabled?(@feature_flag, for: user) | ||
enabled_via_config? or enabled_for_user? | ||
end | ||
|
||
on_ee do | ||
def user_agent() do | ||
"Plausible Verification Agent - if abused, contact [email protected]" | ||
end | ||
else | ||
def user_agent() do | ||
"Plausible Community Edition" | ||
end | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
defmodule Plausible.Verification.Check do | ||
@moduledoc """ | ||
Behaviour to be implemented by specific site verification checks. | ||
`friendly_name()` doesn't necessarily reflect the actual check description, | ||
it serves as a user-facing message grouping mechanism, to prevent frequent message flashing when checks rotate often. | ||
Each check operates on `state()` and is expected to return it, optionally modified, by all means. | ||
`perform_safe/1` is used to guarantee no exceptions are thrown by faulty implementations, not to interrupt LiveView. | ||
""" | ||
@type state() :: Plausible.Verification.State.t() | ||
@callback friendly_name() :: String.t() | ||
@callback perform(state()) :: state() | ||
|
||
defmacro __using__(_) do | ||
quote do | ||
import Plausible.Verification.State | ||
|
||
alias Plausible.Verification.Checks | ||
alias Plausible.Verification.State | ||
alias Plausible.Verification.Diagnostics | ||
|
||
require Logger | ||
|
||
@behaviour Plausible.Verification.Check | ||
|
||
def perform_safe(state) do | ||
perform(state) | ||
catch | ||
_, e -> | ||
Logger.error( | ||
"Error running check #{inspect(__MODULE__)} on #{state.url}: #{inspect(e)}" | ||
) | ||
|
||
put_diagnostics(state, service_error: true) | ||
end | ||
end | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,75 @@ | ||
defmodule Plausible.Verification.Checks do | ||
@moduledoc """ | ||
Checks that are performed during site verification. | ||
Each module defined in `@checks` implements the `Plausible.Verification.Check` behaviour. | ||
Checks are normally run asynchronously, except when synchronous execution is optionally required | ||
for tests. Slowdowns can be optionally added, the user doesn't benefit from running the checks too quickly. | ||
In async execution, each check notifies the caller by sending a message to it. | ||
""" | ||
alias Plausible.Verification.Checks | ||
alias Plausible.Verification.State | ||
|
||
require Logger | ||
|
||
@checks [ | ||
Checks.FetchBody, | ||
Checks.CSP, | ||
Checks.ScanBody, | ||
Checks.Snippet, | ||
Checks.SnippetCacheBust, | ||
Checks.Installation | ||
] | ||
|
||
def run(url, data_domain, opts \\ []) do | ||
checks = Keyword.get(opts, :checks, @checks) | ||
report_to = Keyword.get(opts, :report_to, self()) | ||
async? = Keyword.get(opts, :async?, true) | ||
slowdown = Keyword.get(opts, :slowdown, 500) | ||
|
||
if async? do | ||
Task.start_link(fn -> do_run(url, data_domain, checks, report_to, slowdown) end) | ||
else | ||
do_run(url, data_domain, checks, report_to, slowdown) | ||
end | ||
end | ||
|
||
def interpret_diagnostics(%State{} = state) do | ||
Plausible.Verification.Diagnostics.interpret(state.diagnostics, state.url) | ||
end | ||
|
||
defp do_run(url, data_domain, checks, report_to, slowdown) do | ||
init_state = %State{url: url, data_domain: data_domain, report_to: report_to} | ||
|
||
state = | ||
Enum.reduce( | ||
checks, | ||
init_state, | ||
fn check, state -> | ||
state | ||
|> notify_start(check, slowdown) | ||
|> check.perform_safe() | ||
end | ||
) | ||
|
||
notify_verification_end(state, slowdown) | ||
end | ||
|
||
defp notify_start(state, check, slowdown) do | ||
if is_pid(state.report_to) do | ||
if is_integer(slowdown) and slowdown > 0, do: :timer.sleep(slowdown) | ||
send(state.report_to, {:verification_check_start, {check, state}}) | ||
end | ||
|
||
state | ||
end | ||
|
||
defp notify_verification_end(state, slowdown) do | ||
if is_pid(state.report_to) do | ||
if is_integer(slowdown) and slowdown > 0, do: :timer.sleep(slowdown) | ||
send(state.report_to, {:verification_end, state}) | ||
end | ||
|
||
state | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
defmodule Plausible.Verification.Checks.CSP do | ||
@moduledoc """ | ||
Scans the Content Security Policy header to ensure that the Plausible domain is allowed. | ||
See `Plausible.Verification.Checks` for the execution sequence. | ||
""" | ||
use Plausible.Verification.Check | ||
|
||
@impl true | ||
def friendly_name, do: "We're visiting your site to ensure that everything is working correctly" | ||
|
||
@impl true | ||
def perform(%State{assigns: %{headers: headers}} = state) do | ||
case headers["content-security-policy"] do | ||
[policy] -> | ||
directives = String.split(policy, ";") | ||
|
||
allowed? = | ||
Enum.any?(directives, fn directive -> | ||
String.contains?(directive, PlausibleWeb.Endpoint.host()) | ||
end) | ||
|
||
if allowed? do | ||
state | ||
else | ||
put_diagnostics(state, disallowed_via_csp?: true) | ||
end | ||
|
||
_ -> | ||
state | ||
end | ||
end | ||
|
||
def perform(state), do: state | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
defmodule Plausible.Verification.Checks.FetchBody do | ||
@moduledoc """ | ||
Fetches the body of the site and extracts the HTML document, if available, for | ||
further processing. | ||
See `Plausible.Verification.Checks` for the execution sequence. | ||
""" | ||
use Plausible.Verification.Check | ||
|
||
@impl true | ||
def friendly_name, do: "We're visiting your site to ensure that everything is working correctly" | ||
|
||
@impl true | ||
def perform(%State{url: "https://" <> _ = url} = state) do | ||
fetch_body_opts = Application.get_env(:plausible, __MODULE__)[:req_opts] || [] | ||
|
||
opts = | ||
Keyword.merge( | ||
[ | ||
base_url: url, | ||
max_redirects: 2, | ||
connect_options: [timeout: 4_000], | ||
receive_timeout: 4_000, | ||
max_retries: 3, | ||
retry_log_level: :warning | ||
], | ||
fetch_body_opts | ||
) | ||
|
||
req = Req.new(opts) | ||
|
||
case Req.get(req) do | ||
{:ok, %Req.Response{status: status, body: body} = response} | ||
when is_binary(body) and status in 200..299 -> | ||
extract_document(state, response) | ||
|
||
_ -> | ||
state | ||
end | ||
end | ||
|
||
defp extract_document(state, response) when byte_size(response.body) <= 500_000 do | ||
with true <- html?(response), | ||
{:ok, document} <- Floki.parse_document(response.body) do | ||
state | ||
|> assign(raw_body: response.body, document: document, headers: response.headers) | ||
|> put_diagnostics(body_fetched?: true) | ||
else | ||
_ -> | ||
state | ||
end | ||
end | ||
|
||
defp extract_document(state, response) when byte_size(response.body) > 500_000 do | ||
state | ||
end | ||
|
||
defp html?(%Req.Response{headers: headers}) do | ||
headers | ||
|> Map.get("content-type", "") | ||
|> List.wrap() | ||
|> List.first() | ||
|> String.contains?("text/html") | ||
end | ||
end |
Oops, something went wrong.