From b3c6c0a88dab1817d62418921e501c5e198186d2 Mon Sep 17 00:00:00 2001 From: Benjamin Wohlwend Date: Wed, 27 Oct 2021 16:14:21 +0200 Subject: [PATCH 1/6] Added non-agent-specific docs on span compression --- docs/span-compression.asciidoc | 54 ++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 docs/span-compression.asciidoc diff --git a/docs/span-compression.asciidoc b/docs/span-compression.asciidoc new file mode 100644 index 0000000000..fb7156c908 --- /dev/null +++ b/docs/span-compression.asciidoc @@ -0,0 +1,54 @@ +[[span-compression]] +=== Span compression + +In some cases, the agents may collect large amounts of very similar or identical spans in a transaction. +This can e.g. happen if spans are captured inside a loop, or for unoptimized SQL queries that use multiple queries instead of joins to fetch related data. +In such cases, the upper limit of spans per transaction (by default 500 spans) can be reached quickly, causing the agent to stop capturing possibly more relevant spans for a given transaction. + +Such repeated similar spans often aren't very relevant for themselves, especially if they are of very short duration. +They also can clutter the UI, and cause processing and storage overhead. + +To address this problem, the APM agents can compress such spans into a single span. +The compressed span retains some information on the original spans, such as overall count and duration. + + +[float] +==== Compression strategies + +The agent can select between two strategies to decide if two adjacent spans can be compressed. +Both strategies have the benefit that only one previous span needs to be kept in memory. +This is important to ensure that the agent doesn't require large amounts of memory to enable span compression. + +[float] +===== Same-Kind strategy + +The agent selects this strategy, if two adjacent spans have the same + + * span type + * span subtype + * `destination.service.resource` (e.g. database name) + +[float] +===== Exact-Match strategy + +The agent selects this strategy, if two adjacent spans have the same + + * span name + * span type + * span subtype + * `destination.service.resource` (e.g. database name) + +[float] +===== Settings + +The agent has configuration settings to define upper thresholds in terms of span duration for both strategies. +For the "Same-Kind" strategy, the limit is 5ms. For the "Exact-Match" strategy, the limit is 50ms. +Spans with longer duration are not compressed. Please refer to the agent documentation for specifics. + +[float] +===== Agent support + +Support for span compression is available in these agents: + + * Go + * Python \ No newline at end of file From c1984b3bfc286ed43b44ded7d0756064dd1ceeb5 Mon Sep 17 00:00:00 2001 From: Brandon Morelli Date: Wed, 27 Oct 2021 20:04:59 -0700 Subject: [PATCH 2/6] docs: hook it up and adjust heading levels --- docs/features.asciidoc | 3 +++ docs/span-compression.asciidoc | 16 ++++++++++------ 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/features.asciidoc b/docs/features.asciidoc index e16b55c903..cbf788ee87 100644 --- a/docs/features.asciidoc +++ b/docs/features.asciidoc @@ -12,6 +12,7 @@ * <> * <> * <> +* <> include::./apm-data-security.asciidoc[] @@ -26,3 +27,5 @@ include::./open-telemetry.asciidoc[] include::./log-correlation.asciidoc[] include::./cross-cluster-search.asciidoc[] + +include::./span-compression.asciidoc[] diff --git a/docs/span-compression.asciidoc b/docs/span-compression.asciidoc index fb7156c908..893ef0650a 100644 --- a/docs/span-compression.asciidoc +++ b/docs/span-compression.asciidoc @@ -11,16 +11,17 @@ They also can clutter the UI, and cause processing and storage overhead. To address this problem, the APM agents can compress such spans into a single span. The compressed span retains some information on the original spans, such as overall count and duration. - [float] -==== Compression strategies +[[span-compression-strategy]] +=== Compression strategies The agent can select between two strategies to decide if two adjacent spans can be compressed. Both strategies have the benefit that only one previous span needs to be kept in memory. This is important to ensure that the agent doesn't require large amounts of memory to enable span compression. [float] -===== Same-Kind strategy +[[span-compression-same]] +==== Same-Kind strategy The agent selects this strategy, if two adjacent spans have the same @@ -29,7 +30,8 @@ The agent selects this strategy, if two adjacent spans have the same * `destination.service.resource` (e.g. database name) [float] -===== Exact-Match strategy +[[span-compression-exact]] +==== Exact-Match strategy The agent selects this strategy, if two adjacent spans have the same @@ -39,14 +41,16 @@ The agent selects this strategy, if two adjacent spans have the same * `destination.service.resource` (e.g. database name) [float] -===== Settings +[[span-compression-settings]] +=== Settings The agent has configuration settings to define upper thresholds in terms of span duration for both strategies. For the "Same-Kind" strategy, the limit is 5ms. For the "Exact-Match" strategy, the limit is 50ms. Spans with longer duration are not compressed. Please refer to the agent documentation for specifics. [float] -===== Agent support +[[span-compression-support]] +=== Agent support Support for span compression is available in these agents: From d31c0bde5241d13aa8cd3228172a3e8eb6924840 Mon Sep 17 00:00:00 2001 From: Benjamin Wohlwend Date: Thu, 18 Nov 2021 09:54:15 +0100 Subject: [PATCH 3/6] Apply suggestions from code review Co-authored-by: Brandon Morelli Co-authored-by: Marc Lopez Rubio --- docs/span-compression.asciidoc | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/docs/span-compression.asciidoc b/docs/span-compression.asciidoc index 893ef0650a..8aa36e699d 100644 --- a/docs/span-compression.asciidoc +++ b/docs/span-compression.asciidoc @@ -1,21 +1,28 @@ [[span-compression]] === Span compression -In some cases, the agents may collect large amounts of very similar or identical spans in a transaction. -This can e.g. happen if spans are captured inside a loop, or for unoptimized SQL queries that use multiple queries instead of joins to fetch related data. -In such cases, the upper limit of spans per transaction (by default 500 spans) can be reached quickly, causing the agent to stop capturing possibly more relevant spans for a given transaction. +In some cases, APM agents may collect large amounts of very similar or identical spans in a transaction. +For example, this can happen if spans are captured inside of a loop, or in unoptimized SQL queries that use multiple queries instead of joins to fetch related data. +In such cases, the upper limit of spans per transaction (by default, 500 spans) can be reached quickly, causing the agent to stop capturing potentially more relevant spans for a given transaction. Such repeated similar spans often aren't very relevant for themselves, especially if they are of very short duration. They also can clutter the UI, and cause processing and storage overhead. To address this problem, the APM agents can compress such spans into a single span. -The compressed span retains some information on the original spans, such as overall count and duration. +The compressed span retains most of the original span information, such as overall duration and the number of spans it represents. + +Regardless of the compression strategy, a span is elegible for compression if: + +- It has not propagated its trace context. +- Is an _exit_ span (such as database query spans). +- Its outcome is not `"failure"`. + [float] [[span-compression-strategy]] === Compression strategies -The agent can select between two strategies to decide if two adjacent spans can be compressed. +The APM agent can select between two strategies to decide if two adjacent spans can be compressed. Both strategies have the benefit that only one previous span needs to be kept in memory. This is important to ensure that the agent doesn't require large amounts of memory to enable span compression. @@ -23,7 +30,7 @@ This is important to ensure that the agent doesn't require large amounts of memo [[span-compression-same]] ==== Same-Kind strategy -The agent selects this strategy, if two adjacent spans have the same +The agent selects this strategy if two adjacent spans have the same: * span type * span subtype @@ -33,7 +40,7 @@ The agent selects this strategy, if two adjacent spans have the same [[span-compression-exact]] ==== Exact-Match strategy -The agent selects this strategy, if two adjacent spans have the same +The agent selects this strategy if two adjacent spans have the same: * span name * span type From 719b726b07f02bfb0fd238c86467ab7de289db34 Mon Sep 17 00:00:00 2001 From: Benjamin Wohlwend Date: Mon, 22 Nov 2021 15:50:47 +0100 Subject: [PATCH 4/6] add links to agent docs --- docs/span-compression.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/span-compression.asciidoc b/docs/span-compression.asciidoc index 8aa36e699d..bf14746975 100644 --- a/docs/span-compression.asciidoc +++ b/docs/span-compression.asciidoc @@ -61,5 +61,5 @@ Spans with longer duration are not compressed. Please refer to the agent documen Support for span compression is available in these agents: - * Go - * Python \ No newline at end of file + * Go: {apm-go-ref-v}/configuration.html#config-span-compression-exact-match-duration[`ELASTIC_APM_SPAN_COMPRESSION_EXACT_MATCH_MAX_DURATION`], {apm-go-ref-v}/configuration.html#config-span-compression-same-kind-duration[`ELASTIC_APM_SPAN_COMPRESSION_SAME_KIND_MAX_DURATION`] + * Python: {apm-py-ref-v}/configuration.html#config-span-compression-exact-match-max_duration[`span_compression_exact_match_max_duration`], {apm-py-ref-v}/configuration.html#config-span-compression-same-kind-max-duration[`span_compression_same_kind_max_duration`] \ No newline at end of file From 87561fa9ab8e88e2b3150c35a0bf8107f0ee054c Mon Sep 17 00:00:00 2001 From: Benjamin Wohlwend Date: Mon, 22 Nov 2021 16:17:02 +0100 Subject: [PATCH 5/6] fix (hopefully) refs --- docs/span-compression.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/span-compression.asciidoc b/docs/span-compression.asciidoc index bf14746975..b8933e5b89 100644 --- a/docs/span-compression.asciidoc +++ b/docs/span-compression.asciidoc @@ -61,5 +61,5 @@ Spans with longer duration are not compressed. Please refer to the agent documen Support for span compression is available in these agents: - * Go: {apm-go-ref-v}/configuration.html#config-span-compression-exact-match-duration[`ELASTIC_APM_SPAN_COMPRESSION_EXACT_MATCH_MAX_DURATION`], {apm-go-ref-v}/configuration.html#config-span-compression-same-kind-duration[`ELASTIC_APM_SPAN_COMPRESSION_SAME_KIND_MAX_DURATION`] - * Python: {apm-py-ref-v}/configuration.html#config-span-compression-exact-match-max_duration[`span_compression_exact_match_max_duration`], {apm-py-ref-v}/configuration.html#config-span-compression-same-kind-max-duration[`span_compression_same_kind_max_duration`] \ No newline at end of file + * Go: {apm-go-ref}/configuration.html#config-span-compression-exact-match-duration[`ELASTIC_APM_SPAN_COMPRESSION_EXACT_MATCH_MAX_DURATION`], {apm-go-ref}/configuration.html#config-span-compression-same-kind-duration[`ELASTIC_APM_SPAN_COMPRESSION_SAME_KIND_MAX_DURATION`] + * Python: {apm-py-ref}/configuration.html#config-span-compression-exact-match-max_duration[`span_compression_exact_match_max_duration`], {apm-py-ref}/configuration.html#config-span-compression-same-kind-max-duration[`span_compression_same_kind_max_duration`] \ No newline at end of file From e1b7dc53391a16fcb0b32a7933d549d6c26c67ea Mon Sep 17 00:00:00 2001 From: Brandon Morelli Date: Tue, 23 Nov 2021 11:05:13 -0800 Subject: [PATCH 6/6] Update docs/span-compression.asciidoc --- docs/span-compression.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/span-compression.asciidoc b/docs/span-compression.asciidoc index b8933e5b89..e10c118352 100644 --- a/docs/span-compression.asciidoc +++ b/docs/span-compression.asciidoc @@ -61,5 +61,5 @@ Spans with longer duration are not compressed. Please refer to the agent documen Support for span compression is available in these agents: - * Go: {apm-go-ref}/configuration.html#config-span-compression-exact-match-duration[`ELASTIC_APM_SPAN_COMPRESSION_EXACT_MATCH_MAX_DURATION`], {apm-go-ref}/configuration.html#config-span-compression-same-kind-duration[`ELASTIC_APM_SPAN_COMPRESSION_SAME_KIND_MAX_DURATION`] + * Go: https://www.elastic.co/guide/en/apm/agent/go/master/configuration.html#config-span-compression-exact-match-duration[`ELASTIC_APM_SPAN_COMPRESSION_EXACT_MATCH_MAX_DURATION`], https://www.elastic.co/guide/en/apm/agent/go/master/configuration.html#config-span-compression-same-kind-duration[`ELASTIC_APM_SPAN_COMPRESSION_SAME_KIND_MAX_DURATION`] * Python: {apm-py-ref}/configuration.html#config-span-compression-exact-match-max_duration[`span_compression_exact_match_max_duration`], {apm-py-ref}/configuration.html#config-span-compression-same-kind-max-duration[`span_compression_same_kind_max_duration`] \ No newline at end of file