Handling huge tracing specs #453

felixbarny · 2021-06-21T14:18:12Z

Preview

Closes #449

POC implementation for span compression: https://github.com/felixbarny/apm-agent-java/tree/compressed-spans

apmmachine · 2021-06-21T14:21:30Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-07-19T11:57:56.131+0000
Duration: 3 min 53 sec
Commit: 6821501

Trends 🧪

AlexanderWert

Great Spec so far!

General comment: How about keeping this Spec in one file to avoid getting lost in too many Spec files in general?

specs/agents/tracing-spans-dropped-stats.md

specs/agents/tracing-spans-drop-fast.md

specs/agents/tracing-spans-compress.md

specs/agents/tracing-spans-handling-huge-traces.md

felixbarny

How about keeping this Spec in one file to avoid getting lost in too many Spec files in general?

There are really multiple specs for one theme in a single PR. I'd like to separate different specs in different files. I'd expect the main navigation not to be the GitHub file viewer but the README.md. If it makes things easier, we could even add a breadcrumb to the header of each spec.

specs/agents/tracing-spans-compress.md

specs/agents/tracing-spans-dropped-stats.md

Co-authored-by: Alexander Wert <[email protected]>

specs/agents/tracing-spans-compress.md

Co-authored-by: Alexander Wert <[email protected]>

specs/agents/tracing-spans-compress.md

specs/agents/tracing-spans-dropped-stats.md

Co-authored-by: Colton Myers <[email protected]>

SergeyKleyman

It looks very good for just a few minor comments.

specs/agents/tracing-spans-handling-huge-traces.md

specs/agents/tracing-spans-limit.md

specs/agents/tracing-spans-dropped-stats.md

specs/agents/tracing-spans-compress.md

Co-authored-by: Sergey Kleyman <[email protected]>

specs/agents/tracing-spans-compress.md

specs/agents/tracing-spans-limit.md

specs/agents/tracing-spans-compress.md

specs/agents/tracing-spans-dropped-stats.md

specs/agents/tracing-spans-compress.md

specs/agents/tracing-spans-drop-fast-exit.md

specs/agents/tracing-spans-dropped-stats.md

eyalkoren

LGTM.
Suggested an alternative wording, feel free to ignore

specs/agents/tracing-spans-drop-fast-exit.md

Co-authored-by: Alexander Wert <[email protected]>

SergeyKleyman · 2021-07-19T04:47:50Z

... and exact_match: The only usage we discussed is for single trace view to give user visual cues about multiple spans compressed into just one

@estolfo As we discussed it actually might be useful to index compression_strategy (a new property that replaced exact_match)

…essed-spans

Co-authored-by: eyalkoren <[email protected]>

specs/agents/tracing-spans-destination.md

specs/agents/tracing-spans.md

apmmachine · 2021-07-19T12:01:12Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-07-19T11:57:50.252+0000
Duration: 3 min 1 sec
Commit: 6821501

Trends 🧪

* First draft of handling huge tracing specs * Apply suggestions from code review Co-authored-by: Alexander Wert <[email protected]> * Implement suggestions * Update specs/agents/tracing-spans-compress.md Co-authored-by: Alexander Wert <[email protected]> * Pseudo code for how the strategies work in combination * Add composite.exact_match flag * Apply suggestions from code review Co-authored-by: Colton Myers <[email protected]> * Add breadcrumbs * Add missing table of contents link to AWS tracing spec file * Some clarifications for the destination APIs (elastic#452) * Add limit to dropped_spans_stats * Add implementation section to transaction_max_spans * Move exit span definition from destination spec to span spec * Add exit_span_min_duration spec * Apply suggestions from code review Co-authored-by: Sergey Kleyman <[email protected]> * Fix links, add clarification to max duration * Dropping fast spans requires stats * Rework transaction_max_spans implementation logic * Improve transaction_max_spans: no CAS * Apply suggestions from code review Co-authored-by: Sergey Kleyman <[email protected]> * Update specs/agents/tracing-spans-compress.md * Update specs/agents/tracing-spans-compress.md * Update specs/agents/tracing-spans-handling-huge-traces.md Co-authored-by: Trent Mick <[email protected]> * Renamed same_kind_compression_max_duration config option to span_compression_same_kind_max_duration * Added span_compression_same_kind_max_duration config option * Added span_compression_enabled config option * Update specs/agents/tracing-spans-compress.md Co-authored-by: eyalkoren <[email protected]> * Changed end to sum.us in composite sub-object * Replaced exact_match bool with compression_strategy enum * Update specs/agents/tracing-spans-compress.md Co-authored-by: eyalkoren <[email protected]> * Added outcome requirement to eligible for compression * Added outcome requirement to eligible for compression PART 2 Updated isCompressionEligible() pseudo-code * Added links from tracing-spans.md to tracing-spans-compress.md * Fixed missing isSameKind check in tryToCompressComposite() * Update specs/agents/tracing-spans-drop-fast-exit.md Co-authored-by: Alexander Wert <[email protected]> * Update specs/agents/tracing-spans-compress.md Co-authored-by: Alexander Wert <[email protected]> * Update specs/agents/tracing-spans-drop-fast-exit.md Co-authored-by: Alexander Wert <[email protected]> * Update specs/agents/tracing-spans-compress.md Co-authored-by: Alexander Wert <[email protected]> * Update specs/agents/tracing-spans-compress.md Co-authored-by: Alexander Wert <[email protected]> * Update specs/agents/tracing-spans-compress.md Co-authored-by: Alexander Wert <[email protected]> * Removed "Exit span API" requirement from tracing-spans.md * Update specs/agents/tracing-spans-drop-fast-exit.md Co-authored-by: eyalkoren <[email protected]> * reafctored file structure for handling huge traces * Update specs/agents/tracing-spans-destination.md * Update specs/agents/tracing-spans.md * Update specs/agents/tracing-spans.md Co-authored-by: Alexander Wert <[email protected]> Co-authored-by: Colton Myers <[email protected]> Co-authored-by: Trent Mick <[email protected]> Co-authored-by: eyalkoren <[email protected]> Co-authored-by: Sergey Kleyman <[email protected]> Co-authored-by: Trent Mick <[email protected]> Co-authored-by: Alexander Wert <[email protected]>

First draft of handling huge tracing specs

2bd13cf

felixbarny requested review from SergeyKleyman and AlexanderWert June 21, 2021 14:25

AlexanderWert requested changes Jun 22, 2021

View reviewed changes

felixbarny commented Jun 22, 2021

View reviewed changes

specs/agents/tracing-spans-compress.md Outdated Show resolved Hide resolved

specs/agents/tracing-spans-compress.md Outdated Show resolved Hide resolved

specs/agents/tracing-spans-dropped-stats.md Outdated Show resolved Hide resolved

felixbarny and others added 2 commits June 22, 2021 10:56

Apply suggestions from code review

72d384d

Co-authored-by: Alexander Wert <[email protected]>

Implement suggestions

411f529

AlexanderWert reviewed Jun 22, 2021

View reviewed changes

specs/agents/tracing-spans-compress.md Outdated Show resolved Hide resolved

felixbarny and others added 3 commits June 22, 2021 11:51

Update specs/agents/tracing-spans-compress.md

fd3d879

Co-authored-by: Alexander Wert <[email protected]>

Pseudo code for how the strategies work in combination

42ad300

Add composite.exact_match flag

ae09511

basepi reviewed Jun 23, 2021

View reviewed changes

mikker approved these changes Jun 23, 2021

View reviewed changes

felixbarny and others added 7 commits June 24, 2021 08:09

Apply suggestions from code review

db17364

Co-authored-by: Colton Myers <[email protected]>

Add breadcrumbs

a81d78f

Add missing table of contents link to AWS tracing spec file

af969da

Some clarifications for the destination APIs (elastic#452)

f5c010a

Add limit to dropped_spans_stats

5916a63

Add implementation section to transaction_max_spans

ccf4349

Merge remote-tracking branch 'origin/master' into compressed-spans

9790529

SergeyKleyman approved these changes Jul 5, 2021

View reviewed changes

felixbarny and others added 6 commits July 5, 2021 16:08

Move exit span definition from destination spec to span spec

b318ae6

Add exit_span_min_duration spec

7ab424b

Apply suggestions from code review

bcd4a6d

Co-authored-by: Sergey Kleyman <[email protected]>

Fix links, add clarification to max duration

834ac8b

Dropping fast spans requires stats

42663a2

Rework transaction_max_spans implementation logic

5828651

SergeyKleyman reviewed Jul 6, 2021

View reviewed changes

specs/agents/tracing-spans-compress.md Outdated Show resolved Hide resolved

SergeyKleyman reviewed Jul 6, 2021

View reviewed changes

specs/agents/tracing-spans-limit.md Outdated Show resolved Hide resolved

SergeyKleyman requested review from a team as code owners July 15, 2021 09:00

estolfo approved these changes Jul 15, 2021

View reviewed changes

basepi approved these changes Jul 15, 2021

View reviewed changes

specs/agents/tracing-spans-compress.md Outdated Show resolved Hide resolved

specs/agents/tracing-spans-dropped-stats.md Outdated Show resolved Hide resolved

AlexanderWert approved these changes Jul 16, 2021

View reviewed changes

eyalkoren approved these changes Jul 18, 2021

View reviewed changes

specs/agents/tracing-spans-drop-fast-exit.md Outdated Show resolved Hide resolved

eyalkoren mentioned this pull request Jul 18, 2021

Spans not shown in the UI when their parent is missing elastic/apm-agent-nodejs#2160

Open

SergeyKleyman and others added 6 commits July 19, 2021 06:53

Update specs/agents/tracing-spans-drop-fast-exit.md

4df5afb

Co-authored-by: Alexander Wert <[email protected]>

Update specs/agents/tracing-spans-compress.md

0be6c90

Co-authored-by: Alexander Wert <[email protected]>

Update specs/agents/tracing-spans-drop-fast-exit.md

44c3936

Co-authored-by: Alexander Wert <[email protected]>

Update specs/agents/tracing-spans-compress.md

182d610

Co-authored-by: Alexander Wert <[email protected]>

Update specs/agents/tracing-spans-compress.md

a7d728b

Co-authored-by: Alexander Wert <[email protected]>

Update specs/agents/tracing-spans-compress.md

6b36436

Co-authored-by: Alexander Wert <[email protected]>

SergeyKleyman and others added 5 commits July 19, 2021 08:02

Removed "Exit span API" requirement from tracing-spans.md

2a54365

Merge remote-tracking branch 'felixbarny/compressed-spans' into compr…

d3a4453

…essed-spans

Update specs/agents/tracing-spans-drop-fast-exit.md

990463e

Co-authored-by: eyalkoren <[email protected]>

reafctored file structure for handling huge traces

a8e1e91

Merge commit 'b338fe9e1539180b05ce57ac0cfb8f3c18aa9b88'

916d1fa

AlexanderWert approved these changes Jul 19, 2021

View reviewed changes

specs/agents/tracing-spans-destination.md Outdated Show resolved Hide resolved

specs/agents/tracing-spans.md Outdated Show resolved Hide resolved

specs/agents/tracing-spans.md Outdated Show resolved Hide resolved

AlexanderWert added 3 commits July 19, 2021 13:57

Update specs/agents/tracing-spans-destination.md

48b08c9

Update specs/agents/tracing-spans.md

971c96f

Update specs/agents/tracing-spans.md

6821501

AlexanderWert enabled auto-merge (squash) July 19, 2021 11:58

AlexanderWert merged commit e528576 into elastic:master Jul 19, 2021

trentm mentioned this pull request Apr 25, 2022

feat: implement isCompressionEligable method elastic/apm-agent-nodejs#2656

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handling huge tracing specs #453

Handling huge tracing specs #453

felixbarny commented Jun 21, 2021 •

edited

Loading

apmmachine commented Jun 21, 2021 •

edited

Loading

Build stats

Trends 🧪

AlexanderWert left a comment

felixbarny left a comment

SergeyKleyman left a comment

eyalkoren left a comment

SergeyKleyman commented Jul 19, 2021 •

edited

Loading

apmmachine commented Jul 19, 2021

Build stats

Trends 🧪

Handling huge tracing specs #453

Handling huge tracing specs #453

Conversation

felixbarny commented Jun 21, 2021 • edited Loading

apmmachine commented Jun 21, 2021 • edited Loading

💚 Build Succeeded

Build stats

Trends 🧪

AlexanderWert left a comment

Choose a reason for hiding this comment

felixbarny left a comment

Choose a reason for hiding this comment

SergeyKleyman left a comment

Choose a reason for hiding this comment

eyalkoren left a comment

Choose a reason for hiding this comment

SergeyKleyman commented Jul 19, 2021 • edited Loading

apmmachine commented Jul 19, 2021

💚 Build Succeeded

Build stats

Trends 🧪

felixbarny commented Jun 21, 2021 •

edited

Loading

apmmachine commented Jun 21, 2021 •

edited

Loading

SergeyKleyman commented Jul 19, 2021 •

edited

Loading