Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add "longFieldMaxLength" config var #2193

Merged
merged 10 commits into from
Sep 3, 2021
Merged

Conversation

trentm
Copy link
Member

@trentm trentm commented Aug 13, 2021

elastic/apm#493 specifies a long_field_max_length config option we can use for span.context.db.statement. Currently this is hardcoded to 10000... bytes.

  • We'll also change to be 10000 unicode characters (rather than bytes). This is in line with the json-schema spec for "maxLength" and is now the apm spec for long_field_max_length. See the http-client PR for this part of the change.
  • This also deprecates the existing node.js-agent-only errorMessageMaxLength config option. It now defaults to the longFieldMaxLength value if not specified. Otherwise errorMessageMaxLength is still supported. (We should drop it in the "next major".)
  • This change also applies the truncation to the other fields mentioned in the long_field_max_length spec. This does mean that some existing usage out there might get truncation of some of these fields where they didn't before. That's debatable whether that constitutes a "breaking change". So far this PR takes the point of view that this is not a breaking change that would require either a separate config var (which means not conforming with the APM spec link above) or only applying truncation to these fields in a new major version later. Objections?

Fixes: #1921

Checklist

Before this the 10000 length truncation of the "query" field for
database spans was hardcoded.

Fixes: #1921
@trentm trentm self-assigned this Aug 13, 2021
@github-actions github-actions bot added the agent-nodejs Make available for APM Agents project planning. label Aug 13, 2021
@apmmachine
Copy link
Contributor

apmmachine commented Aug 13, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-09-02T21:49:33.761+0000

  • Duration: 19 min 33 sec

  • Commit: 055769b

Test stats 🧪

Test Results
Failed 0
Passed 20
Skipped 0
Total 20

Trends 🧪

Image of Build Times

Image of Tests

@trentm
Copy link
Member Author

trentm commented Aug 16, 2021

elastic/apm#488 is a discussion for a cross-agent config option name for this var. Let's hold until that discussion plays out.

@trentm
Copy link
Member Author

trentm commented Aug 23, 2021

I've started elastic/apm#493 for the spec addition for this field name: long_field_max_length.

@trentm trentm changed the title feat: Add "queryMaxLength" config var feat: Add "longFieldMaxLength" config var Aug 27, 2021
@trentm
Copy link
Member Author

trentm commented Sep 1, 2021

^^ Integration Tests failure is the current elastic/apm-integration-testing#1188 flake

@trentm
Copy link
Member Author

trentm commented Sep 1, 2021

Here is a small test script to demonstrate longFieldMaxLength working on span.context.db.statement:

const apm = require('./').start({
  longFieldMaxLength: 100000
})

const { Client } = require('@elastic/elasticsearch')
var client = new Client({
  node: 'http://localhost:9200',
  auth: { username: 'admin', password: 'changeme' }
})

const t1 = apm.startTransaction('t1')
// Build up a search body >10k in size.
aggs = {}
for (var i = 0; i < 1000; i++) {
  aggs[`aggs${i}`] = { "terms": { "field": "author" } }
}
client.search(
    {
      body: {
        size: 1,
        aggregations: aggs
      }
    },
    function (err, result) {
      t1.end()
    }
)

@trentm trentm marked this pull request as ready for review September 1, 2021 18:17
@trentm trentm requested a review from astorm September 1, 2021 18:17
Copy link
Contributor

@astorm astorm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks it does the thing the way the agent wants the thing done. Will approve once the package.json is updated to reflect what we want to ship.

docs/configuration.asciidoc Show resolved Hide resolved
package.json Outdated Show resolved Hide resolved
trentm added a commit to elastic/apm-nodejs-http-client that referenced this pull request Sep 2, 2021
- Truncate on number of JS chars (i.e. a count of UCS-2 chars), not bytes.
  This is faster, sufficient for ES storage concerns for large fields, and
  matches the json-schema spec for fields with a "maxLength".
- Remove truncateQueriesAt, add truncateLongFieldsAt which covers that
  and the other fields specified for long_field_max_length.
- Deprecate truncateErrorMessagesAt. If unspecified, the
  truncateLongFieldsAt value will be used.

BREAKING CHANGE.

Refs: elastic/apm-agent-nodejs#2193
@trentm trentm requested a review from astorm September 2, 2021 18:10
@trentm trentm merged commit ec719c2 into master Sep 3, 2021
@trentm trentm deleted the trentm/query-max-length branch September 3, 2021 22:32
dgieselaar pushed a commit to dgieselaar/apm-agent-nodejs that referenced this pull request Sep 10, 2021
…MaxLength` (elastic#2193)

This config option will truncate some transaction, span, and error fields
that can be longer at the configured number of JavaScript characters per
https://github.com/elastic/apm/blob/master/specs/agents/field-limits.md#long_field_max_length-configuration

Notably truncation is now based on a count of JavaScript *characters*
rather than in UTF-8 encoded bytes.

This also deprecates the errorMessageMaxLength config option. It now
defaults to the longFieldMaxLength value if not specified. Otherwise
errorMessageMaxLength is still supported.

Fixes: elastic#1921
astorm pushed a commit that referenced this pull request Oct 4, 2021
…MaxLength` (#2193)

This config option will truncate some transaction, span, and error fields
that can be longer at the configured number of JavaScript characters per
https://github.com/elastic/apm/blob/master/specs/agents/field-limits.md#long_field_max_length-configuration

Notably truncation is now based on a count of JavaScript *characters*
rather than in UTF-8 encoded bytes.

This also deprecates the errorMessageMaxLength config option. It now
defaults to the longFieldMaxLength value if not specified. Otherwise
errorMessageMaxLength is still supported.

Fixes: #1921
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
agent-nodejs Make available for APM Agents project planning.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Expose Max Db Statement Size as a Config Option
3 participants