validate JSON types/subtypes

[beniwohli]

as our default is code.custom, which is not yet part of the spec, we get a lot of warnings with this.

Related issues

Closes #1305

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-10-13T10:22:28.562+0000

• Duration: 29 min 13 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5059
Skipped	3461
Total	8520

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /test linters : Run the Python linters only.

• /test full : Run the full matrix of tests.

• /test benchmark : Run the APM Agent Python benchmarks tests.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[apmmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (68/68)	💚
Files	100.0% (229/229)	💚
Classes	100.0% (229/229)	💚
Lines	91.027% (18148/19937)	👍 0.006
Conditionals	74.544% (2820/3783)	👎 -0.011

[beniwohli]

Running

 grep "UserWarning:" log.txt | sed -n -e 's/^.*UserWarning: //p' | sort | uniq

on the pipeline log results in this list of span types/subtypes that fail the spec:

"app.internal" not marked as used by Python
"db.cassandra" not marked as used by Python
"db.dynamodb" not marked as used by Python
"db.elasticsearch" not marked as used by Python
"db.mongodb" not marked as used by Python
"db.mysql" not marked as used by Python
"db.postgresql" not marked as used by Python
"db.redis" not marked as used by Python
"db.sqlite" not marked as used by Python
"external.http" not marked as used by Python
"messaging.kafka" not marked as used by Python
"messaging.sns" not marked as used by Python
"messaging.sqs" not marked as used by Python
"storage.s3" not marked as used by Python
Span type "a" not found in JSON spec
Span type "aws" not found in JSON spec
Span type "cache" not found in JSON spec
Span type "code" not found in JSON spec
Span type "code.custom" not found in JSON spec
Span type "db.sql" not found in JSON spec
Span type "foo" not found in JSON spec
Span type "task" not found in JSON spec
Span type "test" not found in JSON spec
Span type "test_type" not found in JSON spec
Span type "type" not found in JSON spec
Span type "unknown" not found in JSON spec
Span type "x" not found in JSON spec
Span type "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx…" not found in JSON spec
Span type "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx�" not found in JSON spec
Span type "z" not found in JSON spec
Subtype "None" not allowed for span type "app"
Subtype "None" not allowed for span type "custom"
Subtype "graphql" not allowed for span type "external"
Subtype "postgres" not allowed for span type "db"
Subtype "pyodbc" not allowed for span type "db"

(the "not marked as used by Python" warnings are not really failures)

[basepi]

To be clear, this is validating but not failing? Should we fix up the test cases and fail on not found in JSON spec?

Edit: Nevermind, found it:

At first, only warnings are issued. At a later point, it should return the message as string
which will cause a validation error.