Add initial support to validate the mappings in system tests #2214
Conversation
|
/test |
internal/fields/validate.go
Outdated
|
|
||
| dataStreamName string | ||
|
|
||
| LocalSchema []FieldDefinition |
There was a problem hiding this comment.
New variable to hold just the Field Definitions from the local directory (package).
It is needed to check if a given mapping is related to a field definition with type array.
internal/fields/validate.go
Outdated
| @@ -295,6 +328,8 @@ func createValidatorForDirectoryAndPackageRoot(fieldsParentDir string, finder pa | |||
| } | |||
|
|
|||
| v.Schema = append(fields, v.Schema...) | |||
|
|
|||
| v.LocalSchema = fields | |||
There was a problem hiding this comment.
Added here for completeness, but it is not used in this validator.
Should we remove it from here?
There was a problem hiding this comment.
If not used I would say to remove it.
|
test integrations |
|
Created or updated PR in integrations repository to test this version. Check elastic/integrations#11828 |
There is a difference in the
{
"ecs_date": {
"path_match": [
"*.timestamp",
"*_timestamp",
"*.not_after",
"*.not_before",
"*.accessed",
"created",
"*.created",
"*.installed",
"*.creation_date",
"*.ctime",
"*.mtime",
"ingested",
"*.ingested",
"*.start",
"*.end"
],
"unmatch_mapping_type": "object",
"mapping": {
"type": "date"
}
}
},
{
"ecs_date": {
"path_match": [
"*.timestamp",
"*_timestamp",
"*.not_after",
"*.not_before",
"*.accessed",
"created",
"*.created",
"*.installed",
"*.creation_date",
"*.ctime",
"*.mtime",
"ingested",
"*.ingested",
"*.start",
"*.end",
"*.indicator.first_seen",
"*.indicator.last_seen",
"*.indicator.modified_at",
"*threat.enrichments.matched.occurred"
],
"unmatch_mapping_type": "object",
"mapping": {
"type": "date"
}
}
}Not sure if it could be applied some kind of exception @jsoriano |
|
For the
{
"_embedded_ecs-url_original_to_multifield": {
"path_match": "*.url.original",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "wildcard"
}
}
},
{
"ecs_path_match_wildcard_and_match_only_text": {
"path_match": [
"*.body.content",
"*url.full",
"*url.original"
],
"unmatch_mapping_type": "object",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "wildcard"
}
}
}, |
|
In the second build from integrations, In the first build, this package was not failing: |
|
The error related to This package fails with both 8.14.0 and 8.15.3 stack versions. Mapping created for this field in this package during tests: "process": {
"properties": {
"pid": {
"type": "float"
},
"title": {
"type": "keyword",
"fields": {
"text": {
"type": "match_only_text"
}
}
}
}
}EDIT: It looks like that currently (with fields validation), this is accepted if the definition is long (ECS) and the value has type float: elastic-package/internal/fields/validate.go Lines 1234 to 1247 in 8cc126a |
|
Triggered a new execution in integrations with the rest of packages: |
|
In the latest integrations build, there are some failures in network_traffic package. These look related to the same issue #2214 (comment) There are fields undefined under a
EDIT: Not sure how to detect this via the actual or preview mappings |
|
test integrations |
|
Created or updated PR in integrations repository to test this version. Check elastic/integrations#11828 |
|
Added an exception to validate mappings that define number type (long, float, double) in 5097aee This change is related to the issue found in the comment: #2214 (comment) |
|
Just updated the logic to show warnings when a field is defined as Tested with Examples:
Tested updating |
|
test integrations |
|
Created or updated PR in integrations repository to test this version. Check elastic/integrations#11828 |
| if v.specVersion.LessThan(semver3_0_1) { | ||
| if mappingParameter("type", actual) == "nested" { | ||
| logger.Warnf("Skip validation of nested object (spec version %s): %s", path, v.specVersion) | ||
| isNestedParent = true | ||
| } | ||
| } |
There was a problem hiding this comment.
Implementation before 3.0.1 ignored everything under nested objects, right? Couldn't we return nil directly here?
There was a problem hiding this comment.
Maybe it's not needed, but I didn't return here with nil to allow us to show warnings with all the missing definitions.
| "type": "keyword", | ||
| }, | ||
| }, | ||
| actual: map[string]any{ |
There was a problem hiding this comment.
Can this case happen? Than an actual mapping doesn't have a field that is in the preview?
There was a problem hiding this comment.
I was thinking that maybe this could happen since in system testing it is appended all the ECS definitions depending on the stack version or import_mappings.
Maybe there could be a case (e.g. with the ecs@mappings component template without import_mappings) where some field does not match any dynamic template but it is in the ECS.
Would that make sense @jsoriano ?
There was a problem hiding this comment.
I mean that in this actual mapping, the foo field should be also present, right?
internal/fields/mappings_test.go
Outdated
| }, | ||
| }, | ||
| expectedErrors: []string{ | ||
| `field "user" is undefined: missing definition for path`, |
There was a problem hiding this comment.
If the user is in the mapping, and is not external, it should be in the preview, right?
There was a problem hiding this comment.
That's right. As user is in the schema, it should have a mapping in the preview too. I'll fix this test case.
💚 Build Succeeded
History
cc @mrodm |
|
test integrations |
|
Created or updated PR in integrations repository to test this version. Check elastic/integrations#11828 |
Relates #2206
Add validation of the mappings when running system tests.
This process compares the mappings installed by Fleet (preview mappings) with the ones obtained after ingesting new documents into Elasticsearch.
This validation for the time being is behind an environment variable as a technical preview:
Assumptions considered while working on this PR: #2206 (comment)
As part of this PR:
Author's checklist
prometheusfailing since some fields depend on dynamic templates (not implemented yet)ti_anomali_logsdbandauth0_logsdb