[system tests] Validate fields are documented via the mappings generated after ingesting docs comparing them to the preview mappings

[mrodm]

In order to try to be agnostic to the structure of the documents ingested to run the validation in system tests. It would be helpful to run validations comparing the mapping definitions instead.

These validations should be done with these two sets of mappings:

• Mappings and dynamic templates installed by Fleet before ingesting any doc.
  • These preview mappings can be retrieved using this simulated API:

    POST /_index_template/_simulate/<index_template_name>
    
    # Example
    POST /_index_template/_simulate/logs-microsoft_dhcp.log
    

• Mappings and dynamic templates that are present after ingesting the docs as part of the system tests.
  • These mappings can be retrieved using this API:

    GET /<data_stream_test>/_mapping/
    

Being available these two sets of mappings, it would be needed to validate whether or not each mapping present in the data stream (after ingesting docs) has a corresponding mapping in the preview.

It should be taken into account that:

• Some fields should be skipped (e.g. agent.*, host.*, ...) as it is done nowadays (link)
• Ignore for now those mappings that are not found in the preview (to be done in the next step).
  • To be done in a follow-up issue.
• TBD check if any mapping that does not exist in the preview is part of the ECS.

For now, the errors that could be raised as part of this validation should not make fail the validation process, therefore:

• Ignore at this step mappings not found in the preview, to be done in the next step.
• Enable via environment variable this mapping or show just just warning messages.

To be tested:

• Run these validations in stack 7.x
• Run these validations in stack 8.x
• Run these validations in input and integration packages.
• Run these validations in Stacks with LogsDB enabled (synthetics).

[mrodm]

Some assumptions made when adding this new validation based on mappings:

• Objects with type array are skipped.
  • These field definition do not generate any mapping.
  • Type array is checked using the local field definitions in the package.
    • Requires to load in memory also the field definitions from the data stream or input package.
  • These kind of definitions were allowed in spec versions <= 2.0.0 (link)
• Empty objects present in the actual mappings are skipped:

  "_tmp": {
    "type": "object"  
  }

• If a field is defined as a constant_keyword, just check/validate the value if both (preview and actual mappings) have "value" parameter.
• If an object contains dynamic: true:
  • skip the validation of those field mappings under a mapping like
    • should they be validated using the dynamic template definitions?

      "field": {
        "dynamic": true,
        "type": "object"
      }

  • The other field mappings must be checked as regular field mappings.

[mrodm]

While working on #2214 , there have been some issues found while testing with the packages in the integration repository that are not related to the dynamic templates. Support to validate mappings with the dynamic templates is going to be tackle in #2207

Issues found are detailed in #2214 (comment)

In summary, those issues are:

• Failures related to different mappings found in ECS
  • Due to the embedded ECS or the ecs@mappings component template
  • Some examples:
    • Add initial support to validate the mappings in system tests #2214 (comment)
    • Add initial support to validate the mappings in system tests #2214 (comment)
• Failures while validating mappings with type long, float or double Add initial support to validate the mappings in system tests #2214 (comment)
• Nested objects Add initial support to validate the mappings in system tests #2214 (comment)
  • Skip for spec versions <3.0.1
• Array of objects defined as group in the package Add initial support to validate the mappings in system tests #2214 (comment)