model: create a specification for valid event docs

[axw]

We currently have hand-written model types, whose Transform methods are responsible for setting fields according to our schema. The schema is defined by fields.yml, and is only partially covered by ECS.

In the Kibana APM app code there are Typescript type definitions for the document _source: https://github.com/elastic/kibana/tree/master/x-pack/plugins/apm/typings/es_schemas/raw. In theory we could generate these from our field definitions, and with some annotations to indicate which fields are "required" (expected to always be produced by APM Server) we could avoid having ?: on every property.

If we had a more complete model specification, we could use that to generate Typescript type definitions as well as the Go code for our own model Transform methods. Furthermore, in the future we might build on this to avoid the need for creating memory-heavy common.MapStr objects by generating code following an approach like described in #3565 (comment). The main point being that relying on code generation makes the change easier than if we continue hand writing everything.

Finally, with a spec we could produce a more efficient codec for local event storage (i.e. for tail-based sampling): #4120.

[axw]

One idea I have in mind is to have the model types defined in protobuf, perhaps with some extensions (custom options) to define the JSON mapping. We could then:

• generate the Go transformation code
• generate the Typescript type definitions
• directly implement a more efficient (protobuf) codec for local storage

It might be excessive, but we could then also consider using CUE protobuf extensions to define constraints (e.g. field zed is required when the metricset.name field value is "foo").

[dgieselaar]

@axw for what it's worth, with the Fields API we can generate a typescript type from the index mapping on the fly. But we won't be able to differentiate between required and non-required, or single and multiple value fields.

[axw]

@dgieselaar Nice. I suppose in theory we could add field metadata to fill in that gap. Field metadata is a limited resource though, so probably best to go with a statically generated definition still.