Add CloudFoundry Resource Attributes

[KarstenSchnitter]

Proposal for #622:

Using the CloudFoundry documentation, provides a set of resource attribute describing CloudFoundry workloads. Both applications as well as system components are addressed. Helpful resources:

• https://github.com/cloudfoundry/loggregator-api#v2-envelope
• https://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html#VCAP-APPLICATION
• https://bosh.io/docs/jobs/#properties-spec
• https://docs.cloudfoundry.org/devguide/deploy-apps/streaming-logs.html

Changes

Introduces a resource attributes convention for CloudFoundry. Defines resource attributes to identify CloudFoundry applications and system components, enabling and improving OpenTelemetry support. The proposed attributes are selected to provide an as small as possible payload. The changes follow the proposal in #622.

Merge requirement checklist

• CONTRIBUTING.md guidelines followed.
• CHANGELOG.md updated for non-trivial changes.
• schema-next.yaml updated with changes to existing conventions.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: KarstenSchnitter / name: Karsten Schnitter (ad85e02, d20c70f, b5335d5, 9ce2881, 887ade0, e7ac8f2, 5da2eac, 402c93e, d77ec4b, d41832c, b2da4c9, fe5afa0, f4efa59, 1183038, 99a037d, 7a9b0b4, 7bd0436, 2313a54, 54d7997, 060b67e, 742e03b, 736bee4, 99b1cb8, 4808dfe, 2f991a8, d319458, ed7ff7a)
• ✅ login: joaopgrassi / name: Joao Grassi (d722e21)

[KarstenSchnitter]

This PR is meant as a proposal for discussion around #622. I am unsure about when and how to fulfil the merge requirements checklist. A changelog entry seems to be necessary. I do not know, how to verify the contribution guidelines. Locally a make check ran without issues.

[joaopgrassi]

@KarstenSchnitter

A changelog entry seems to be necessary

Yes, please add an entry to the CHANGELOG.md file. You can see an example here: #615

I do not know, how to verify the contribution guidelines. Locally a make check ran without issues.

I approved the CI run and some checks are failing. Not sure how the make check passed, but you can run them separately and they for sure should also fail locally.

[KarstenSchnitter]

@joaopgrassi I will take care of the issues and the changelog. Is there anything else?

[KarstenSchnitter]

@joaopgrassi I noticed some unmerged changes on my side. Hopefully, this resolves the checks.

[joaopgrassi]

I only looked at a high level yesterday, and I'm not experienced with CloudFoundry, so I will need to read a bit to understand the context so I can review it. I will try to do it in the upcoming days.

[FWinkler79]

I think we should review the naming of some of the attributes.
The OTEL colleagues should weigh in here, as they probably know best about the usual naming conventions.

[chombium]

Hi @FWinkler79,
thanks for the tips and the review. I'm one of the maintainers in the Cloud Foundry's logging and metrics team. We've discussed with @KarstenSchnitter and @ctlong and we also had the same doubts about the source_id, source_type and instance_id labels. We've added them as they are currently used in the Cloud Foundry's logging and metrics stack, but after checking the existing conventions, I've found the attributes-registry/source.md and general/attributes.md and I think that we should change them to source.id, source.type and instance.id.

There is one more open question to clear: In Cloud Foundry the source type label is only used for logs and not for metrics and traces. We are not sure what do do about it and if we need to define it in some specific way. @joaopgrassi Any ideas about this?

[KarstenSchnitter]

I provided a revised version with the fields renamed as suggested by @FWinkler79. Furthermore, I added some clarification on the source_type, especially the requirement level. For the three use-cases in #622 I added a comment, how to obtain the appropriate value for use-case 1, where the value needs to be set to a static value. This is the reason, why I think source.typeshould be part of this convention, but only apply to logs as now reflected in the requirement level. The branch to be merged was rebased on main in the process.

[FWinkler79]

Thanks for your reply and changes.

@chombium:

There is one more open question to clear: In Cloud Foundry the source type label is only used for logs and not for metrics and traces. We are not sure what do do about it and if we need to define it in some specific way. @joaopgrassi Any ideas about this?

I am not 100% sure, but I recently updated our internal version of the OTEL SemConv to the latest version and noticed that the attributes for different signal types are now (rather) cleanly separated into log-attributes, trace-attributes, etc.
Not sure the documentation already properly reflects that, but at least the YAML specifications do.
E.g. you can find the entire yaml-based model files here. There you find folders for logs, metrics, traces and recource attributes.
To avoid data duplication the common "base attributes" that are shared between signal types are now maintained under the "registry" folder. It's not entirely cleaned up yet, but I think that's the intention.

For example, in the registry folder you can find http.yaml, in the trace folder you also find http.yaml. The trace-related HTTP attributes simply reference HTTP attributes defined in the file under registry.

In a similar way, I think you could define log-specific attributes.

[chombium]

Thanks for the prompt change @KarstenSchnitter and for the suggestions and digging into the definitions @FWinkler79. @joaopgrassi Is there something else to be done apart from the suggestions about the source type parameter?

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[KarstenSchnitter]

@chombium , @ctlong, can you provide a status of your deliberations? Maybe we can agree on an initial convention and improve over time?

[joaopgrassi]

Hi @KarstenSchnitter !

We changed how the CHANGELOG.md is managed. Please take a look at https://github.com/open-telemetry/semantic-conventions/blob/main/CONTRIBUTING.md#adding-a-changelog-entry to see what needs to be done. Sorry for the disruption.

[FWinkler79]

@KarstenSchnitter I think you still wanted to add something here, right?

[KarstenSchnitter]

I think, I have addressed all open questions, but need some feedback from @lmolkova before marking them as resolved. If the current state is acceptable, there is nothing to be changed from my side.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[KarstenSchnitter]

@joaopgrassi and @lmolkova is there a way we can make progress here? The PR is approved by subject matter experts from the Cloud Foundry community.

[joaopgrassi]

I will try and take a look at it again in the upcoming days.

[joaopgrassi]

Please take a look at the checks failing. Also update the issue templates by running make generate-gh-issue-templates and make fix to update any left over tables.

One general ask I have to the group is: Resource attributes must be immutable - that means a resource is created during the app start up with say CloudFoundry attributes. Once the app is up and running, these attributes cannot change. I wanted to make this clear so you as the experts do the exercise in seeing if there's any such attributes introduced here that are affected by this constraint.

Another fact is that there's effort currently in a new group for Resource and Entities. The reason I'm bringing it up is since this PR is adding resource attributes I'm not sure if we want to block it. Maybe we allow it to be part of the registry as a first step.

@jsuereth could you please share your thoughts here?

[KarstenSchnitter]

I took care of the prefixes and hopefully all validation errors. I can confirm, that all resource attributes are immutable. Once an application instance is started, all its processes will have uniquely and immutably defined resource attributes of the proposed set.

[KarstenSchnitter]

There is an issue on the main branch with the Github link to the registry, which has a 404 error:

https://github.com/open-telemetry/semantic-conventions/blob/b855a36cf4b8a3c5461ad16fa4ace0c615d11c2a/CONTRIBUTING.md?plain=1#L225C15-L225C94

This fails the tests, but is not caused by this PR.

[KarstenSchnitter]

The latest merge should fix the issue with the main branch.