Loki Write: Remove Internal Labels

[matt-gp]

PR Description

This change is to add logic so that labels that start with __ are filtered out. Labels that start with __ are used internally for things such as metadata. If a user want to preserve the contents of one of these internal labels, then can do so using relabelling to remove the __.

This change also adds tests which check various different scenarios.

Which issue(s) this PR fixes

Fixes #4412

Notes to the Reviewer

The docs already state that labels with __ get removed, so I don't see the need to update these.

PR Checklist

• CHANGELOG.md updated
• Tests updated

[baurmatt]

Is there a specific reason why externalLabels starting with __ should be treated differently? To my knowledge, __ is core design concept of Prometheus and labels are always treat as Prometheus-internal: https://prometheus.io/docs/concepts/data_model/

I don't see a reason why loki.write should handle this and other way.

[matt-gp]

I thought because an externalLabel is something that a user would intentionally add, then we may want to include them. Happy to remove them though if needed.

[matt-gp]

Updated to remove __ labels from externalLabels as well

[kalleep]

After thinking about this a bit I think we should not do this directly in the component but here instead https://github.com/grafana/alloy/blob/main/internal/component/common/loki/client/batch.go#L116

Pretty sure we can just remove the without argument there and just skip any label that have __ as prefix