exporters/prometheus: improve error handling

[dashpole]

Problem Statement

The prometheus exporter currently uses otel.Handle() for all errors encountered during translation. The prometheus client has other ways of handling errors that give users more flexibility, and are more aligned with prometheus best-practices.

Proposed Solution

For errors that involve dropping data, we should use NewInvalidMetric https://github.com/prometheus/client_golang/blob/8dfb3558355027a1806ce87370bdad604ca2cef6/prometheus/metric.go#L138

This defaults to returning an error on the http request to the metrics endpoint, but can be configured to ignore errors, or to panic on error, and also logs the error. See https://github.com/prometheus/client_golang/blob/8dfb3558355027a1806ce87370bdad604ca2cef6/prometheus/promhttp/http.go#L372 in HandlerOpts.

Alternatives

Do nothing

Prior Art

See examples in the prometheus process collector: https://github.com/prometheus/client_golang/blob/8dfb3558355027a1806ce87370bdad604ca2cef6/prometheus/process_collector.go#L155

Additional Context

Came up during the discussion of error handling during translation (e.g. an invalid label): https://app.slack.com/client/T08PSQ7BQ/C01LSCJBXDZ?cdn_fallback=1

[dashpole]

cc @ywwg @ArthurSens @bwplotka

[adity1raut]

/assign

[pellared]

@adity1raut, please first finish working on the other issues that are assigned to you.

[tongoss]

@MrAlias , @pellared , @dashpole :

Can I work on this issue?

[dashpole]

Sure. Let me know if you have any questions.