Generate application metrics as a part of Google Cloud Operations suite configuration

[minherz]

Describe request or inquiry

Using Helm charts or kustomize it is possible to enable Online Boutique to generate o11y data such as traces or profiling information. Application logs are captured in the vanilla deployment automatically when Online Boutique is deployed on GKE.
To demonstrate the full specter of the o11y data the application should generate monitoring metrics as well. Despite the described behavior the current version (0.9.0) does not generate application level metrics.

The following metrics are recommended for generation:

• grpc statistics -- it is a "quick win" since it should be possible to capture grpc stats on the server side or the [client side] by leveraging ootb functionality of OTel SDK.
• application stats -- some custom stats that is relevant to operation of Online Boutique, for example the number of currency conversions or number of product details that have been reviewed

Monitoring should be initialized in each service that will generate metrics in addition to tracing. See the [example] for the reference.

It is recommended to use frontend and productcatalogservice for metric generation.

[client side] : https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc#example-NewClientHandler
[example]: https://github.com/GoogleCloudPlatform/golang-samples/blob/main/opentelemetry/instrumentation/app/setup.go

What purpose/environment will this feature serve?

Adding application level monitoring metrics allows to show all relevant services included into Google Cloud Operations suit and also completes the set of o11y data types supported in Google Cloud.