feat(logfwd): add log labels

[barrettj12]

This PR builds on the previous log forwarding PRs (#209, #252, #256, #267) by allowing the user to specify labels for logs forwarded from Pebble.

Labels will be defined in a new field labels inside the log target:

log-targets:
  tgt1:
    type: loki
    location: https://my.loki.server.com:3100/loki/api/v1/push
    labels:
      key1: val1
      key2: val2

Labels will support environment variable substitutions ($ENV_VAR), and these will be interpreted "dynamically" in the context of the relevant service.

Notes on the design

Labels can contain $ENVIRONMENT_VARIABLES, which will be substituted in the environment of the relevant service. However, the "service environment" could mean:

1. the environment map as defined in the plan
2. the actual environment in which the process is run

We decided to use (1) instead, as it is more explicit, can be "statically" evaluated just from the plan, and doesn't require extra data to be passed through from the service manager and cached.

Furthermore, anytime we change the labels, we first flush any logs currently in the buffer, so that these logs are sent with the correct (old) labels.

QA steps

Set up a simple logging service:

#!/bin/bash
while true; do
  echo "hello"
  sleep 1
done

Set up a plan using labels and environment variables:

services:
  svc1: &svc1
    command: /path/to/logsvc
    startup: enabled
    override: merge
    environment:
      OWNER: alice
      IP: 100.32.2.1
      PORT: 9090

  svc2:
    <<: *svc1
    environment:
      OWNER: bob
      IP: 100.32.2.5
      PORT: 3000

log-targets:
  tgt1:
    type: loki
    override: merge
    location: http://10.42.0.222:3100/loki/api/v1/push
    services: [all]
    labels:
      owner: user-$OWNER
      address: http://${IP}:${PORT}

Run Pebble:

PEBBLE_DEBUG=1 go run ./cmd/pebble run

and verify that the labels are being correctly interpreted.

Links

Jira card: JUJU-4643

[hpidcock]

Looks like a good change. Just need to consider the problem we discovered next week.

[benhoyt]

Few minor suggestions and a suggested fix for the SetLabels concurrency problem.

[hpidcock]

Changes look good, thanks for sticking with this.

[niemeyer]

Thanks for the updates. Some further notes on one unrelated change done, though:

[benhoyt]

I've gone through this once more and added a bunch of nit comments. Let's fix those up and get the PR description into shape, do final QA, and then ping me and we can try to merge this week!

[benhoyt]

Can we make this name more specific, like TestXyzRace with what it's actually testing a race in or for? And after that let's remove the redundant doc comment.

[barrettj12]

This is a pretty non-specific test, it's just designed to hammer the log gatherer and find any potential race conditions. It's now finding some artificial race conditions though. Needs a rewrite.

[barrettj12]

I rewrote the test so it passes. I'm still not sure what to call it though - do you have any suggestions?

[benhoyt]

Looks good! I'd like @hpidcock to give this one last look, but I'm ready to merge.