feat(logfwd): support forwarding logs to Loki

[barrettj12]

Add a loki.Client type that encodes sends Pebble logs to Loki. This implements the logstate.logClient interface introduced in #256.

There have also been some changes to the logClient interface. Semantically, clients shouldn't flush inside the Write method - the flush should be up to the gatherer. Rename Write to AddLog for clarity, and add a NumBuffered method to help the gatherer determine when to flush.

QA steps

Deploy a Loki server for testing - this can be done with Juju. For convenience, deploy Grafana too, to view the logs.

juju bootstrap microk8s c
juju add-model cos
juju deploy loki-k8s --trust
juju deploy grafana-k8s --trust
juju integrate loki-k8s grafana-k8s:grafana-source

Wait for Loki and Grafana to reach a stable state.

Create a simple logging service:

#!/bin/bash
while true; do
  echo "hello"
  sleep 1
done

and a simple plan using this service:

services:
  svc1: &logsvc
    command: /path/to/logsvc
    startup: enabled
    override: merge
  svc2: *logsvc

log-targets:
  loki:
    type: loki
    location: http://[loki_ip]:3100/loki/api/v1/push
    override: merge
    services: [all]

[loki_ip] is the unit/app IP address from the deployed Loki in juju status.

Run Pebble:

go run ./cmd/pebble run

Get the admin password for Grafana using

juju run grafana-k8s/0 get-admin-password

and then visit [grafana_ip]:3000 in a browser, and log in with username admin and the obtained password.

If you go to the "Explore" tab and change the data source to Loki, you should be able to see logs forwarded from Pebble.

---

JUJU-4205

[benhoyt]

Nice work!

[flotter]

Mostly questions for my own understanding, and perhaps something you can find useful.

[benhoyt]

Looking good! Left a whole bunch of comments -- they're all minor though.

[tlm]

lgtm @barrettj12, just some comments for your consideration.

[niemeyer]

Gentlemen, it looks like some of the lines have been blurred with this PR. I was optimistic that we'd have a nice and sweet PR since this was just implementing the Loki client, but instead this went back into the behavior that we have agreed to over the past few weeks and changed it, which unfortunately makes me ask questions about things that we had agreements on instead of it being a happy walk.

Would it be possible to do a bit less in this PR, and implement a simpler Loki client using the APIs that we had already settled on? I realize that's probably not great news from your end, so maybe have a look at these comments with a cup of tee and let me know what I got wrong?

[barrettj12]

Just did a test pushing 2000 logs/sec at Pebble, with a fake Loki server always returning 429. It used about 10% of my laptop's CPU dealing with this, but the memory seemed to stay bounded at 0.1%. So I don't think we have an issue here with memory leaking.

[benhoyt]

A couple of very nitty comments, but looks good to me!

[hpidcock]

Feel like we are finally at the end of this long road. LGTM.

[niemeyer]

Almost there, Jordan. Thanks for your persistence.

[niemeyer]

Thanks, final points on that same issue, and we should be good to go:

[niemeyer]

Thank you! There's still a minor related to the on going conversations, but per note below. Let's please address it in a future PR.

[niemeyer]

We have the same issue of entries not being released here.

I'll go ahead and merge this as we've been here for a bit :), but let's please fix this at the next opportunity.