Anyone use Prometheus for their local system monitoring?

[codygman]

One downside of very custom systems like Nix and Devos allow you to build is sometimes weird things happen.

An idea I've had but not tried is to setup Prometheus to track metrics that could make sense of those weirdnesses.

Thoughts?

If useful, maybe something like this would be useful in Devos community config?

[blaggacao]

Thoughts?

I assume there is a systemd journald exporter.

I'm currently working on a kubenix overhaul to be included in devos, since I have a sense that prometheus might be most easily deployed into a kubernetes. Not sure though. Maybe might be just my current perspective since I'm working on it.

If useful, maybe something like this would be useful in Devos community config?

Is there a nixos config for a prometheus server? If so, this might be cheaply achievable already now.

[kraftnix]

I'm currently using Prometheus, Loki/Promtail and Grafana to monitor my home server and it has been working fairly well.

This particular monitoring stack is definitely more suited towards devos configurations which have a central server, although I have run it on my dev laptop before (but stopped as it seemed like overkill).

My code would need significant cleanup + abstraction before it is generally usable as it's quite tuned towards my specific setup at the moment.

[codygman]

Well consider me highly interested even for very messy code. At some point in the future I'll be forced to do this anyway to get to the bottom of hard to reproduce performance issues in my increasingly complicated devos based system.

It just got more complicated with the addition of using talon voice for coding since sometimes the voice recognition will spike the CPU. Then there's the problem of emacs sometimes spiking and... Well you get the idea 🙂

[blaggacao]

I'll definitly want to get in touch with you for talon. I might have a use case, as well, if it works. :-)

[kraftnix]

Well consider me highly interested even for very messy code.

I'll really have to detangle it first, it's very integrated with both systemd-nspawn containers (metrics stack runs inside nspawn containers and aggregates journal logs form other nspawn containers) as well as some ZFS dataset provisioning.

I have on my list to refactor it to mix and match what exporters + setup the machine should use but until I actually abstract that I don't think it will be very useful.

---

One side note which might be worth putting into another discussion. I do like the idea of adding something like this to the community branch, but I think we might need to have a discussion about how best to organise community profiles / pkgs / modules.

My current setup is a fork of devos where I merge in upstream changes when there is a new release (or changes I want to test). This means cherry-picking profiles / pkgs / modules from any community or other branch becomes a hassle to manage and keep up to date. I'm wondering if anyone else has a better workflow for using devos with community packages.

Presumably (i have not yet tried) you could pull in the community flake, and integrate the packages you want, however I have not yet tried to do so. If that is the case then we should probably add how to do it to the documentation, and perhaps encourage more community branches / forks which are focused on specific usecases. My metrics stack (and some of my other abstractable configurations) seem suited towards a server-style community branch, it would be interesting to have branches for server-specific configurations, android, kubenix etc.

My side note seems to have become longer than my reply, perhaps it's worth it's own discussion page.

[blaggacao]

This means cherry-picking profiles / pkgs / modules from any community or other branch becomes a hassle to manage and keep up to date. I'm wondering if anyone else has a better workflow for using devos with community packages.

With the current version of digga-lib, we can have a good sharing model for devos. The workflow here would be to add the modules you'd like as external modules from an external flake (even if it's the template's community beanch).

---

Now, community still lags the lighting speed of evolution (and processes that lack behind) in those last days.

[nrdxp]

We use it at work with bitte, but that's a completely different environment (nomad clusters). I haven't dug into how the monitoring is set up just yet, but there may be some useful Nix bits to steal.