The Observability Tier: Watching 12 Nodes with InfluxDB, Telegraf, and Loki

Twelve nodes, fifty-nine guests, hardware spanning a fanless Celeron to a Ryzen 9 5950X. That is not a fleet you watch by SSHing in and running htop. The observability tier turns the whole cluster into a few dashboards and a search box: metrics flowing into InfluxDB, logs pooling in Loki, and something loud enough to tell me a node fell over before I notice it myself. The guiding idea is that "is it healthy?" is really three different questions, and each one wants a different tool.

Three questions, three stores

• How is it performing? Time-series metrics (CPU, RAM, disk, network) in InfluxDB, collected by Telegraf.
• What actually happened? Logs in Loki, queried with LogQL.
• Is it up? Status checks in Uptime Kuma, driving the public status page.

Trying to answer all three with one tool is how you end up with a metrics system full of log spam, or a log system you are begging to compute averages. Keep them separate and each one stays good at its job.

Architecture

   each node/guest
   ┌───────────┐   metrics    ┌────────────┐
   │ Telegraf  │ ───────────▶ │  InfluxDB  │ ◀── Pulse (Proxmox-native view)
   └───────────┘              └─────┬──────┘
   ┌───────────┐   logs             │
   │ log agent │ ──────────▶ ┌──────▼─────┐        ┌───────────┐
   └───────────┘             │    Loki    │ ─────▶ │  Grafana  │  dashboards
                             └────────────┘        └───────────┘

   Uptime Kuma ─────▶ public status page
   watch-your-lan · myspeed · patchmon ── the long tail of niche signals

Telegraf and InfluxDB: the metrics spine

Telegraf is the collection agent; InfluxDB is the time-series store it writes to. Telegraf's input plugins read CPU, memory, disk, and network per host and push them out on an interval. It is a push model, so there is no central scraper to be the single thing that breaks.

# telegraf.conf (excerpt): what to collect, where to send it
[[inputs.cpu]]
  percpu = true
  totalcpu = true
[[inputs.mem]]
[[inputs.disk]]
[[inputs.net]]

[[outputs.influxdb_v2]]
  urls = ["http://influxdb.internal:8086"]
  token = "$INFLUX_TOKEN"
  organization = "homelab"
  bucket = "metrics"

InfluxDB performance lives and dies by series count, which is the product of your tag values. Tagging by node and guest (dimensions there are a few dozen of) stays cheap; tagging by anything unbounded, a request ID, an ephemeral PID, a timestamp, explodes the series count until queries crawl and the box swaps. The tags are chosen on purpose, never something that grows without limit.

Loki: logs without the Elasticsearch tax

Loki aggregates logs the cheap way: it indexes labels, not the full text of every line. That is what lets a tiny container be a legitimate log store for a cluster this size. It isn't building a giant inverted index of every word, it is bucketing streams by label and letting you grep within them via LogQL. That economy depends on keeping the label set small, so labels stay to the handful of dimensions worth slicing by (host, job, service) and everything else lives in the log line, searched at query time. Add a high-cardinality label (a user ID, a session token, a container hash) and every unique value becomes its own stream, inverting the cost model you came to Loki to escape.

This is also where the tiers touch. The same DMZ and reverse-proxy logs that the isolation tier ships to Wazuh for detection get a copy in Loki for fast, ad-hoc "what happened at 2:14am" searching. Detection and forensics want different tools, and both get fed.

Pulse: Proxmox-native, no assembly required

Not everything should be hand-rolled from Telegraf inputs. Pulse is Proxmox-aware monitoring: node and guest state at a glance, without me writing exporters for every PVE metric I care about. It is the quick-look layer over the cluster; InfluxDB is the historical record underneath.

The long tail

The niche signals each earn a tiny container:

• watch-your-lan shows who and what is actually on the network, so a new device is something I find out about.
• myspeed tracks WAN throughput over time, so "the internet feels slow" becomes a graph instead of a vibe.
• patchmon reports which guests are behind on patches, which is observability pointed at my own hygiene.
• Uptime Kuma runs the up/down checks that drive the public status page (the same one the isolation-and-privacy tier publishes through its reverse proxy).

The up/down checks are deliberately kept off the stack they report on. If they depended on the same core infrastructure they are supposed to watch, the outage that matters most would be the one that takes the status page down with it, so decoupling them means "is the lab up?" can still be answered when the lab is not up.

Why it's designed this way

• Agents push; no central scraper. Telegraf sends to InfluxDB on its own schedule, so there is no one poller whose failure blinds the whole fleet.
• Separate stores for separate questions. Metrics, logs, and status don't share a backend, so each stays cheap and fast at the one thing it does.
• Tiny footprints on purpose. Observability should not consume the capacity it is meant to measure, so each collector gets the smallest container that does the job.
• Status is decoupled. The thing that says "we're down" does not live inside the thing that is down.

What's next

• Unify the dashboards in Grafana over both InfluxDB and Loki, so metrics and the logs that explain them sit on one pane.
• Alerting into the same webhook the inventory platform already uses, so a threshold breach and a posture regression arrive through one channel.
• Feed usage back into compliance. The posture scanner currently flags high RAM allocation with no usage data. InfluxDB is that usage data, so piping it back would turn a guess into a measurement and retire the finding.