Monitoring usage metrics on the homelab
I have a very small and underpowered kubernetes cluster at home (see Homelab #1 and Homelab #2), and I'm always concerned that I'm overutilizing its resources when I want to add another service to it.
For example, sometimes my FreshRSS site is not responding. Is that because there is a network problem, or am I running out of memory, or something completely different.
One option could be to run Prometheus and Grafana, scraping metrics from each service at regular intervals. The problem with that is that I'm already running on a low-power system, so spending a significant chunk of the total resources only on MONITORING how many resources I'm using seems like a bad idea.
Instead, partly inspired by Markus' love for SQLite, I'm doing a much simpler thing:
- Run
kubectl top nodesandkubectl top podsevery 5 minutes with an entry incrontab. - Do a very simple parsing of the output
- Insert it into a SQLite database with a timestamp.
The database is a local file, stored on the same computer running k3s. When I want to look at how my system has been behaving recently, I can copy the whole database to my desktop and start digging into it.
No fuss, minimal overhead, no fancy dashboard. It's weird to do YAGNI on my homelab, because I don't NEED any of it.
Appendix:
The script I use to collect metrics:
#!/bin/bash
set -euo pipefail
# Database location
DB_PATH="${DB_PATH:-/var/lib/k8s-metrics/metrics.db}"
TIMESTAMP=$(date -u +"%Y-%m-%d %H:%M:%S")
# Full path to kubectl (k3s)
KUBECTL="${KUBECTL:-/usr/local/bin/kubectl}"
# Ensure database directory exists
mkdir -p "$(dirname "$DB_PATH")"
# Initialize database if it doesn't exist
sqlite3 "$DB_PATH" <<EOF
CREATE TABLE IF NOT EXISTS node_metrics (
timestamp TEXT NOT NULL,
node_name TEXT NOT NULL,
cpu_cores TEXT NOT NULL,
cpu_percent TEXT NOT NULL,
memory_bytes TEXT NOT NULL,
memory_percent TEXT NOT NULL,
PRIMARY KEY (timestamp, node_name)
);
CREATE TABLE IF NOT EXISTS pod_metrics (
timestamp TEXT NOT NULL,
namespace TEXT NOT NULL,
pod_name TEXT NOT NULL,
cpu_cores TEXT NOT NULL,
memory_bytes TEXT NOT NULL,
PRIMARY KEY (timestamp, namespace, pod_name)
);
CREATE INDEX IF NOT EXISTS idx_node_timestamp ON node_metrics(timestamp);
CREATE INDEX IF NOT EXISTS idx_pod_timestamp ON pod_metrics(timestamp);
EOF
# Collect node metrics
$KUBECTL top nodes --no-headers | while read -r name cpu cpu_pct memory memory_pct; do
sqlite3 "$DB_PATH" <<EOF
INSERT INTO node_metrics (timestamp, node_name, cpu_cores, cpu_percent, memory_bytes, memory_percent)
VALUES ('$TIMESTAMP', '$name', '$cpu', '$cpu_pct', '$memory', '$memory_pct');
EOF
done
# Collect pod metrics
$KUBECTL top pods -A --no-headers | while read -r namespace name cpu memory; do
sqlite3 "$DB_PATH" <<EOF
INSERT INTO pod_metrics (timestamp, namespace, pod_name, cpu_cores, memory_bytes)
VALUES ('$TIMESTAMP', '$namespace', '$name', '$cpu', '$memory');
EOF
done
echo "Metrics collected at $TIMESTAMP"
The crontab entry