ArchiveBox-Telemetrie: Exporteur + Telegraf-Container fuer Knoten ohne sudo
archibald hat kein sudo und kein linger, aber Docker-Gruppe. Muster wie bei der llama-swap-Telemetrie auf dem ROG: ein Nutzer-Cron fuehrt den Exporteur aus, Telegraf laeuft als Container daneben und pusht per Remote Write zum Collector im Mesh. - scripts/archivebox-telemetrie.py: liest die ArchiveBox-Logs und die SQLite-DB IM Container (Host-User mm darf index.sqlite3 nicht oeffnen, Container-User 911 schon) und schreibt eine Prometheus-Exposition. Kein node-Label -- das setzt Telegraf per global_tags. - scripts/install-archivebox-telemetrie.sh: installiert Exporteur, Cron-Eintrag (jede Minute) und den Container, mit --dry-run. - stacks/telegraf-container/: telegraf 1.39, network_mode host, inputs.file (data_format prometheus), outputs.prometheus_remote_write.
This commit is contained in:
kalina committed
1 parent
235cbc68dc
commit
6b169c77e6
4 files changed
+376
No files matched your search
Executable
+210
@@ -0,0 +1,210 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""ArchiveBox-Runner-Telemetrie fuer den Knoten archibald.
|
||||
|
||||
Laeuft als Nutzer `mm` per cron (jede Minute) und schreibt Prometheus-Exposition
|
||||
in eine .prom-Datei, die eine Telegraf-Container-Instanz liest und per
|
||||
Remote Write an den Collector (imac) schickt.
|
||||
|
||||
Warum ueberhaupt ein Exporteur: ArchiveBox' eigener Healthcheck prueft nur den
|
||||
Webserver, nicht den Runner. Am 03.10.2026 blieb der Runner sieben Tage in einer
|
||||
Neustart-Schleife (uid-Mismatch im Archivbaum, PermissionError beim mkdir des
|
||||
Crawl-Verzeichnisses), waehrend Healthcheck und API gruen waren und jeder neue
|
||||
Snapshot dauerhaft `queued` blieb.
|
||||
|
||||
Warum die Werte aus dem Container gelesen werden: der Host-Nutzer `mm` darf
|
||||
index.sqlite3 (uid 911) nicht oeffnen; `docker exec` laeuft als Container-User.
|
||||
Die Logdateien dagegen liegen im Host-Dateisystem und sind direkt lesbar.
|
||||
|
||||
Bewusst KEIN `node`-Label in der Ausgabe -- das setzt Telegraf per
|
||||
[global_tags] (sonst entstehen zwei Serien derselben Identitaet).
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
ARCHIVEBOX_DIR = "/home/mm/archivebox"
|
||||
OUT_DIR = "/home/mm/abx-telemetrie"
|
||||
OUT_FILE = os.path.join(OUT_DIR, "abx.prom")
|
||||
LOG_FILE = os.path.join(OUT_DIR, "exporteur.log")
|
||||
CONTAINER = "archivebox-archivebox-1"
|
||||
MAX_LOG_BYTES = 512 * 1024
|
||||
|
||||
LOGS = os.path.join(ARCHIVEBOX_DIR, "data", "logs")
|
||||
SUPERVISORD_LOG = os.path.join(LOGS, "supervisord.log")
|
||||
ERRORS_LOG = os.path.join(LOGS, "errors.log")
|
||||
|
||||
# Im Container: Status-Verteilung + Alter des aeltesten offenen Snapshots/Crawls.
|
||||
DB_QUERY = r"""
|
||||
import json, sqlite3, time
|
||||
from datetime import datetime, timezone
|
||||
c = sqlite3.connect("file:/data/index.sqlite3?mode=ro", uri=True)
|
||||
now = time.time()
|
||||
|
||||
def age_of(value):
|
||||
# ArchiveBox (Django, USE_TZ) speichert created_at als TEXT in UTC.
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
dt = datetime.fromisoformat(value.replace(" ", "T"))
|
||||
except ValueError:
|
||||
return None
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=timezone.utc)
|
||||
else:
|
||||
dt = value
|
||||
return now - dt.timestamp()
|
||||
|
||||
out = {}
|
||||
out["snapshots"] = {str(s): int(n) for s, n in c.execute(
|
||||
"select status, count(*) from core_snapshot group by 1")}
|
||||
out["snapshots_total"] = int(c.execute(
|
||||
"select count(*) from core_snapshot").fetchone()[0])
|
||||
row = c.execute("select min(created_at) from core_snapshot where status!='sealed'").fetchone()
|
||||
out["oldest_open_snapshot_age"] = age_of(row[0] if row else None)
|
||||
# Alarmsignal: ein Snapshot im Zustand `queued` ist NICHT abgeholt. Ein
|
||||
# `started`-Snapshot darf legitim lange laufen (Video-Crawl, gemessen > 1 h) --
|
||||
# deshalb getrennt fuehren und `queued` explizit als 0 ausgeben, wenn nichts
|
||||
# wartet: eine fehlende Serie waere von "keine Warteschlange" nicht zu
|
||||
# unterscheiden.
|
||||
row = c.execute("select min(created_at) from core_snapshot where status='queued'").fetchone()
|
||||
out["oldest_queued_snapshot_age"] = age_of(row[0] if row else None) or 0
|
||||
out["crawls"] = {str(s): int(n) for s, n in c.execute(
|
||||
"select status, count(*) from crawls_crawl group by 1")}
|
||||
row = c.execute("select min(created_at) from crawls_crawl where status!='sealed'").fetchone()
|
||||
out["oldest_open_crawl_age"] = age_of(row[0] if row else None)
|
||||
print("JSON:" + json.dumps(out))
|
||||
"""
|
||||
|
||||
|
||||
def sh(args, timeout=20):
|
||||
try:
|
||||
p = subprocess.run(args, capture_output=True, text=True, timeout=timeout)
|
||||
return p.returncode, p.stdout, p.stderr
|
||||
except subprocess.TimeoutExpired:
|
||||
return 124, "", "timeout"
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return 1, "", str(exc)
|
||||
|
||||
|
||||
def line_count(path):
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
return sum(1 for _ in fh)
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def stat_or_none(path):
|
||||
try:
|
||||
st = os.stat(path)
|
||||
return st.st_size, st.st_mtime
|
||||
except OSError:
|
||||
return None, None
|
||||
|
||||
|
||||
def main():
|
||||
now = time.time()
|
||||
metrics = [] # (name, wert) ohne Labels
|
||||
labeled = [] # (name, labels-dict, wert)
|
||||
notes = []
|
||||
|
||||
# --- Logdateien (Host-Sicht) ---
|
||||
size, mtime = stat_or_none(ERRORS_LOG)
|
||||
if size is not None:
|
||||
metrics.append(("archivebox_runner_errors_log_bytes", size))
|
||||
lines = line_count(ERRORS_LOG)
|
||||
if lines is not None:
|
||||
metrics.append(("archivebox_runner_errors_log_lines", lines))
|
||||
metrics.append(("archivebox_runner_errors_log_age_seconds", int(now - mtime)))
|
||||
|
||||
spawns = None
|
||||
try:
|
||||
with open(SUPERVISORD_LOG, "r", errors="replace") as fh:
|
||||
spawns = fh.read().count("spawned: 'worker_runner'")
|
||||
except OSError:
|
||||
notes.append("supervisord.log nicht lesbar")
|
||||
if spawns is not None:
|
||||
metrics.append(("archivebox_runner_spawns_total", spawns))
|
||||
|
||||
# --- Container: laeuft er, laeuft der Runner, was haengt in der Queue ---
|
||||
rc, out, err = sh(["docker", "inspect", "-f", "{{.State.Running}}", CONTAINER], timeout=15)
|
||||
container_up = 1 if (rc == 0 and out.strip() == "true") else 0
|
||||
metrics.append(("archivebox_container_up", container_up))
|
||||
|
||||
runner_up = 0
|
||||
if container_up:
|
||||
rc, out, _ = sh(["docker", "exec", CONTAINER, "sh", "-c",
|
||||
"pgrep -f 'archivebox run' >/dev/null && echo 1 || echo 0"], timeout=20)
|
||||
runner_up = 1 if out.strip().endswith("1") else 0
|
||||
metrics.append(("archivebox_runner_up", runner_up))
|
||||
|
||||
if container_up:
|
||||
rc, out, err = sh(["docker", "exec", CONTAINER, "python3", "-c", DB_QUERY], timeout=30)
|
||||
payload = None
|
||||
for line in out.splitlines():
|
||||
if line.startswith("JSON:"):
|
||||
import json
|
||||
payload = json.loads(line[5:])
|
||||
if payload is None:
|
||||
notes.append("DB-Abfrage fehlgeschlagen: %s" % (err.strip()[:120] or rc))
|
||||
else:
|
||||
metrics.append(("archivebox_snapshots_total", payload["snapshots_total"]))
|
||||
for status, n in sorted(payload["snapshots"].items()):
|
||||
labeled.append(("archivebox_snapshots", {"status": status}, n))
|
||||
for status, n in sorted(payload["crawls"].items()):
|
||||
labeled.append(("archivebox_crawls", {"status": status}, n))
|
||||
age = payload.get("oldest_open_snapshot_age")
|
||||
if age is not None:
|
||||
metrics.append(("archivebox_oldest_open_snapshot_age_seconds", int(age)))
|
||||
metrics.append(("archivebox_oldest_queued_snapshot_age_seconds",
|
||||
int(payload.get("oldest_queued_snapshot_age") or 0)))
|
||||
age = payload.get("oldest_open_crawl_age")
|
||||
if age is not None:
|
||||
metrics.append(("archivebox_oldest_open_crawl_age_seconds", int(age)))
|
||||
else:
|
||||
notes.append("Container laeuft nicht")
|
||||
|
||||
metrics.append(("archivebox_telemetrie_timestamp_seconds", int(now)))
|
||||
|
||||
# --- Ausgabe (atomar) ---
|
||||
os.makedirs(OUT_DIR, mode=0o755, exist_ok=True)
|
||||
buf = ["# HELP archivebox_telemetrie_timestamp_seconds Zeitpunkt der Erhebung.",
|
||||
"# TYPE archivebox_telemetrie_timestamp_seconds gauge",
|
||||
"# TYPE archivebox_container_up gauge",
|
||||
"# TYPE archivebox_runner_up gauge",
|
||||
"# TYPE archivebox_runner_spawns_total counter",
|
||||
"# TYPE archivebox_snapshots gauge",
|
||||
"# TYPE archivebox_crawls gauge"]
|
||||
for name, val in metrics:
|
||||
buf.append("%s %s" % (name, val))
|
||||
for name, labels, val in labeled:
|
||||
lbl = ",".join('%s="%s"' % (k, v) for k, v in sorted(labels.items()))
|
||||
buf.append("%s{%s} %s" % (name, lbl, val))
|
||||
tmp = OUT_FILE + ".tmp"
|
||||
with open(tmp, "w") as fh:
|
||||
fh.write("\n".join(buf) + "\n")
|
||||
os.chmod(tmp, 0o644)
|
||||
os.replace(tmp, OUT_FILE)
|
||||
|
||||
# --- Laufprotokoll (gedeckelt) ---
|
||||
try:
|
||||
if os.path.exists(LOG_FILE) and os.path.getsize(LOG_FILE) > MAX_LOG_BYTES:
|
||||
os.replace(LOG_FILE, LOG_FILE + ".1")
|
||||
with open(LOG_FILE, "a") as fh:
|
||||
fh.write("%s ok container=%d runner=%d spawns=%s %s\n" % (
|
||||
time.strftime("%Y-%m-%dT%H:%M:%S"), container_up, runner_up,
|
||||
spawns, ("; ".join(notes) if notes else "")))
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
if notes:
|
||||
print("Hinweise: " + "; ".join(notes), file=sys.stderr)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Executable
+90
@@ -0,0 +1,90 @@
|
||||
#!/bin/sh
|
||||
# =============================================================================
|
||||
# ArchiveBox-Runner-Telemetrie auf einem Knoten OHNE sudo einrichten.
|
||||
#
|
||||
# Drei Teile, alle ohne Root:
|
||||
# 1. Exporteur-Skript nach ~/bin (liest Logs + DB im Container, schreibt .prom)
|
||||
# 2. Cron-Eintrag (jede Minute) -- der Cron-Daemon des Hosts, kein Nutzer-Timer
|
||||
# (linger=no wuerde Nutzer-Units beim Abmelden abraeumen)
|
||||
# 3. Telegraf-Container (Docker-Gruppe) mit Remote Write an den Collector
|
||||
#
|
||||
# Aufruf: sh install-archivebox-telemetrie.sh [--dry-run]
|
||||
# Vorbedingungen: python3, docker (Docker-Gruppe), cron laufen.
|
||||
# =============================================================================
|
||||
set -eu
|
||||
|
||||
DRY="${1:-}"
|
||||
NODE="${NODE_NAME:-archibald}"
|
||||
HOME_DIR="${HOME:-/home/mm}"
|
||||
BIN="$HOME_DIR/bin/archivebox-telemetrie.py"
|
||||
STACK_DIR="$HOME_DIR/stacks/telegraf-archibald"
|
||||
PROM_URL="${PROM_URL:-http://100.83.134.46:9090/api/v1/write}"
|
||||
|
||||
say() { printf '%s\n' "$*"; }
|
||||
run() {
|
||||
if [ "$DRY" = "--dry-run" ]; then say "DRY: $*"; else eval "$@"; fi
|
||||
}
|
||||
|
||||
say "== Vorbedingungen =="
|
||||
for c in python3 docker crontab; do
|
||||
command -v "$c" >/dev/null 2>&1 || { say "FEHLER: $c fehlt"; exit 1; }
|
||||
say "ok: $c -> $(command -v "$c")"
|
||||
done
|
||||
docker ps >/dev/null 2>&1 || { say "FEHLER: Docker nicht nutzbar (Gruppe?)"; exit 1; }
|
||||
say "ok: Docker nutzbar"
|
||||
|
||||
say "== 1. Exporteur =="
|
||||
run "mkdir -p '$HOME_DIR/bin' '$HOME_DIR/abx-telemetrie'"
|
||||
run "chmod 755 '$HOME_DIR/abx-telemetrie'"
|
||||
if [ "$DRY" != "--dry-run" ]; then
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
run "cp '$here/archivebox-telemetrie.py' '$BIN'"
|
||||
run "chmod 755 '$BIN'"
|
||||
fi
|
||||
|
||||
say "== 2. Cron (jede Minute, idempotent) =="
|
||||
LINE="* * * * * LC_ALL=C /usr/bin/python3 $BIN >/dev/null 2>&1"
|
||||
if [ "$DRY" = "--dry-run" ]; then
|
||||
say "DRY: crontab-Zeile -> $LINE"
|
||||
else
|
||||
tmp=$(mktemp)
|
||||
crontab -l 2>/dev/null | grep -v 'archivebox-telemetrie.py' > "$tmp" || true
|
||||
printf '%s\n' "$LINE" >> "$tmp"
|
||||
crontab "$tmp"
|
||||
rm -f "$tmp"
|
||||
say "crontab jetzt:"
|
||||
crontab -l | sed 's/^/ /'
|
||||
fi
|
||||
|
||||
say "== 3. Telegraf-Container =="
|
||||
run "mkdir -p '$STACK_DIR'"
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
if [ "$DRY" != "--dry-run" ]; then
|
||||
run "cp '$here/../stacks/telegraf-container/docker-compose.yml' '$STACK_DIR/'"
|
||||
run "cp '$here/../stacks/telegraf-container/telegraf.conf' '$STACK_DIR/'"
|
||||
# NODE_NAME/PROM_URL aus der Datei gegen die hier gesetzten Werte stellen.
|
||||
run "sed -i \"s|NODE_NAME: .*|NODE_NAME: $NODE|\" '$STACK_DIR/docker-compose.yml'"
|
||||
run "sed -i \"s|PROM_URL: .*|PROM_URL: $PROM_URL|\" '$STACK_DIR/docker-compose.yml'"
|
||||
fi
|
||||
|
||||
say "== Probelauf des Exporteurs =="
|
||||
if [ "$DRY" != "--dry-run" ]; then
|
||||
run "/usr/bin/python3 '$BIN'"
|
||||
say "--- $HOME_DIR/abx-telemetrie/abx.prom (Kopf) ---"
|
||||
head -20 "$HOME_DIR/abx-telemetrie/abx.prom"
|
||||
fi
|
||||
|
||||
say "== Container starten =="
|
||||
if [ "$DRY" != "--dry-run" ]; then
|
||||
( cd "$STACK_DIR" && docker compose up -d )
|
||||
sleep 20
|
||||
docker logs --since 60s telegraf-archibald 2>&1 | grep -c 'E!' | sed 's/^/ E!-Zeilen: /'
|
||||
docker exec telegraf-archibald sh -c 'cat /var/lib/abx-prom/abx.prom | head -3' \
|
||||
| sed 's/^/ Container sieht: /'
|
||||
fi
|
||||
|
||||
say ""
|
||||
say "== Abnahme (auf dem Collector ausfuehren) =="
|
||||
say "curl -s --data-urlencode 'query=archivebox_telemetrie_timestamp_seconds' http://<collector>:9090/api/v1/query"
|
||||
say "curl -s --data-urlencode 'query=count by (node)(archivebox_runner_up)' http://<collector>:9090/api/v1/query"
|
||||
say "OK -- Knoten $NODE eingerichtet."
|
||||
@@ -0,0 +1,32 @@
|
||||
# =============================================================================
|
||||
# Telegraf als CONTAINER auf einem Knoten OHNE sudo.
|
||||
#
|
||||
# Fuer Knoten mit System-Telegraf ist install.sh der richtige Weg; hier genuegt
|
||||
# die Docker-Gruppe, weil nur die fertige .prom-Datei gelesen und per Remote
|
||||
# Write gepusht wird. Kein /hostfs, kein docker.sock, kein privileged -- nichts
|
||||
# davon braucht eine dateibasierte App-Metrik.
|
||||
#
|
||||
# Werte kommen aus der Umgebung (Telegraf expandiert ${...} beim Laden), damit
|
||||
# kein Geheimnis und keine IP doppelt im Repo steht.
|
||||
# =============================================================================
|
||||
services:
|
||||
telegraf:
|
||||
image: telegraf:1.39 # gleiche Version wie der Collector (imac)
|
||||
container_name: telegraf-archibald
|
||||
restart: unless-stopped
|
||||
# host-Netz: derselbe Weg, den ein curl auf dem Host zum Collector nimmt.
|
||||
# Ueber eine Bridge kaeme ein zusaetzlicher NAT/forward-Hop hinzu, der in
|
||||
# NetBird-Firewall-Regeln haengenbleiben kann.
|
||||
network_mode: host
|
||||
environment:
|
||||
NODE_NAME: archibald
|
||||
# Direkter Mesh-Weg, kein Reverse-Proxy: der Collector laeuft mit
|
||||
# --web.enable-remote-write-receiver und OHNE web.config.file, also ohne
|
||||
# Auth. Ueber den Proxy-Weg waere ein Bearer-Token noetig.
|
||||
PROM_URL: http://100.83.134.46:9090/api/v1/write
|
||||
volumes:
|
||||
- ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
|
||||
# VERZEICHNIS-Mount, bewusst kein Datei-Mount: der Cron-Exporteur ersetzt
|
||||
# die .prom-Datei atomar per mv -- ein Datei-Mount haette danach weiter
|
||||
# den alten Inode und lieferte still veraltete Werte.
|
||||
- /home/mm/abx-telemetrie:/var/lib/abx-prom:ro
|
||||
@@ -0,0 +1,44 @@
|
||||
# =============================================================================
|
||||
# Telegraf-Konfiguration fuer einen CONTAINER-Knoten ohne sudo.
|
||||
#
|
||||
# Variablen kommen aus der Container-Umgebung (docker-compose `environment`),
|
||||
# NICHT aus /etc/default/telegraf -- das gibt es auf solchen Knoten nicht.
|
||||
# =============================================================================
|
||||
|
||||
[global_tags]
|
||||
node = "${NODE_NAME}"
|
||||
|
||||
[agent]
|
||||
# Der Exporteur schreibt einmal pro Minute; 60 s Abtastung ist das Maximum,
|
||||
# das keine Werte doppelt sieht. Jitter, damit Cron-Schreiben und Abtastung
|
||||
# nicht im Gleichtakt kollidieren.
|
||||
interval = "60s"
|
||||
flush_interval = "60s"
|
||||
collection_jitter = "5s"
|
||||
hostname = "${NODE_NAME}"
|
||||
omit_hostname = false
|
||||
# Reserve fuer Ausfaelle: Telegraf puffert nur, solange der Push scheitert.
|
||||
metric_buffer_limit = 200000
|
||||
|
||||
# Der Cron-Exporteur schreibt die .prom-Datei atomar (mv).
|
||||
# data_format = "prometheus" ist PFLICHT: das Skript besitzt seine Metriknamen
|
||||
# schon. Mit "influx" bekaeme jede Serie ein _value-Suffix (oder der Parser
|
||||
# scheiterte ganz) und jedes Panel bliebe leer, ohne Fehlermeldung.
|
||||
[[inputs.file]]
|
||||
files = ["/var/lib/abx-prom/abx.prom"]
|
||||
data_format = "prometheus"
|
||||
|
||||
# ---- Metriken -> Prometheus Remote Write (Collector imac) ----
|
||||
[[outputs.http]]
|
||||
url = "${PROM_URL}"
|
||||
data_format = "prometheusremotewrite"
|
||||
timeout = "60s"
|
||||
# Ein 400 ist PERMANENT: dieselbe Nutzlast liefert dieselbe Antwort. Ohne
|
||||
# diese Liste bleibt der abgelehnte Batch als FIFO-Kopf stehen, wird endlos
|
||||
# wiederholt und blockiert jede dahinter gesammelte Serie.
|
||||
non_retryable_statuscodes = [400, 409, 413, 422]
|
||||
[outputs.http.headers]
|
||||
Content-Type = "application/x-protobuf"
|
||||
Content-Encoding = "snappy"
|
||||
X-Prometheus-Remote-Write-Version = "0.1.0"
|
||||
# Kein Authorization-Header: direkter Mesh-Weg ohne Auth (siehe compose).
|
||||
Reference in new issue
Block a user