gpuwatch_

a small watchdog and a long memory for NVIDIA GPUs on Linux.

16.9 KB · python 3.9+ stdlib only · no telemetry, nothing runs here
sha256 4e83a45455603d37eee89b2368e5a80d8022b2dfd4be0e5eca261d59ae5161df

Process monitors forget everything after three minutes. gpuwatch polls your GPU every 2 seconds for safety and every minute for history, keeps months of it in one SQLite file, and draws the day/week/month charts you actually want to look at — in under 1,000 lines of dependency-free Python.

01 · the watchdog (2s loop)

orange — GPU ≥ 84°C held 30s, any thermal-throttle counter increment (instant), a dead-fan heuristic (fan ≤ 15% while ≥ 120W and ≥ 65°C), or kernel Xid errors.
red — GPU ≥ 90°C held 10s.
Alerts to Telegram via the Bot API; optional --enforce runs your own command (stop the offending workload). Deliberately no auto-reboot: killing the workload idles the GPU in seconds; poweroff stays opt-in.
10-minute cooldown after any action, so a hot GPU can't make a daemon flap.
Reboot detection via boot_id; heartbeat event every 15 min so silence is never mistaken for health.

02 · three-tier history, one SQLite file

tier	resolution	kept	what
samples_fast	2 s	48 h ring	forensics: what exactly happened at 3:47am
samples	1 min avg/min/max	months–years	GPU util, temp, fan, power, VRAM, clocks + CPU %, CPU temp, RAM
hourly	rollup	indefinite (~8.8k rows/yr)	min/avg/max, energy Wh, active minutes, event counts, throttle µs

Auto-migrates older schema versions and backfills missing hourly rollups on start.
WAL mode; the reporter reads while the collector writes. One file, one backup.

03 · the dashboard (reporter.py, port 8484)

Read-only Chart.js panel: 8 stat cards with sparklines, 3 synced chart panes (compute, thermals/clocks, power) — 10 charts, one crosshair across all of them.
Temperature envelope band (min/mean/max) instead of a single misleading line; gap shading when the collector was down.
45-day heatmap — active minutes, watt-hours, events, throttle µs — click any cell to zoom that day.
CSV export and a small JSON API (/api/summary, /api/history, /api/heatmap, /api/events, /api/csv) so you can script against your own history.
make_report.py renders a PNG digest of the last N hours for cron/Telegram posts (needs matplotlib).

04 · install

tar xzf gpuwatch-public.tar.gz && cd gpuwatch
sudo ./install.sh        # files to /opt/gpuwatch, systemd units, self-verify
# then open http://localhost:8484

Requires: Linux, NVIDIA GPU with working nvidia-smi, Python 3.9+, systemd (or run the scripts by hand).
Collector ships observe-only. To arm enforcement, add --enforce --orange-cmd "..." to the unit; Telegram alerts need a @BotFather token in two Environment= lines.
install.sh is idempotent; uninstall is systemctl disable --now gpuwatch gpuwatch-reporter && rm -r /opt/gpuwatch /var/lib/gpuwatch /etc/systemd/system/gpuwatch*.service.

// this page hosts the download only — nothing from gpuwatch is installed or running on this server. source is a single README + three scripts; read them, they are short.