Monitor DMIT with Prometheus + Grafana

Why Prometheus + Grafana

Monitoring & Backup covers the lightweight approach (uptime probes + email alerts). Once you run multiple instances and need historical trends and visual dashboards, Prometheus (metrics collection/storage) + Grafana (visualisation) + Node Exporter (host metrics) is the open-source standard. This guide walks a minimal deployment on DMIT.

Component roles

Component	Responsibility
Prometheus	Periodically pulls (scrapes) metrics from nodes and stores time series
Node Exporter	Exposes host CPU, memory, disk and network metrics on port 9100
Grafana	Reads the Prometheus datasource and renders dashboards
Alertmanager	Receives rule-triggered alerts, deduplicates and routes notifications

Install Node Exporter (every monitored host)

apt install prometheus-node-exporter -y
systemctl enable --now prometheus-node-exporter
curl -s localhost:9100/metrics | head -3   # verify output

Install Prometheus (monitoring hub)

apt install prometheus -y

Edit /etc/prometheus/prometheus.yml to add scrape targets:

scrape_configs:
  - job_name: "dmit-nodes"
    scrape_interval: 15s
    static_configs:
      - targets:
          - "10.0.0.11:9100"
          - "10.0.0.12:9100"
          - "10.0.0.13:9100"

systemctl restart prometheus
# open http://SERVER:9090/targets — all should be UP

Install Grafana and import a dashboard

apt install -y adduser libfontconfig1 musl
wget https://dl.grafana.com/oss/release/grafana_11.*_amd64.deb
apt install -y ./grafana_11.*_amd64.deb
systemctl enable --now grafana-server

Open http://SERVER:3000, log in with admin/admin, change the password immediately
Data sources → add Prometheus → URL http://localhost:9090
Dashboards → Import → community classic ID 1860 (Node Exporter Full) → pick the datasource

Core alert rules

Edit /etc/prometheus/alert.rules.yml:

groups:
  - name: host-alerts
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels: {severity: critical}
        annotations: {summary: "{{ $labels.instance }} is down"}

      - alert: HighCPU
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 10m
        annotations: {summary: "{{ $labels.instance }} CPU above 85% for 10m"}

      - alert: DiskAlmostFull
        expr: (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes) > 0.9
        for: 5m
        annotations: {summary: "{{ $labels.instance }} disk over 90%"}   # see troubleshooting guide

      - alert: MemoryPressure
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) > 0.9
        for: 5m
        annotations: {summary: "{{ $labels.instance }} available memory below 10%"}

Mount it via rule_files: ["alert.rules.yml"] in prometheus.yml and restart.

Security essentials

Never expose 9090/3000/9100 raw to the internet: firewall to your monitoring range, or reverse-proxy via Nginx with HTTPS + Basic Auth (see HTTPS Setup)
For cross-public-internet scraping use a WireGuard/private tunnel — metric endpoints have no auth of their own
Create viewer-only Grafana accounts; no weak admin passwords

Common issues

Target shows DOWN?

Verify locally with curl localhost:9100/metrics; then from the hub test port 9100 through the firewall (baseline in the Security Checklist).

Dashboards show no data?

Check the Grafana datasource URL and that samples exist in the time range; query the up metric directly in Prometheus to confirm.

Time-series data filling the disk?

Set retention: --storage.tsdb.retention.time=30d, or attach long-term storage.

Load Balancing, Monitoring & Backup, Troubleshooting Guide.

Data checked: 2026-08-18. Software versions follow official documentation.