Lightweight System Monitoring for Pi

Content

1. 01Project Overview
2. 02Architecture & Data Flow
3. 03Metrics Collection
4. 04Live Web Dashboard
5. 05Grafana Integration
6. 06Build & Deploy Pipeline
7. 07Debugging Adventures
8. 08Key Learnings
9. 09Full Documentation (Notion)

Project Overview

Pi-Monitor is a lightweight system monitoring agent written in Rust, designed to run on the same Raspberry Pi 3A+ that hosts our custom RustPi Linux distribution. It reads system metrics directly from /proc and /sys, serves them in multiple formats, and includes a built-in web dashboard — all in a single 741KB static binary.

This project is completely separate from RustPi — it has its own repository, its own build pipeline, and its own Vagrant VM. The binary is deployed via SCP and runs as a standalone process. RustPi stays untouched.

Endpoints

Architecture & Data Flow

Data flows one direction: the kernel exposes raw counters in /proc → our Rust code parses those text files into structured data → the HTTP server formats it for whichever endpoint was requested.

The CPU Sampling Problem

Every metric except CPU is a point-in-time snapshot — read /proc/meminfo and you get current memory usage. CPU is different: /proc/stat gives cumulative tick counts since boot. To get a percentage, you need two readings separated by a time interval, compute the difference, and calculate what fraction of elapsed ticks were spent in each state.

This is solved with a background tokio task that samples /proc/stat every 2 seconds and stores the computed percentages in an Arc<Mutex<CpuMetrics>>. HTTP handlers read the latest values without blocking.

Metrics Collection

Five collectors, each parsing a different /proc file format:

CPU Usage Algorithm

/// Compare two /proc/stat snapshots and compute percentages
pub fn calculate_usage(prev: &[RawCpuCounters], curr: &[RawCpuCounters]) -> CpuMetrics {
    let user_diff = curr.user.saturating_sub(prev.user);
    let system_diff = curr.system.saturating_sub(prev.system);
    let idle_diff = curr.idle.saturating_sub(prev.idle);
    let total_diff = user_diff + system_diff + idle_diff + ...;
    
    CpuUsage {
        user_percent: (user_diff as f64 / total_diff as f64) * 100.0,
        idle_percent: (idle_diff as f64 / total_diff as f64) * 100.0,
        ...
    }
}

Live Web Dashboard

The dashboard is a single HTML page embedded in the binary at compile time using Rust's include_str!() macro. It polls /json every 2 seconds and renders all metrics with canvas-drawn graphs and interactive modals.

Click any card to expand — modals show detailed breakdowns with larger fonts and more data

Grafana Integration

Since Pi-Monitor speaks native Prometheus format, it integrates directly with Prometheus + Grafana for historical monitoring with time-range queries, alerting, and all the features a production monitoring stack provides.

PromQL Queries

# CPU usage (excluding idle)
100 - pi_cpu_usage_percent{cpu="total", mode="idle"}

# Memory usage percentage
pi_memory_used_bytes / pi_memory_total_bytes * 100

# Network throughput (bytes/sec)
rate(pi_network_receive_bytes_total{interface="eth0"}[1m])

Build & Deploy Pipeline

The entire workflow is scripted — build in a Vagrant VM, deploy via SCP, auto-start on login.

# Build the static binary (inside Vagrant VM)
./scripts/build.sh
# Output: 741KB static aarch64 musl binary

# Deploy to Pi (stops old instance, uploads, starts)
PI_HOST=10.0.0.111 ./scripts/deploy.sh

# Setup Prometheus + Grafana on Mac
./scripts/monitoring-setup.sh 10.0.0.111

Release Optimizations

[profile.release]
opt-level = "s"      # Optimize for binary size
lto = true           # Link-time optimization
codegen-units = 1    # Better optimization, slower build
strip = true         # Remove debug symbols
panic = "abort"      # No unwinding machinery

Debugging Adventures

Lessons learned while building and deploying Pi-Monitor:

Key Learnings

Technical Skills Gained

1. CPU Usage Is Not Straightforward

Unlike memory or disk, CPU percentage requires diffing two snapshots of cumulative counters over time. This is why every monitoring tool needs a background sampling loop.

2. Separate Reading from Parsing

Every /proc parser has a read function (touches the filesystem) and a parse function (takes a string). This makes unit testing trivial — paste real Pi data into tests without needing /proc.

3. Embed Assets at Compile Time

Using include_str!() to bake HTML/CSS/JS into the binary means zero runtime dependencies. Edit the files separately during development, get a single self-contained binary for deployment.

“The best monitoring tool is the one you understand completely — because when it breaks at 3am, you can fix it.”

Full Documentation

Want the complete technical deep-dive? The full Pi-Monitor documentation covers every /proc file format, every Rust module line-by-line, the async architecture, dashboard design decisions, and every issue encountered with its resolution.