← Back to LabDebora’s Lab / Publication

From a slow AI job to a clear next step.

A local observability prototype that turns a distributed workload delay into an operator decision. Built to run with Grafana, Prometheus and Docker.

Written from practice↓ Read the field note

This is a local product prototype for a familiar infrastructure moment: a distributed AI job is running, one worker falls behind and the person on call needs to decide whether to wait or intervene.

Start with the person who has to decide

Infrastructure dashboards can expose every available reading while leaving the important question unanswered. In this experiment, the job is not treated as a wall of charts. It is a situation with a delay, a likely cause and a decision that has consequences.

The interface begins with the job's state and the recommended next step. Supporting readings explain why that recommendation exists, rather than asking someone to assemble the story across several panels.

Replay one incident, not an entire cloud

The prototype keeps the scenario deliberately small: four workers process the same workload and one becomes a straggler. Progress diverges, the expected completion time changes and the time at risk becomes visible.

The controls replay baseline, drift and blocked states. They are a local incident simulation designed to test the explanation and interaction, not a claim about a live customer workload.

Use real tooling behind the concept

The companion stack is Docker-ready. Prometheus is the collector: every few seconds it asks the local workload service for readings such as worker progress, pressure and estimated time remaining, then stores them as time series. Grafana is the viewer: it asks Prometheus for those readings, turns them into panels and evaluates alert rules.

The local service produces the same worker-level signals that the product prototype translates into a decision. Grafana is intentionally not the hero. It is the inspectable technical layer behind the experiment: useful for verifying a query, following a signal or debugging an alert.

Make the boundary visible

The stack can measure real container activity when it runs locally. The AI workload itself is simulated because this Mac is not a GPU cluster. That boundary stays visible in the copy and source code.

If the same structure ran beside a real workload, the simulated worker readings could be replaced by the platform's job and hardware telemetry without changing the decision model.

Local product prototype

A decision layer, with observability underneath.

A Grafana-inspired working view for one distributed AI job. Replay the incident, inspect the signals and see what should happen next.

Last 45 minutes
Replay mode · local workload simulation

Job progress

70%

Workers in range

3 / 4

Cost at risk

€14 at risk

Worker progress

completion
worker-01worker-02worker-03worker-04

ETA variance

seconds
+15 minjob completion delay

Container pressure

usage
cluster baselineworker-04

Operator decision

Progress diverging
Inspect worker-04.

One worker is falling behind the group. The gap is widening.

Check input wait and memory pressure
Data source: Prometheus local workload simulatorTechnical layer: Grafana + Prometheus + Docker