This is a local product prototype for a familiar infrastructure moment: a distributed AI job is running, one worker falls behind and the person on call needs to decide whether to wait or intervene.
Start with the person who has to decide
Infrastructure dashboards can expose every available reading while leaving the important question unanswered. In this experiment, the job is not treated as a wall of charts. It is a situation with a delay, a likely cause and a decision that has consequences.
The interface begins with the job's state and the recommended next step. Supporting readings explain why that recommendation exists, rather than asking someone to assemble the story across several panels.
Replay one incident, not an entire cloud
The prototype keeps the scenario deliberately small: four workers process the same workload and one becomes a straggler. Progress diverges, the expected completion time changes and the time at risk becomes visible.
The controls replay baseline, drift and blocked states. They are a local incident simulation designed to test the explanation and interaction, not a claim about a live customer workload.
Use real tooling behind the concept
The companion stack is Docker-ready. Prometheus is the collector: every few seconds it asks the local workload service for readings such as worker progress, pressure and estimated time remaining, then stores them as time series. Grafana is the viewer: it asks Prometheus for those readings, turns them into panels and evaluates alert rules.
The local service produces the same worker-level signals that the product prototype translates into a decision. Grafana is intentionally not the hero. It is the inspectable technical layer behind the experiment: useful for verifying a query, following a signal or debugging an alert.
Make the boundary visible
The stack can measure real container activity when it runs locally. The AI workload itself is simulated because this Mac is not a GPU cluster. That boundary stays visible in the copy and source code.
If the same structure ran beside a real workload, the simulated worker readings could be replaced by the platform's job and hardware telemetry without changing the decision model.
A decision layer, with observability underneath.
A Grafana-inspired working view for one distributed AI job. Replay the incident, inspect the signals and see what should happen next.
Job progress
70%Workers in range
3 / 4Cost at risk
€14 at riskWorker progress
completionETA variance
secondsContainer pressure
usageOperator decision
Progress divergingOne worker is falling behind the group. The gap is widening.
Check input wait and memory pressure