The job model

Anything that touches infrastructure is a job. Creating a server, attaching an address, building a Kubernetes cluster, deleting a load balancer: all the same machinery.

The queue is a table

Jobs live in a PostgreSQL table, not a separate broker. That is a deliberate simplification with one decisive benefit: a job and the intent that produced it commit in the same transaction. There is no window where an address was allocated but the job that uses it was lost, or the reverse.

Each row carries its kind, the region it belongs to, the resource it acts on, an attempt counter, a run_after timestamp for backoff, the lease holder, and a list of steps for progress display.

Four phases

Every job implements the same shape.

Observe. Read the world. Not the database — the world. Does this VM exist? What is its current configuration? The lookup is by tag, so a job finds its own resource even if someone renamed it.

Plan. Compare the world with the intent and produce the list of steps that are still missing. A job whose work is already done produces an empty plan, and that is the normal, successful outcome of a re-run.

Apply. Execute the steps in order. Each step is individually idempotent and waits for the hypervisor task to actually complete.

Verify. Read the result back and write what was observed. This is where the panel’s status comes from — never from “the apply call did not throw”.

Idempotence is not optional

Running a job twice must do nothing the second time. This is not a nicety; it is what makes retries safe, and retries are unavoidable when the other end of the wire is a hypervisor cluster.

In practice it means jobs are written against state, not steps. A job does not remember “I was at step 4”. It looks at what exists and asks “what is still missing?”. That is why a job interrupted by a power cut resumes correctly, and why an operator can safely re-run one.

Leases

A job is claimed by exactly one worker, which stamps the row with its identity and a timestamp. If that worker disappears — the agent VM is rebooted, the process is killed — the lease expires and another worker reclaims the job.

The lease is refreshed while the job reports progress. A long job that is genuinely working is not stolen from underneath itself; a job whose worker died is not stuck forever. Both failure modes were observed in practice and both are handled.

Failure and backoff

A failed job increments its attempt counter, records the error and schedules itself for later with exponential backoff, up to a maximum attempt count.

Errors are classified, and the distinction changes what the user sees:

• Temporary — the hypervisor was busy, a lock was held, an upstream timed out. The job retries. The resource is not marked failed, because showing “error” for a transient condition pushes users to delete and rebuild something that was about to succeed.
• Permanent — the request cannot succeed as specified. The job stops and the resource carries the reason.

Reading job progress

GET /v1/jobs/{id} returns the status, the attempt count, the step list with per-step state, and the error if there is one.

Status	Meaning
pending	Queued, not yet claimed
running	Claimed and executing; steps shows progress
retry	Failed temporarily, scheduled for another attempt
succeeded	Terminal, successful
failed	Terminal, out of attempts or permanently rejected

Poll the job, not the resource, and stop on a terminal status. See Asynchronous operations.

Two-phase deletion

Deletion deserves its own mention because it is where an irreversible mistake would live.

Pressing delete does not destroy anything. It marks the resource as deleting, stops it where that applies, and starts a buffer. Within that window you can cancel and get everything back. When the buffer expires, a job removes the resource for real.

The buffer length is an installation setting. Ten minutes is a common value.

There is a matching safety rule in the deletion job itself: before destroying a virtual machine it re-checks the tags to confirm the machine is the one it was asked to delete and that it is a customer machine. A VM carrying a platform role — a region agent, for example — is refused unconditionally, no matter what the job says. That guard exists because a near-miss happened: a stale row pointed at the identifier of the region’s own agent VM.