Asynchronous operations

Every operation that touches infrastructure is asynchronous. Getting the waiting right is most of what separates reliable automation from flaky automation.

The shape of a mutating call

HTTP/1.1 202 Accepted
{
  "server": { "id": "srv_…", "status": "creating", … },
  "job":    { "id": "job_…", "kind": "ServerCreate", "status": "pending" }
}

202 means accepted, not done. The resource is not in its final state. The API has validated your request, allocated what needed allocating, and queued the work.

Job states

Status	Terminal	Meaning
pending	no	Queued, not yet claimed
running	no	Executing; steps shows progress
retry	no	Failed temporarily, scheduled for another attempt
succeeded	yes	Done
failed	yes	Out of attempts, or permanently rejected

retry is not a failure. Hypervisors are busy, locks are held, upstreams time out. A job in retry is still on its way.

Waiting correctly

wait_for_job() {
  local job="$1" deadline=$((SECONDS + 1800))
  while [ $SECONDS -lt $deadline ]; do
    local body status
    body=$(curl -s -H "$AUTH" "$Z9/jobs/$job")
    status=$(jq -r .status <<<"$body")
    case "$status" in
      succeeded) return 0 ;;
      failed)    jq -r .error <<<"$body" >&2; return 1 ;;
    esac
    sleep 5
  done
  echo "timed out waiting for $job" >&2
  return 2
}

Four things this does, all of which matter:

1. Polls the job, not the resource. The job has a terminal state; a resource’s status field is a description, not a completion signal.
2. Stops on failed. Without this the loop runs until the timeout on every failure.
3. Waits five seconds between polls. Faster achieves nothing.
4. Has a deadline. Cluster creation takes minutes; thirty minutes is a safe ceiling for anything.

Reading progress

GET /v1/jobs/{id} includes a steps array — the plan the job produced and where it is. Useful for showing progress; not a completion signal. Steps are derived from what the job found missing, so two runs of the same operation can legitimately have different steps.

Idempotence and retries

Jobs are idempotent: running one twice does nothing the second time. So a failed job that is retried resumes correctly, and re-issuing an operation whose result you are unsure of is safe.

Jobs are also written against state, not steps: a job does not remember it was at step four, it looks at what exists and asks what is missing. That is why an interrupted job resumes correctly rather than repeating work.

Deletion is not instant

A delete returns quickly and the resource stays visible in a terminating state for a buffer period — commonly ten minutes — during which it can be cancelled. Automation that deletes and immediately asserts the resource is gone will fail. Wait for it to disappear, or accept the terminating state as success.

Kubernetes clusters are the exception: no buffer, because once you have confirmed that a cluster should go, its etcd data is gone in every sense that matters.

Common mistakes

Mistake	What happens
Treating 202 as done	You use a resource that is not ready
Polling the resource instead of the job	You never see the failure, only a status that stops changing
Retrying a 409 unchanged	The same conflict forever. Read the message; it names the precondition
Polling every 100 ms	Load, no benefit
No timeout	A hung script