Operations
Upgrade order
Section titled “Upgrade order”When a release changes both the panel and the agent:
- Publish the agent release. Regions pick it up on their own timer, verify the checksum, and roll back if the new binary does not start.
- Update the expected agent version in the control plane environment and deploy the panel.
That order matters when a release adds a field appliances must understand: an old appliance receiving new configuration may refuse it, while a new appliance receiving old configuration is designed to be safe.
Panel upgrades run migrations first, then roll out. Image tags carry a timestamp and commit, so rolling back is redeploying a previous tag.
What to watch
Section titled “What to watch”| Signal | Where | Why it matters |
|---|---|---|
| Agent last-seen | Region page | A quiet agent makes jobs look stuck rather than failed |
Jobs in retry | Jobs list | A few are normal. A growing pile means an upstream is unwell |
| Appliance applied version vs intended | Load balancer / storage pages | A gap that does not close means the appliance cannot fetch or cannot apply |
| Certificate status | Certificates | error with a backoff means issuance is failing repeatedly |
| Pool free addresses | IP pools | Running out blocks gateway and load balancer creation |
| Database size and backups | Your database | Intent, allocations and the audit trail live there |
Common situations
Section titled “Common situations”A job is stuck in running. Its worker may have died. The lease expires and another
worker reclaims it; that is designed behaviour, not something to force. If it recurs,
check whether the agent is restarting.
An appliance never converges. Its subnet almost certainly has no route to the panel. The appliance pulls its own configuration; without egress it installs and waits.
A cluster shows degraded after you fixed the cause. Use the re-verify action. Cluster health used to be measured only at the end of the creation job, so a fixed problem could leave a stale verdict; measurement is now its own operation.
Certificates keep failing. Read the error on the listener. Backoff is exponential up to a day — deliberately, because retrying a failing ACME order in a loop consumes the domain’s weekly rate limit. After fixing the cause use the explicit retry, which clears the backoff.
A tenant says a deleted server is still visible. That is the deletion buffer. It will disappear.
Rotating credentials
Section titled “Rotating credentials”Proxmox token. Create a new token, update the agent’s environment, restart the agent, delete the old token. The agent is the only holder.
Edge token. Same shape.
Provider tokens (DNS, registrar) live in the control plane. Update the secret and restart the API.
Do this on suspicion, not only on schedule. If a credential file may have been exposed — pasted into a chat, printed by a verbose command, committed by accident — rotate every secret in it rather than the one you think leaked.
Backups
Section titled “Backups”The control plane database is what matters. It holds intent, allocations and the audit trail. Losing it does not stop virtual machines — nothing depends on the panel — but you lose the record of who owns what.
Customer data is Proxmox’s business. Use Proxmox Backup Server. There is no backup product in the panel and snapshots are not backups.
The region configuration file is versioned in your repository, which is where it belongs.
Taking a region out of service
Section titled “Taking a region out of service”Set read_only: true in the region configuration and deploy. Reads keep working, write
jobs are refused with a clear reason, and nothing pretends the region is gone.
Adding capacity
Section titled “Adding capacity”Add the node to the Proxmox cluster and prepare it — storage, network, SDN. Then add it to
the region’s nodes list. Placement starts using it on the next creation. A node absent
from the list never receives a machine, which is what lets you add hardware before it is
ready.