Atomic Nginx release runbook

This runbook describes the provider-neutral release transaction. It is not authorization to connect to a server or publish the portfolio. Replace every angle-bracket placeholder only at the explicit deployment checkpoint.

Release-root contract

Provision one dedicated absolute directory such as <release-root>. Never use /, /var, /srv, a home directory, or an existing web root directly. The directory must be owned by a narrowly scoped activation service account and contain:

<release-root>/
  .portfolio-release-root       exact bytes: portfolio-release-root/v1\n
  .control/release-trust.json   canonical audited signing trust policy
  control/state.json            canonical JSON, owner read/write only
  inbox/                        upload identity write-only handoff boundary

The initial state is the canonical form of:

{
  "activeTransaction": null,
  "currentManifestDigest": null,
  "currentReleaseId": null,
  "latestLedgerHeadHash": "<protected-live-ledger-head>",
  "latestLedgerRevision": 0,
  "publisherFence": 0,
  "releaseTrustPolicyDigest": "<sha256-of-protected-release-trust-policy>",
  "schema": "portfolio-release-control-state/v1"
}

Provisioning must write that state through the protected control plane, not by copying an example with placeholder values. The upload identity may create package files only in inbox; it must not own or be able to write the release root, .control, control, releases, receipts, or current. The activation service owns those protected paths and derives immutable release ownership from the provisioned root itself—UID/GID are never supplied by a release caller. Nginx needs read/traverse access to current; it does not need write permission. current points to releases/<release-id>/public, while signed metadata and the SBOM remain in the immutable sibling .release directory and are outside the served document root. Signing keys, the approval ledger, server credentials, and private source remain outside the release package and public root.

The scripts reject non-absolute roots, broad roots, missing sentinels, symlinked verifier modules, unknown canonical fields, and non-canonical JSON.

Build a deterministic package

After the isolated privacy/signing stage has produced a canonical signed release envelope and SBOM:

node scripts/build-release.mjs \
  --dist <exact-dist-directory> \
  --signed-manifest <signed-release-envelope.json> \
  --sbom <sbom.cdx.json> \
  --authority-root <release-root> \
  --protected-state '<canonical-protected-state-json>' \
  --output <new-release.tar>

The production builder loads the single audited verifier and the digest-bound trust policy from the protected authority root. It does not accept verifier code, a policy path, or a caller-provided clock. It validates the typed admission, requires the directory to equal the manifest closure exactly, and checks every byte count and SHA-256 hash. Its ustar output has sorted entries, fixed time/ownership/modes, no links, and exactly these metadata paths:

.release/package-index.json
.release/sbom.cdx.json
.release/signed-release-manifest.json

The remaining entries are exactly the signed deploy files. Building twice from identical inputs must produce identical archive bytes.

Internal staged smoke command

Preparation requires an operator-provisioned absolute executable. It is called without a shell as:

<smoke-command> --release-dir <immutable-release-directory> --manifest-digest <digest>

The executable must expose that directory only through a non-public, authenticated/internal Nginx location, request the staged index and representative hashed/media assets, verify the expected 200/status, headers, MIME/range behavior, and return non-zero on any mismatch. Its standard output becomes digest-bound smoke evidence. Do not point public Nginx at the staged directory.

Prepare without changing visibility

Place the completed package in <release-root>/inbox through the upload identity. Preparation verifies signature, exact predecessor, live ledger, trust-policy digest, fence, hashes, and system time; extracts without using the system tar extractor; makes all release files 0444 and directories 0555 under activation-service ownership; runs the internal smoke; rechecks protected state; and writes a durable readiness receipt. It does not create or change current.

scripts/deploy/prepare-release.sh \
  <release-root> <release-root>/inbox/<new-release.tar> <transaction-id> \
  <absolute-smoke-command>

Inspect the resulting release ID and readiness receipt before activation. A preparation failure leaves the existing current pointer untouched.

Fenced atomic activation

Supply the exact predecessor digest (none only for the first release) and the next fence printed by the verified admission:

scripts/deploy/activate-release.sh \
  <release-root> <transaction-id> <expected-current-digest-or-none> <expected-next-fence>

Activation obtains the activation-service transaction lock, re-verifies the signed envelope against current protected state, and immediately before publication rereads the exact receipt, ledger head/revision, trust-policy digest, predecessor, pointer, fence, and expiry. It writes a durable committing intent, creates the relative current.next symlink, and renames it atomically over current. Every protected control-plane writer must honor the same service lock/CAS protocol; direct state or pointer writes are forbidden.

A failure before the rename leaves the predecessor visible.
A crash after the rename is recovered by repeating the same command: the script observes the new pointer and finalizes the receipt/state without another visibility change.
A second concurrent or stale invocation fails after the winner advances the pointer/fence.
Never edit current, current.next, control state, or receipts by hand.

After activation, run the independent public smoke suite. If it fails, use the signed rollback flow below.

Controlled rollback

Rollback does not replay the selected release's old manifest. The signing pipeline must first create a new candidate whose predecessor is the currently committed manifest, whose ledger/approvals are current, and whose artifact closure is byte-identical to the selected immutable release. Package that new signed envelope as usual, then run:

scripts/deploy/rollback-release.sh \
  <release-root> <selected-old-release-id> <release-root>/inbox/<newly-signed-rollback-release.tar> <transaction-id> \
  <absolute-smoke-command> <current-manifest-digest> <expected-next-fence>

The script verifies that the rollback package uses a fresh manifest digest and exactly matches the selected old artifact closure, then performs the same readiness and atomic pointer transaction. Record both activation and rollback receipts in the release record.

Orphan staging recovery

On startup or retry, a durable preparing intent is recovered idempotently: the activation service proves that its target is not public, removes only that transaction's partial/release/receipt artifacts, clears the intent, and starts verification again. If the public pointer unexpectedly names a preparing release, automatic cleanup stops for privileged investigation. Only hidden directories named <safe-transaction-id>.partial beneath the validated dedicated root may otherwise be cleaned. Active transaction staging is retained. After protected-state recovery has confirmed that no transaction can resume:

scripts/deploy/prepare-release.sh <release-root> --cleanup-orphans

The cleanup path validates the absolute root and sentinel before traversal, rejects unexpected entries or symlinks, and never touches immutable releases, current, or paths outside <release-root>/staging.

Release record

For every attempt record, without secrets:

package SHA-256 and signed candidate/manifest digest;
build ID, release ID, predecessor, live-ledger head/revision, and fence;
readiness and activation receipt digests;
staged and public smoke outcomes;
selected rollback release and the new rollback manifest digest, when applicable;
operator authorization and UTC timestamps.

8.1 KiB Raw Blame History