Atomic Nginx release runbook

This runbook describes the provider-neutral release transaction. It is not authorization to connect to a server or publish the portfolio. Replace every angle-bracket placeholder only at the explicit deployment checkpoint.

Release-root contract

Provision one dedicated absolute directory such as <release-root>. Never use /, /var, /srv, a home directory, or an existing web root directly. The directory must be owned by the narrowly scoped deployment account and contain:

<release-root>/
  .portfolio-release-root       exact bytes: portfolio-release-root/v1\n
  control/state.json            canonical JSON, owner read/write only

The initial state is the canonical form of:

{
  "activeTransaction": null,
  "currentManifestDigest": null,
  "currentReleaseId": null,
  "latestLedgerHeadHash": "<protected-live-ledger-head>",
  "latestLedgerRevision": 0,
  "publisherFence": 0,
  "schema": "portfolio-release-control-state/v1"
}

Provisioning must write that state through the protected control plane, not by copying an example with placeholder values. The deployment account needs write access only to this dedicated root. Nginx needs read/traverse access to current; it does not need permission to write releases or control state. current points to releases/<release-id>/public, while signed metadata and the SBOM remain in the immutable sibling .release directory and are therefore outside the served document root. Signing keys, the approval ledger, trust policy, server credentials, and private source remain outside the release package and public root.

The scripts reject non-absolute roots, broad roots, missing sentinels, symlinked verifier modules, unknown canonical fields, and non-canonical JSON.

Build a deterministic package

After the isolated privacy/signing stage has produced a canonical signed release envelope and SBOM:

node scripts/build-release.mjs \
  --dist <exact-dist-directory> \
  --signed-manifest <signed-release-envelope.json> \
  --sbom <sbom.cdx.json> \
  --verifier-module <absolute-path-to-audited-sign-release.mjs> \
  --trust-policy <external-canonical-trust-policy.json> \
  --protected-state '<canonical-protected-state-json>' \
  --now <trusted-rfc3339-time> \
  --output <new-release.tar>

The builder verifies the real Ed25519 envelope through verifySignedReleaseEnvelope, validates the typed admission, requires the directory to equal the manifest closure exactly, and checks every byte count and SHA-256 hash. Its ustar output has sorted entries, fixed time/ownership/modes, no links, and exactly these metadata paths:

.release/package-index.json
.release/sbom.cdx.json
.release/signed-release-manifest.json

The remaining entries are exactly the signed deploy files. Building twice from identical inputs must produce identical archive bytes.

Internal staged smoke command

Preparation requires an operator-provisioned absolute executable. It is called without a shell as:

<smoke-command> --release-dir <immutable-release-directory> --manifest-digest <digest>

The executable must expose that directory only through a non-public, authenticated/internal Nginx location, request the staged index and representative hashed/media assets, verify the expected 200/status, headers, MIME/range behavior, and return non-zero on any mismatch. Its standard output becomes digest-bound smoke evidence. Do not point public Nginx at the staged directory.

Prepare without changing visibility

Use the deployment account's numeric UID/GID. Preparation verifies signature, exact predecessor, live ledger, fence, hashes, and time; extracts without using the system tar extractor; makes all release files 0444 and directories 0555; runs the internal smoke; rechecks time and protected state; and writes an immutable readiness receipt. It does not create or change current.

scripts/deploy/prepare-release.sh \
  <release-root> <new-release.tar> <transaction-id> \
  <absolute-verifier-module> <external-trust-policy.json> <absolute-smoke-command> \
  <deployment-uid> <deployment-gid> <trusted-rfc3339-time>

Inspect the resulting release ID and readiness receipt before activation. A preparation failure leaves the existing current pointer untouched.

Fenced atomic activation

Supply the exact predecessor digest (none only for the first release) and the next fence printed by the verified admission:

scripts/deploy/activate-release.sh \
  <release-root> <transaction-id> <expected-current-digest-or-none> <expected-next-fence> \
  <absolute-verifier-module> <external-trust-policy.json> <trusted-rfc3339-time>

Activation obtains an exclusive transaction lock, re-verifies the signed envelope against current protected state, rechecks receipt/ledger/predecessor/fence/expiry, writes a durable committing intent, creates the relative current.next symlink, and renames it atomically over current. Nothing else changes public visibility.

A failure before the rename leaves the predecessor visible.
A crash after the rename is recovered by repeating the same command: the script observes the new pointer and finalizes the receipt/state without another visibility change.
A second concurrent or stale invocation fails after the winner advances the pointer/fence.
Never edit current, current.next, control state, or receipts by hand.

After activation, run the independent public smoke suite. If it fails, use the signed rollback flow below.

Controlled rollback

Rollback does not replay the selected release's old manifest. The signing pipeline must first create a new candidate whose predecessor is the currently committed manifest, whose ledger/approvals are current, and whose artifact closure is byte-identical to the selected immutable release. Package that new signed envelope as usual, then run:

scripts/deploy/rollback-release.sh \
  <release-root> <selected-old-release-id> <newly-signed-rollback-release.tar> <transaction-id> \
  <absolute-verifier-module> <external-trust-policy.json> <absolute-smoke-command> \
  <deployment-uid> <deployment-gid> <current-manifest-digest> <expected-next-fence> <trusted-rfc3339-time>

The script verifies that the rollback package uses a fresh manifest digest and exactly matches the selected old artifact closure, then performs the same readiness and atomic pointer transaction. Record both activation and rollback receipts in the release record.

Orphan staging recovery

Only hidden directories named <safe-transaction-id>.partial beneath the validated dedicated root may be cleaned. Active transaction staging is retained. After protected-state recovery has confirmed that no transaction can resume:

scripts/deploy/prepare-release.sh <release-root> --cleanup-orphans

The cleanup path validates the absolute root and sentinel before traversal, rejects unexpected entries or symlinks, and never touches immutable releases, current, or paths outside <release-root>/staging.

Release record

For every attempt record, without secrets:

package SHA-256 and signed candidate/manifest digest;
build ID, release ID, predecessor, live-ledger head/revision, and fence;
readiness and activation receipt digests;
staged and public smoke outcomes;
selected rollback release and the new rollback manifest digest, when applicable;
operator authorization and UTC timestamps.

7.3 KiB Raw Blame History