# Atomic Nginx release runbook This runbook describes the provider-neutral release transaction. It is not authorization to connect to a server or publish the portfolio. Replace every angle-bracket placeholder only at the explicit deployment checkpoint. ## Release-root contract Provision one dedicated absolute directory such as ``. Never use `/`, `/var`, `/srv`, a home directory, or an existing web root directly. The directory must be owned by a narrowly scoped activation service account and contain: ```text / .portfolio-release-root exact bytes: portfolio-release-root/v1\n .control/release-trust.json canonical audited signing trust policy control/state.json canonical JSON, owner read/write only inbox/ upload identity write-only handoff boundary ``` The initial state is the canonical form of: ```json { "activeTransaction": null, "currentManifestDigest": null, "currentReleaseId": null, "latestLedgerHeadHash": "", "latestLedgerRevision": 0, "publisherFence": 0, "releaseTrustPolicyDigest": "", "schema": "portfolio-release-control-state/v1" } ``` Provisioning must write that state through the protected control plane, not by copying an example with placeholder values. The upload identity may create package files only in `inbox`; it must not own or be able to write the release root, `.control`, `control`, `releases`, receipts, or `current`. The activation service owns those protected paths and derives immutable release ownership from the provisioned root itself—UID/GID are never supplied by a release caller. Nginx needs read/traverse access to `current`; it does not need write permission. `current` points to `releases//public`, while signed metadata and the SBOM remain in the immutable sibling `.release` directory and are outside the served document root. Signing keys, the approval ledger, server credentials, and private source remain outside the release package and public root. The scripts reject non-absolute roots, broad roots, missing sentinels, symlinked verifier modules, unknown canonical fields, and non-canonical JSON. ## Build a deterministic package After the isolated privacy/signing stage has produced a canonical signed release envelope and SBOM: ```sh node scripts/build-release.mjs \ --dist \ --signed-manifest \ --sbom \ --authority-root \ --protected-state '' \ --output ``` The production builder loads the single audited verifier and the digest-bound trust policy from the protected authority root. It does not accept verifier code, a policy path, or a caller-provided clock. It validates the typed admission, requires the directory to equal the manifest closure exactly, and checks every byte count and SHA-256 hash. Its ustar output has sorted entries, fixed time/ownership/modes, no links, and exactly these metadata paths: ```text .release/package-index.json .release/sbom.cdx.json .release/signed-release-manifest.json ``` The remaining entries are exactly the signed deploy files. Building twice from identical inputs must produce identical archive bytes. ## Internal staged smoke command Preparation requires an operator-provisioned absolute executable. It is called without a shell as: ```text --release-dir --manifest-digest ``` The executable must expose that directory only through a non-public, authenticated/internal Nginx location, request the staged index and representative hashed/media assets, verify the expected 200/status, headers, MIME/range behavior, and return non-zero on any mismatch. Its standard output becomes digest-bound smoke evidence. Do not point public Nginx at the staged directory. ## Prepare without changing visibility Place the completed package in `/inbox` through the upload identity. Preparation verifies signature, exact predecessor, live ledger, trust-policy digest, fence, hashes, and system time; extracts without using the system tar extractor; makes all release files `0444` and directories `0555` under activation-service ownership; runs the internal smoke; rechecks protected state; and writes a durable readiness receipt. It does not create or change `current`. ```sh scripts/deploy/prepare-release.sh \ /inbox/ \ ``` Inspect the resulting release ID and readiness receipt before activation. A preparation failure leaves the existing `current` pointer untouched. ## Fenced atomic activation Supply the exact predecessor digest (`none` only for the first release) and the next fence printed by the verified admission: ```sh scripts/deploy/activate-release.sh \ ``` Activation obtains the activation-service transaction lock, re-verifies the signed envelope against current protected state, and immediately before publication rereads the exact receipt, ledger head/revision, trust-policy digest, predecessor, pointer, fence, and expiry. It writes a durable committing intent, creates the relative `current.next` symlink, and renames it atomically over `current`. Every protected control-plane writer must honor the same service lock/CAS protocol; direct state or pointer writes are forbidden. - A failure before the rename leaves the predecessor visible. - A crash after the rename is recovered by repeating the same command: the script observes the new pointer and finalizes the receipt/state without another visibility change. - A second concurrent or stale invocation fails after the winner advances the pointer/fence. - Never edit `current`, `current.next`, control state, or receipts by hand. After activation, run the independent public smoke suite. If it fails, use the signed rollback flow below. ## Controlled rollback Rollback does not replay the selected release's old manifest. The signing pipeline must first create a new candidate whose predecessor is the currently committed manifest, whose ledger/approvals are current, and whose artifact closure is byte-identical to the selected immutable release. Package that new signed envelope as usual, then run: ```sh scripts/deploy/rollback-release.sh \ /inbox/ \ ``` The script verifies that the rollback package uses a fresh manifest digest and exactly matches the selected old artifact closure, then performs the same readiness and atomic pointer transaction. Record both activation and rollback receipts in the release record. ## Orphan staging recovery On startup or retry, a durable `preparing` intent is recovered idempotently: the activation service proves that its target is not public, removes only that transaction's partial/release/receipt artifacts, clears the intent, and starts verification again. If the public pointer unexpectedly names a preparing release, automatic cleanup stops for privileged investigation. Only hidden directories named `.partial` beneath the validated dedicated root may otherwise be cleaned. Active transaction staging is retained. After protected-state recovery has confirmed that no transaction can resume: ```sh scripts/deploy/prepare-release.sh --cleanup-orphans ``` The cleanup path validates the absolute root and sentinel before traversal, rejects unexpected entries or symlinks, and never touches immutable releases, `current`, or paths outside `/staging`. ## Release record For every attempt record, without secrets: - package SHA-256 and signed candidate/manifest digest; - build ID, release ID, predecessor, live-ledger head/revision, and fence; - readiness and activation receipt digests; - staged and public smoke outcomes; - selected rollback release and the new rollback manifest digest, when applicable; - operator authorization and UTC timestamps.