132 lines
8.1 KiB
Markdown
132 lines
8.1 KiB
Markdown
# Atomic Nginx release runbook
|
|
|
|
This runbook describes the provider-neutral release transaction. It is not authorization to connect to a server or publish the portfolio. Replace every angle-bracket placeholder only at the explicit deployment checkpoint.
|
|
|
|
## Release-root contract
|
|
|
|
Provision one dedicated absolute directory such as `<release-root>`. Never use `/`, `/var`, `/srv`, a home directory, or an existing web root directly. The directory must be owned by a narrowly scoped activation service account and contain:
|
|
|
|
```text
|
|
<release-root>/
|
|
.portfolio-release-root exact bytes: portfolio-release-root/v1\n
|
|
.control/release-trust.json canonical audited signing trust policy
|
|
control/state.json canonical JSON, owner read/write only
|
|
inbox/ upload identity write-only handoff boundary
|
|
```
|
|
|
|
The initial state is the canonical form of:
|
|
|
|
```json
|
|
{
|
|
"activeTransaction": null,
|
|
"currentManifestDigest": null,
|
|
"currentReleaseId": null,
|
|
"latestLedgerHeadHash": "<protected-live-ledger-head>",
|
|
"latestLedgerRevision": 0,
|
|
"publisherFence": 0,
|
|
"releaseTrustPolicyDigest": "<sha256-of-protected-release-trust-policy>",
|
|
"schema": "portfolio-release-control-state/v1"
|
|
}
|
|
```
|
|
|
|
Provisioning must write that state through the protected control plane, not by copying an example with placeholder values. The upload identity may create package files only in `inbox`; it must not own or be able to write the release root, `.control`, `control`, `releases`, receipts, or `current`. The activation service owns those protected paths and derives immutable release ownership from the provisioned root itself—UID/GID are never supplied by a release caller. Nginx needs read/traverse access to `current`; it does not need write permission. `current` points to `releases/<release-id>/public`, while signed metadata and the SBOM remain in the immutable sibling `.release` directory and are outside the served document root. Signing keys, the approval ledger, server credentials, and private source remain outside the release package and public root.
|
|
|
|
The scripts reject non-absolute roots, broad roots, missing sentinels, symlinked verifier modules, unknown canonical fields, and non-canonical JSON.
|
|
|
|
## Build a deterministic package
|
|
|
|
After the isolated privacy/signing stage has produced a canonical signed release envelope and SBOM:
|
|
|
|
```sh
|
|
node scripts/build-release.mjs \
|
|
--dist <exact-dist-directory> \
|
|
--signed-manifest <signed-release-envelope.json> \
|
|
--sbom <sbom.cdx.json> \
|
|
--authority-root <release-root> \
|
|
--protected-state '<canonical-protected-state-json>' \
|
|
--output <new-release.tar>
|
|
```
|
|
|
|
The production builder loads the single audited verifier and the digest-bound trust policy from the protected authority root. It does not accept verifier code, a policy path, or a caller-provided clock. It validates the typed admission, requires the directory to equal the manifest closure exactly, and checks every byte count and SHA-256 hash. Its ustar output has sorted entries, fixed time/ownership/modes, no links, and exactly these metadata paths:
|
|
|
|
```text
|
|
.release/package-index.json
|
|
.release/sbom.cdx.json
|
|
.release/signed-release-manifest.json
|
|
```
|
|
|
|
The remaining entries are exactly the signed deploy files. Building twice from identical inputs must produce identical archive bytes.
|
|
|
|
## Internal staged smoke command
|
|
|
|
Preparation requires an operator-provisioned absolute executable. It is called without a shell as:
|
|
|
|
```text
|
|
<smoke-command> --release-dir <immutable-release-directory> --manifest-digest <digest>
|
|
```
|
|
|
|
The executable must expose that directory only through a non-public, authenticated/internal Nginx location, request the staged index and representative hashed/media assets, verify the expected 200/status, headers, MIME/range behavior, and return non-zero on any mismatch. Its standard output becomes digest-bound smoke evidence. Do not point public Nginx at the staged directory.
|
|
|
|
## Prepare without changing visibility
|
|
|
|
Place the completed package in `<release-root>/inbox` through the upload identity. Preparation verifies signature, exact predecessor, live ledger, trust-policy digest, fence, hashes, and system time; extracts without using the system tar extractor; makes all release files `0444` and directories `0555` under activation-service ownership; runs the internal smoke; rechecks protected state; and writes a durable readiness receipt. It does not create or change `current`.
|
|
|
|
```sh
|
|
scripts/deploy/prepare-release.sh \
|
|
<release-root> <release-root>/inbox/<new-release.tar> <transaction-id> \
|
|
<absolute-smoke-command>
|
|
```
|
|
|
|
Inspect the resulting release ID and readiness receipt before activation. A preparation failure leaves the existing `current` pointer untouched.
|
|
|
|
## Fenced atomic activation
|
|
|
|
Supply the exact predecessor digest (`none` only for the first release) and the next fence printed by the verified admission:
|
|
|
|
```sh
|
|
scripts/deploy/activate-release.sh \
|
|
<release-root> <transaction-id> <expected-current-digest-or-none> <expected-next-fence>
|
|
```
|
|
|
|
Activation obtains the activation-service transaction lock, re-verifies the signed envelope against current protected state, and immediately before publication rereads the exact receipt, ledger head/revision, trust-policy digest, predecessor, pointer, fence, and expiry. It writes a durable committing intent, creates the relative `current.next` symlink, and renames it atomically over `current`. Every protected control-plane writer must honor the same service lock/CAS protocol; direct state or pointer writes are forbidden.
|
|
|
|
- A failure before the rename leaves the predecessor visible.
|
|
- A crash after the rename is recovered by repeating the same command: the script observes the new pointer and finalizes the receipt/state without another visibility change.
|
|
- A second concurrent or stale invocation fails after the winner advances the pointer/fence.
|
|
- Never edit `current`, `current.next`, control state, or receipts by hand.
|
|
|
|
After activation, run the independent public smoke suite. If it fails, use the signed rollback flow below.
|
|
|
|
## Controlled rollback
|
|
|
|
Rollback does not replay the selected release's old manifest. The signing pipeline must first create a new candidate whose predecessor is the currently committed manifest, whose ledger/approvals are current, and whose artifact closure is byte-identical to the selected immutable release. Package that new signed envelope as usual, then run:
|
|
|
|
```sh
|
|
scripts/deploy/rollback-release.sh \
|
|
<release-root> <selected-old-release-id> <release-root>/inbox/<newly-signed-rollback-release.tar> <transaction-id> \
|
|
<absolute-smoke-command> <current-manifest-digest> <expected-next-fence>
|
|
```
|
|
|
|
The script verifies that the rollback package uses a fresh manifest digest and exactly matches the selected old artifact closure, then performs the same readiness and atomic pointer transaction. Record both activation and rollback receipts in the release record.
|
|
|
|
## Orphan staging recovery
|
|
|
|
On startup or retry, a durable `preparing` intent is recovered idempotently: the activation service proves that its target is not public, removes only that transaction's partial/release/receipt artifacts, clears the intent, and starts verification again. If the public pointer unexpectedly names a preparing release, automatic cleanup stops for privileged investigation. Only hidden directories named `<safe-transaction-id>.partial` beneath the validated dedicated root may otherwise be cleaned. Active transaction staging is retained. After protected-state recovery has confirmed that no transaction can resume:
|
|
|
|
```sh
|
|
scripts/deploy/prepare-release.sh <release-root> --cleanup-orphans
|
|
```
|
|
|
|
The cleanup path validates the absolute root and sentinel before traversal, rejects unexpected entries or symlinks, and never touches immutable releases, `current`, or paths outside `<release-root>/staging`.
|
|
|
|
## Release record
|
|
|
|
For every attempt record, without secrets:
|
|
|
|
- package SHA-256 and signed candidate/manifest digest;
|
|
- build ID, release ID, predecessor, live-ledger head/revision, and fence;
|
|
- readiness and activation receipt digests;
|
|
- staged and public smoke outcomes;
|
|
- selected rollback release and the new rollback manifest digest, when applicable;
|
|
- operator authorization and UTC timestamps.
|