Skip to content

packs: single-registration deploy for zero-downtime canary rolls - #20

Merged
pipozzz merged 1 commit into
mainfrom
feat/pack-single-registration-canary
Oct 5, 2026
Merged

pipozzz merged 1 commit into
mainfrom
feat/pack-single-registration-canary

Conversation

@pipozzz

@pipozzz pipozzz commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Problem

Nomad-Pack deploys did two Nomad registrations per deploy:

  1. nomad-pack run registers the pack as a Nomad-native service (invisible to Traefik's consulCatalog), then
  2. a post-deploy patch re-registers it to Consul with the Traefik tags.

With a canary update stanza, each registration is its own canary transition. During the churn Traefik drops a service that momentarily has no passing instance (the count=1 cutover window) → a 404 window on every pack canary roll (observed live on goliash: ~90s of 404). This is why canary works for the panel/compose apps (single, canary-aware registration) but not for packs.

Fix — one clean registration

The deploy shell now renders only (nomad-pack render --to-dir). The panel then, in one pass:

  • parses each rendered job (/v1/jobs/parse),
  • re-stamps the pack.* meta so nomad-pack destroy and getPackJobIds still match,
  • folds in ingress (provider=consul, Traefik tags, canary_tags = dark-until-promoted, shutdown_delay = graceful drain) and scaling (count/resources/autoscaling),
  • registers the job once, then waits for the rollout (shared waitForRollout poller).

One registration = one canary transition, with canary_tags keeping the old alloc serving until the new one is healthy + promoted. Same mechanism the panel job already uses.

Changes

  • builders/nomad.ts — getBuildNomadPackCommand renders instead of runs; add packRenderDir.
  • setup/pack-nomad.ts — nomadParseHCL, registerRenderedPackJobs (parse + meta re-stamp + patch + register).
  • setup/pack-domains.ts — buildPackJobPatch; canary_tags + shutdown_delay on patched services.
  • setup/deploy-monitor.ts — extract/reuse waitForRollout (shared with the detached-nomad path).
  • services/compose.ts — deploy + rebuild call registerPackDeployment; drop the old re-patch.
  • test — assert the render command shape.

Scope / follow-up

startCompose (start-after-stop) still uses the old pattern — but it's start-from-stopped (no running service to keep up), so the canary/404 concern doesn't apply. Optional to unify later.

Validation

  • scripts/check.sh passes: server prod build ok, 529 tests (incl. 2 new), app-tsc unchanged (8 pre-existing informational errors).
  • nomad-pack destroy teardown unchanged (meta re-stamped).
  • Not yet live-tested — recommend merging, rolling, then one goliash redeploy watched end-to-end (expect domain stays 200 throughout).

🤖 Generated with Claude Code

Nomad-Pack deploys used TWO Nomad registrations: 'nomad-pack run' registered the
pack as a Nomad-native service, then a post-deploy patch re-registered it to
Consul (provider + Traefik tags). With a canary update stanza each registration
is its own canary transition, and Traefik consulCatalog drops a service with no
passing instance during the churn → a 404 window on every pack canary roll.

Collapse it into ONE registration: the deploy shell now 'nomad-pack render
--to-dir' only; the panel then parses each rendered job (/v1/jobs/parse),
re-stamps the pack.* meta (so nomad-pack destroy + getPackJobIds still match),
folds in ingress (provider=consul, tags, canary_tags, shutdown_delay) + scaling,
and registers once, then waits for the rollout (shared waitForRollout poller).
One registration = one clean canary transition.

- builders/nomad.ts: getBuildNomadPackCommand renders instead of runs; packRenderDir.
- pack-nomad.ts: nomadParseHCL + registerRenderedPackJobs (+ meta).
- pack-domains.ts: buildPackJobPatch; canary_tags + shutdown_delay on the service.
- deploy-monitor.ts: extract/reuse waitForRollout.
- compose.ts: deploy + rebuild call registerPackDeployment; drop the re-patch.
- test: assert the render command shape.

startCompose (start-after-stop) keeps the old path — no running service to keep
up there, so the canary/404 concern doesn't apply; optional follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions github-actions Bot closed this Oct 5, 2026
@pipozzz pipozzz reopened this Oct 5, 2026
@pipozzz
pipozzz merged commit 5562ffa into main Oct 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant