Repository navigation
packs: single-registration deploy for zero-downtime canary rolls - #20
Merged
Merged
Conversation
Nomad-Pack deploys used TWO Nomad registrations: 'nomad-pack run' registered the pack as a Nomad-native service, then a post-deploy patch re-registered it to Consul (provider + Traefik tags). With a canary update stanza each registration is its own canary transition, and Traefik consulCatalog drops a service with no passing instance during the churn → a 404 window on every pack canary roll. Collapse it into ONE registration: the deploy shell now 'nomad-pack render --to-dir' only; the panel then parses each rendered job (/v1/jobs/parse), re-stamps the pack.* meta (so nomad-pack destroy + getPackJobIds still match), folds in ingress (provider=consul, tags, canary_tags, shutdown_delay) + scaling, and registers once, then waits for the rollout (shared waitForRollout poller). One registration = one clean canary transition. - builders/nomad.ts: getBuildNomadPackCommand renders instead of runs; packRenderDir. - pack-nomad.ts: nomadParseHCL + registerRenderedPackJobs (+ meta). - pack-domains.ts: buildPackJobPatch; canary_tags + shutdown_delay on the service. - deploy-monitor.ts: extract/reuse waitForRollout. - compose.ts: deploy + rebuild call registerPackDeployment; drop the re-patch. - test: assert the render command shape. startCompose (start-after-stop) keeps the old path — no running service to keep up there, so the canary/404 concern doesn't apply; optional follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Nomad-Pack deploys did two Nomad registrations per deploy:
nomad-pack runregisters the pack as a Nomad-native service (invisible to Traefik'sconsulCatalog), thenWith a canary
updatestanza, each registration is its own canary transition. During the churn Traefik drops a service that momentarily has no passing instance (thecount=1cutover window) → a 404 window on every pack canary roll (observed live on goliash: ~90s of 404). This is why canary works for the panel/compose apps (single, canary-aware registration) but not for packs.Fix — one clean registration
The deploy shell now renders only (
nomad-pack render --to-dir). The panel then, in one pass:/v1/jobs/parse),pack.*meta sonomad-pack destroyandgetPackJobIdsstill match,provider=consul, Traefik tags,canary_tags= dark-until-promoted,shutdown_delay= graceful drain) and scaling (count/resources/autoscaling),waitForRolloutpoller).One registration = one canary transition, with
canary_tagskeeping the old alloc serving until the new one is healthy + promoted. Same mechanism the panel job already uses.Changes
builders/nomad.ts—getBuildNomadPackCommandrenders instead of runs; addpackRenderDir.setup/pack-nomad.ts—nomadParseHCL,registerRenderedPackJobs(parse + meta re-stamp + patch + register).setup/pack-domains.ts—buildPackJobPatch;canary_tags+shutdown_delayon patched services.setup/deploy-monitor.ts— extract/reusewaitForRollout(shared with the detached-nomad path).services/compose.ts— deploy + rebuild callregisterPackDeployment; drop the old re-patch.Scope / follow-up
startCompose(start-after-stop) still uses the old pattern — but it's start-from-stopped (no running service to keep up), so the canary/404 concern doesn't apply. Optional to unify later.Validation
scripts/check.shpasses: server prod build ok, 529 tests (incl. 2 new), app-tsc unchanged (8 pre-existing informational errors).nomad-pack destroyteardown unchanged (meta re-stamped).🤖 Generated with Claude Code