Skip to content

feat(inference)serve Qwen3-8B directly through Envoy, drop Semantic Router - #24

Merged
sojournerdev merged 1 commit into
mainfrom
update
Oct 2, 2026
Merged

sojournerdev merged 1 commit into
mainfrom
update

Conversation

@sojournerdev

Copy link
Copy Markdown
Owner

What

  • Removed Semantic Router. Envoy now sends ai.home.arpa straight to vLLM.
  • Flattened ai/ into inference/.
  • Swapped the model from Qwen3-4B to Qwen3-8B.
  • vLLM now deletes the old pod before starting a new one (Recreate), so upgrades can't deadlock on the GPU.

Why

  • We're setting Semantic Router aside for now; one less moving part.
  • Qwen3-8B is the best model that fits the 8 GB GPU.
  • Upgrades used to hang: the new pod waited for the GPU while the old pod still held it.

Checklist

  • mise run ci passes

@sojournerdev
sojournerdev merged commit b809f39 into main Oct 2, 2026
1 check passed
@sojournerdev
sojournerdev deleted the update branch October 2, 2026 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant