Private AI Code Assistant on Red Hat OpenShift

Deploy a self-hosted AI coding assistant on OpenShift, with private inference unless you enable external model backends.

Table of Contents

1. Detailed description
   • Architecture diagrams
2. Requirements
   • Minimum hardware requirements
   • Minimum software requirements
   • Required user permissions
3. Deploy
   • Option 1: ROSA (AWS)
   • Option 2: ARO (Azure)
   • Option 3: Existing OpenShift
   • Validating the deployment
   • Delete
4. Documentation
5. References
6. Tags

Detailed description

The challenge

Teams want IDE-integrated AI coding help without sending proprietary source to public model APIs. They also need governance: who can call the model, which model is served, and where data resides.

Our solution

This quickstart deploys:

• OpenShift Dev Spaces workspaces offering OpenCode (default), Continue, Cline, or Roo Code
• On-cluster vLLM inference through llm-d — default model Qwen/Qwen3.6-35B-A3B-FP8, configurable for compatible models
• MaaS / RHCL AI Gateway with per-developer API keys, enabled by default
• GitOps Helm charts (ROSA/ARO) or Helm-only install on an existing cluster

Features

• Source and inference stay inside your OpenShift network boundary when configured that way
• OpenAI-compatible /v1 API for IDE extensions
• vLLM prefix caching; EPP off by default
• Multi-developer namespaces with ConfigMap-prewired extensions
• Grafana/guardrails by default; ROSA/ARO Langfuse/OTel; optional MCP

Solution stack

Layer	Technology
IDE	OpenShift Dev Spaces
Auth front door	MaaS / RHCL (maas-default-gateway)
Routing	llm-d + Gateway API; optional EPP
Serving	vLLM / KServe LLMInferenceService
Platform	Red Hat OpenShift AI, NVIDIA GPU Operator, NFD
Delivery	Unified Helm charts in charts/ (ArgoCD or make)

Architecture diagrams

Developers → Dev Spaces → maas-default-gateway → (optional guardrails / Semantic Router) → llm-d → vLLM on NVIDIA GPU.

Illustrative inference traffic flow:

Full diagrams: docs/architecture.md.

Requirements

Minimum hardware requirements

• GPU: 1× NVIDIA GPU with enough VRAM for the model at FP8 (e.g. L40S 48 GB, A100 80 GB, or H100 NVL)
• Storage: PVC for model cache, typically 50–100+ Gi
• CPU / memory: enough for Dev Spaces workspaces plus platform operators (size for team concurrency)

Details: docs/requirements.md, docs/benchmarks.md.

Minimum software requirements

Pick one path.

All paths

• OpenShift 4.20+
• Red Hat OpenShift AI 3.4 (3.3 minimum)
• OpenShift Dev Spaces
• NVIDIA GPU Operator + NFD
• Hugging Face token
• oc CLI

Existing OpenShift — operators above already installed, plus RHCL, helm v3. See deploy_existing_openshift/README.md.

ROSA — Terraform >= 1.4.6, AWS CLI, rosa CLI, ROSA entitlement. See PCA_Deployment_ROSA/README.md.

ARO — Terraform >= 1.4.6, Azure CLI, ARO + GPU quota. See PCA_Deployment_ARO/README.md.

Required user permissions

• ROSA / ARO: Cloud admin to create the cluster, then OpenShift cluster-admin to install operators and sync GitOps.
• Existing OpenShift: OpenShift cluster-admin to deploy AI serving, DevSpaces, and (optional) the demo IDP.

Deploy

Choose one path. Do not mix ArgoCD GitOps (ROSA/ARO) with the existing-OpenShift Helm flow on the same cluster unless you know how ownership overlaps.

Option 1: ROSA (AWS)

Terraform provisions ROSA HCP and a GPU pool; ArgoCD syncs charts/ with values-rosa.yaml.

→ PCA_Deployment_ROSA/README.md

Option 2: ARO (Azure)

Terraform provisions ARO and a GPU MachineSet; ArgoCD syncs charts/ with values-aro.yaml.

→ PCA_Deployment_ARO/README.md

Option 3: Existing OpenShift

Cluster already has RHOAI, GPU Operator, and Dev Spaces. Install RHCL first, then:

# AI serving once
make ai-serving-deploy-existing-openshift HF_TOKEN=hf_xxx

# DevSpaces: N users (dev-user1..N) or a single DEV_USER
make devspace-deploy-existing-openshift N=<n>
# or: make devspace-deploy-existing-openshift DEV_USER=dev-user1

→ deploy_existing_openshift/README.md

Validating the deployment

After deploy, run cluster smoke tests (not CI):

make smoke
make smoke AI_NAMESPACE=ai-serving DEV_USER=dev-user1

→ tests/cluster-smoke/README.md

Delete

Existing OpenShift — remove with make:

make undeploy-existing-openshift
# or separately:
make devspace-undeploy-existing-openshift          # all demo users (or DEV_USER= / N=)
make ai-serving-undeploy-existing-openshift DELETE_NAMESPACE=1

ARO — follow Destroying the Cluster.

ROSA — follow Destroying the Cluster.

Documentation

Doc	Topic
docs/architecture.md	Stack, traffic path, diagrams
docs/requirements.md	Hardware, software, permissions
docs/ide-and-extensions.md	Dev Spaces, OpenCode / Continue / Cline / Roo Code
docs/models-and-routing.md	vLLM, llm-d, RHCL
docs/semantic-router.md	Official vLLM SR hop, extras, inject
docs/models-and-hardware.md	Changing models and accelerators
docs/benchmarks.md	Performance highlights + links
docs/customization.md	Prompts, rules, MCP, quality gates
AGENTS.md	Maintainer chart / wave map

References

• Red Hat OpenShift AI
• OpenShift Dev Spaces
• llm-d
• vLLM
• Kuadrant / RHCL

Tags

• Title: Private AI Code Assistant on Red Hat OpenShift
• Description: Deploy a self-hosted AI coding assistant on OpenShift, with private inference unless you enable external model backends.
• Industry: Media and IT services
• Product: OpenShift AI
• Use case: AI coding assistant, private inference
• Status: published