Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .changes/+heroku-self-hosting.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
支持通过 GitHub Actions 将自托管实例部署到 Heroku 与 Neon。
1 change: 1 addition & 0 deletions .changes/+public-http-sdk.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Core Host SDK 0.3 提供公共 HTTP 基址读取接口,服务包版本仍由独立发行流程管理。
1 change: 1 addition & 0 deletions .changes/+web-extension-install.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
修复 Web 无法通过所选 Core Host 安装 Python-only Extension 的接入缺口,复用既有精确版本安装与兼容性校验,不改变 Extension Host SDK 契约。
1 change: 1 addition & 0 deletions .changes/+worktree-ssh-socket.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
修复深层 worktree 路径下开发数据库 SSH control socket 超长导致启动失败的问题,保留实例独立的 tunnel 与清理归属。
120 changes: 120 additions & 0 deletions .github/workflows/self-host-heroku-neon.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
name: Deploy self-hosted InKCre to Heroku

on:
workflow_dispatch:

permissions:
contents: read

concurrency:
group: self-host-heroku-neon-${{ github.repository }}
cancel-in-progress: false

env:
CORE_DATABASE_PASSWORD: ${{ secrets.CORE_DATABASE_PASSWORD }}
HEAD_SHA: ${{ github.sha }}
HEROKU_API_KEY: ${{ secrets.HEROKU_API_KEY }}
IMAGE_TAG: inkcre-self-host-schema:${{ github.sha }}
JWT_SECRET: ${{ secrets.JWT_SECRET }}
NEON_API_KEY: ${{ secrets.NEON_API_KEY }}
NEON_PROJECT_ID: ${{ vars.NEON_PROJECT_ID }}
POSTGREST_DATABASE_PASSWORD: ${{ secrets.POSTGREST_DATABASE_PASSWORD }}
SOURCE_REVISION: ${{ github.sha }}

jobs:
deploy:
name: Converge Heroku and Neon deployment
runs-on: ubuntu-latest
services:
postgres:
image: >-
pgvector/pgvector:pg17@sha256:d2ef61f42ef767baa5a1475393303cc235bcd92febd9d7014eddb48b41f3bad0
env:
POSTGRES_DB: inkcre
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U postgres -d inkcre"
--health-interval 5s
--health-timeout 5s
--health-retries 12
steps:
- name: Checkout exact selected commit
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5
with:
ref: ${{ github.sha }}
persist-credentials: false

- name: Validate required fork settings
id: settings
env:
HEROKU_APP_PREFIX: ${{ vars.HEROKU_APP_PREFIX }}
run: bash scripts/automation/self_host.sh validate-heroku

- name: Build exact service and transport images
run: |
bash scripts/automation/runtime_artifact.sh build-schema-source
bash scripts/automation/runtime_contract.sh export
bash scripts/automation/runtime_artifact.sh stage-schema-evidence
bash scripts/automation/runtime_contract.sh restore
docker build --platform linux/amd64 --provenance=false \
--build-arg "SOURCE_REVISION=$HEAD_SHA" --target service \
--tag "inkcre-production-web:$HEAD_SHA" .
bash scripts/automation/runtime_artifact.sh build-production-transports

- name: Install pinned Heroku CLI
run: npm install --global heroku@11.8.1

- name: Resolve the Neon default branch owner coordinates
id: neon
run: bash scripts/automation/self_host.sh resolve-neon

- name: Resolve self-hosted Heroku apps
id: heroku
env:
EXPECTED_DEPLOYMENT_PROFILE: ${{ steps.settings.outputs.deployment_profile }}
HEROKU_APP_NAME: ${{ steps.settings.outputs.app_name }}
POSTGREST_APP_NAME: ${{ steps.settings.outputs.postgrest_app_name }}
run: bash scripts/automation/production_delivery.sh resolve-apps

- name: Converge the self-hosted database contract
id: database
env:
DATABASE_ENVIRONMENT: runtime
MIGRATION_DATABASE_URL: ${{ steps.neon.outputs.migration_database_url }}
POOLED_OWNER_DATABASE_URL: ${{ steps.neon.outputs.source_database_url }}
run: bash scripts/automation/production_delivery.sh converge-database

- name: Push and release self-hosted peer processes
id: release
env:
APP_NAME: ${{ steps.heroku.outputs.app_name }}
DATABASE_URL: ${{ steps.database.outputs.core_database_url }}
INKCRE_DEPLOYMENT_PROFILE: ${{ steps.settings.outputs.deployment_profile }}
PEER_NAME: ${{ steps.heroku.outputs.app_name }}
POSTGREST_APP_NAME: ${{ steps.heroku.outputs.postgrest_app_name }}
POSTGREST_DATABASE_URL: ${{ steps.database.outputs.postgrest_database_url }}
run: bash scripts/automation/production_delivery.sh release

- name: Converge self-hosted Peer advertisement
env:
APP_NAME: ${{ steps.heroku.outputs.app_name }}
DATABASE_URL: ${{ steps.database.outputs.core_database_url }}
WEB_URL: ${{ steps.heroku.outputs.web_url }}
run: bash scripts/automation/production_delivery.sh advertise

- name: Probe self-hosted peers
env:
APP_NAME: ${{ steps.heroku.outputs.app_name }}
CORE_IMAGE_DIGEST: ${{ steps.release.outputs.core_local_image_id }}
CORE_IMAGE_LABEL: Exact local image ID
CORE_LOCAL_IMAGE_ID: ${{ steps.release.outputs.core_local_image_id }}
DEPLOYMENT_SUMMARY_TITLE: Self-hosted InKCre on Heroku + Neon
POSTGREST_APP_NAME: ${{ steps.heroku.outputs.postgrest_app_name }}
POSTGREST_RELEASE: ${{ steps.release.outputs.postgrest_release }}
POSTGREST_URL: ${{ steps.heroku.outputs.postgrest_url }}
WEB_RELEASE: ${{ steps.release.outputs.web_release }}
WEB_URL: ${{ steps.heroku.outputs.web_url }}
run: bash scripts/automation/production_delivery.sh probe
13 changes: 7 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,15 @@ Developer setup and shared-skill notes: [CONTRIBUTING.md](CONTRIBUTING.md)

## Browser-Only Self-Hosting

A repository owner can deploy their own InKCre instance to Neon and two Render Docker
services without cloning this repository. Forking is the onboarding mechanism; the deployed
instance is independent from InKCre's canonical production environment. Configure the six
documented GitHub Secrets/Variables, then run the checked-in `Deploy self-hosted InKCre`
workflow. The JWT signing secret remains private and grants full Peer authority.
A repository owner can deploy their own InKCre instance to Neon and either Render or Heroku
without cloning this repository. Forking is the onboarding mechanism; the deployed instance is
independent from InKCre's canonical production environment. Configure the provider profile's
documented GitHub Secrets/Variables, then run its checked-in self-host workflow. The JWT signing
secret remains private and grants full Peer authority.

Exact onboarding steps, runtime limits, and cleanup:
[Self-Hosting On Render And Neon](docs/40-deployment/render-neon-self-host.md).
[Render and Neon](docs/40-deployment/render-neon-self-host.md) or
[Heroku and Neon](docs/40-deployment/heroku-neon-self-host.md).

## Security

Expand Down
3 changes: 3 additions & 0 deletions app/business/extension/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
DisableExtensionCommand,
EnableExtensionCommand,
ExtensionManagementCommand,
InstallExtensionCommand,
PatchExtensionConfigCommand,
)
from app.schemas.peer import PeerProtocolRequest, PeerProtocolResponse, PeerRef
Expand Down Expand Up @@ -218,6 +219,8 @@ async def manage_local(
command: ExtensionManagementCommand,
) -> InstalledExtension:
"""Execute one already-validated command without entering delegation."""
if isinstance(command, InstallExtensionCommand):
return await self.install(command.extension, command.version)
if isinstance(command, EnableExtensionCommand):
return await self.enable(command.extension)
if isinstance(command, DisableExtensionCommand):
Expand Down
13 changes: 13 additions & 0 deletions app/http.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
"""Public HTTP address supplied by this Core's deployment configuration."""

from app.business.peer import PeerManager


async def get_public_http_base_url() -> str | None:
"""Return the validated external base, preserving its path, or None if unset.

Core-owned configuration remains stored on this Core's Peer row. Extensions
consume this address without depending on that persistence layout or guessing
a public URL from request headers or the listening socket.
"""
return (await PeerManager.get_current_config_async()).http_public_base_url
2 changes: 2 additions & 0 deletions app/schemas/extension/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
ExtensionManagementCommand,
ExtensionModel,
ExtensionName,
InstallExtensionCommand,
PatchExtensionConfigCommand,
)

Expand All @@ -17,5 +18,6 @@
"ExtensionManagementCommand",
"ExtensionModel",
"ExtensionName",
"InstallExtensionCommand",
"PatchExtensionConfigCommand",
]
13 changes: 12 additions & 1 deletion app/schemas/extension/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,14 @@
)


class InstallExtensionCommand(pydantic.BaseModel):
model_config = pydantic.ConfigDict(extra="forbid", frozen=True)

action: typing.Literal["install"]
extension: ExtensionName
version: str = pydantic.Field(pattern=EXTENSION_SEMVER_PATTERN)


class EnableExtensionCommand(pydantic.BaseModel):
model_config = pydantic.ConfigDict(extra="forbid", frozen=True)

Expand All @@ -42,7 +50,10 @@ class PatchExtensionConfigCommand(pydantic.BaseModel):


ExtensionManagementCommand: typing.TypeAlias = typing.Annotated[
EnableExtensionCommand | DisableExtensionCommand | PatchExtensionConfigCommand,
InstallExtensionCommand
| EnableExtensionCommand
| DisableExtensionCommand
| PatchExtensionConfigCommand,
pydantic.Field(discriminator="action"),
]

Expand Down
4 changes: 2 additions & 2 deletions app/version.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"""Single source of truth for the Core and Core Extension Host SDK version."""
"""Core Extension Host SDK version, independent of the service package version."""

CORE_VERSION = "0.2.0"
CORE_VERSION = "0.3.0"
3 changes: 3 additions & 0 deletions docs/30-unit-tdd/business-pipeline-and-authority.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,9 @@ runtime 路径禁止同步 session、scoped session 和直接驱动连接;`scr
- Business owners retain typed codecs and non-delegating local seams。Current exact inbounds are
`core.semantic_retrieval.v1`、`core.feature_retrieval.lexical.v1`、`core.organization.rumination.v1` and exact-target
`core.extension.management.v1`。
- Extension management accepts exact-version `install` alongside enable, disable and config patch. It invokes the same
Core Host install boundary as REST, including compatibility and version-change guards; it neither enables the
Extension nor acquires a browser distribution. Older Peers reject the unknown action rather than routing elsewhere.
- `core.peer.protocol.http.v1` owns normalized query/headers/body envelopes、Peer JWT and HTTP response projection。
Generic failover occurs only after pre-dispatch failure or exact `InkCre-Peer-Execution: not-executed`;a normal domain
response or outcome-unknown stops。
Expand Down
21 changes: 20 additions & 1 deletion docs/30-unit-tdd/memos-extension.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,31 @@ Memos adapter 在 root 内组合:

PAT 是 nullable ordinary extension config:`memos_pat_` 加 32 个 ASCII alphanumeric。它属于当前
deployment trust boundary,不建立 session、refresh token、PAT table 或 terminal-user record。比较在
request time 执行,因此有效 config update 后 replace/revoke 立即生效,无需重建 routes。
request time 执行。每个受保护请求通过现有 `EXTENSION_HOST.get` 读取 canonical config,再恢复
`MemosConfig` 类型,不使用启动时的配置快照。因此有效保存完成后发起的请求采用新 PAT;撤销后
旧 PAT 被拒绝,无需重建 routes。已通过鉴权的请求不会被追溯取消。读取失败不回退旧缓存,
invalid config 拒绝鉴权且不回显配置内容。这是一次短数据库读取,不建立配置同步或缓存机制。

ExtensionHost 的通用 config pipeline 先把 patch 与 current config shallow-merge,再用 `config_cls`
验证 complete next value,随后写 DB 并替换 live config。invalid config 不改变 durable/runtime state;
disabled extension 可以预配置 PAT。

Web 等已准入 Peer 也可通过普通数据库配置入口保存,不必将事实写入改成 Core 命令委派。
上述 Memos 请求时读取覆盖这条正常顺序路径,不承诺所有 Extension 的内存资源都会自动重建。

## Connection Preparation

运行中的 Memos 发布 `memos.connection.v1`,固定 GET `/memos/connection`,使用 Peer JWT,
返回仅含 `server_url` 的对象。地址由 Core 进程内 `app.http.get_public_http_base_url()` 和 Memos
自有挂载路径组成;Core-owned 配置仍是唯一公共基址权威,路径前缀保留,未配置时返回 409。
Web 不解释 Peer config 或从 management advertisement 猜测 Memos 地址。该读取不返回 PAT,
不修改配置,也不是运行证明或外部客户端可达性检查;disable 随 Extension 生命周期撤销入口。

初始化沿用浏览器生成/复用 PAT、既有 patch_config 保存、尚未 enabled 才启用的顺序。
在线 Peer + enabled 是用户流程的 best-effort 运行假设,不新增 running 状态或重复启用门槛。
地址读取失败只报告该次失败,不撤销已保存配置或 enabled。外部 Memos API 仍使用 PAT,
Peer JWT 不因此成为 Memos 客户端凭据。

Enable/start 发布一个 retained extension-owned route set;disable/close 直接从 FastAPI dispatch 与
OpenAPI surface 撤销这组 routes。Re-enable 重新发布同一 ownership surface,不重复注册。close cleanup
失败时 routes 仍保持撤销,runtime entry 保留,后续 close/enable 可以继续完成 reconciliation。
Expand Down
2 changes: 2 additions & 0 deletions docs/40-deployment/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ GitHub workflow and composite-action YAML owns only GitHub event selection, perm
- [runtime-orchestration.md](runtime-orchestration.md)
- [mcp-sink.md](mcp-sink.md)
- [render-neon-self-host.md](render-neon-self-host.md)
- [heroku-neon-self-host.md](heroku-neon-self-host.md)

## Checked-In Runtime Anchors

Expand All @@ -40,6 +41,7 @@ GitHub workflow and composite-action YAML owns only GitHub event selection, perm
- `.github/workflows/preview-deploy.yml`
- `.github/workflows/production-deploy.yml`
- `.github/workflows/self-host-render-neon.yml`
- `.github/workflows/self-host-heroku-neon.yml`
- `scripts/deploy_render_neon.py`
- `scripts/generate-openapi.py`
- `docs/openapi.json`
2 changes: 2 additions & 0 deletions docs/40-deployment/development-environment.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ tracked files never own a hostname, user, key, or machine path.
The SSH provider sends an allowlisted build context and one bounded Compose payload to the
remote host. Remote services publish dynamic remote-loopback ports and an instance-owned
OpenSSH control tunnel maps independent local-loopback ports to them.
The descriptor keeps the absolute control-socket path, while SSH runs from its parent directory
with the short socket name so long worktree paths do not exceed Unix-domain socket limits.

Runtime state is written to `.runtime/database/<core-svc-instance>/`. Its `runtime.json`,
`profile.json`, and `readiness.json` record:
Expand Down
70 changes: 70 additions & 0 deletions docs/40-deployment/heroku-neon-self-host.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Self-Hosting On Heroku And Neon

## Purpose And Boundary

The checked-in `Deploy self-hosted InKCre to Heroku` workflow gives a repository owner a
browser-only path from a public `core-py` fork to two Heroku Container Stack apps backed by
the owner's Neon default branch:

- `HEROKU_APP_PREFIX-core` runs the complete `core-py` Peer;
- `HEROKU_APP_PREFIX-postgrest` exposes the executable peer database contract.

The workflow builds the selected commit as `linux/amd64` images, creates missing apps in the
Heroku US region, and runs one Eco web dyno for each app. Heroku charges and Eco sleep behavior
remain the deployment owner's responsibility. This deployment is independent from InKCre's
canonical production environment.

The `JWT_SECRET` grants full admitted-Peer authority. Keep it private. The two database-role
passwords are also persistent deployment credentials: reruns must use the same values unless the
database roles and both app configurations are rotated together.

## Browser-Only Onboarding

1. Fork this repository and enable GitHub Actions for the fork.
2. Create a Neon project and an API key that can access it. Its default branch must retain the
standard `neondb` database and `neondb_owner` role. Copy the project ID.
3. Create a Heroku account with billing enabled and an API key.
4. In **Settings → Secrets and variables → Actions**, add these repository settings.

| Kind | Exact name | Meaning |
| --- | --- | --- |
| Secret | `NEON_API_KEY` | Can resolve and mutate `NEON_PROJECT_ID` |
| Secret | `HEROKU_API_KEY` | Can create, configure, release, and scale the two apps |
| Secret | `JWT_SECRET` | At least 32 bytes; owner-only Peer signing authority |
| Secret | `CORE_DATABASE_PASSWORD` | At least 32 bytes; persistent `inkcre_core` role password |
| Secret | `POSTGREST_DATABASE_PASSWORD` | At least 32 bytes; persistent `authenticator` role password |
| Variable | `NEON_PROJECT_ID` | Target Neon project identity |
| Variable | `HEROKU_APP_PREFIX` | Unique 3–18 character lowercase app prefix |

Use independently generated values for all three credential secrets. Do not reuse a provider API
key or another password as the JWT secret.

5. Open **Actions → Deploy self-hosted InKCre to Heroku → Run workflow** and select the branch
to deploy.
6. Read the workflow summary for the public app URLs, Peer identity, exact commit, and Heroku
releases. Keep using the repository's private `JWT_SECRET` when an admitted client asks for
the signing key.

## Convergence And Safety

The workflow validates all settings before provider mutation, builds the exact selected commit,
resolves the Neon default branch owner coordinates, creates missing Heroku apps, and refuses an
existing app unless its deployment profile matches the same Neon project. It then initializes the
database contract, stores only role-specific URLs in Heroku config, releases both images, converges
the Core Peer's public address, and verifies Core readiness plus the authenticated PostgREST
read/write/deny contract.

The Neon owner URL exists only in the GitHub Actions job. It is never placed in either Heroku app.
Workflow output contains no JWT, database password, or database URL. Rerunning the workflow is a
convergence operation and does not infer credential rotation or cleanup authority.

Heroku's [Container Registry documentation](https://devcenter.heroku.com/articles/container-registry-and-runtime)
defines the image release and `linux/amd64` requirements. Its
[config-var documentation](https://devcenter.heroku.com/articles/config-vars) describes the
persistent runtime configuration used by this profile.

## Cleanup

Delete the two exact Heroku apps when the deployment is no longer wanted. Delete the Neon project
only if it is dedicated to this deployment and its data is disposable. Both actions are explicitly
owner-controlled and intentionally absent from the deployment workflow.
14 changes: 11 additions & 3 deletions docs/40-deployment/native-extension-distribution.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,9 +95,17 @@ Registry-origin database reads finish before Registry HTTP and wheel acquisition
blocking artifact clients run in a worker without a database session. Startup failure or cancellation
withdraws the current publication and releases its runtime claim.

First-party producer metadata targets >=0.2.0 <0.3.0. These changes require new immutable Extension
Core Host SDK 0.3 adds `app.http.get_public_http_base_url()` without changing the async lifecycle
or persistence API. Its authority is `app/version.py`, independently of the Core service package
version in `pyproject.toml`; preparing a service Release PR does not change the Host SDK version.

GitHub, Learn English, Mail, RSS, Telegram, and Twitter retain their SDK 0.2 lower bound and support
SDK 0.3 through `>=0.2.0 <0.4.0`. Memos uses the new HTTP address API and requires
`>=0.3.0 <0.4.0`. Compatibility metadata changes require new immutable Extension
releases; an old wheel's range must not be widened. Before upgrading a deployment, disable affected
old Extensions, adopt the matching Core/Extension releases, restart where a loaded wheel was replaced,
and then enable them. A persisted old enabled[] intent is not silently removed on failed cold restore.
The migration PR remains unreleasable until the complete runtime migration and compatible artifacts
are ready.
For Extensions with both Python and Module Federation distributions, prepare both at the same new
Extension version, even when the browser code is unchanged. Do not upgrade only the Python
association and lose the browser distribution required by the installed Extension. Compatible
artifacts must be available before the deployment upgrade.
Loading
Loading