feat(models): support extension-scoped shared weight groups

[DrHepa]

Summary

This PR implements extension-scoped shared weight groups for model extensions.

Multiple model nodes within the same extension can now reuse one or more physical weight groups while retaining their own capability identity and optional node-private weights.

This supports both primary use cases described in #343:

• Fully shared checkpoints used by multiple sibling nodes, such as SenseNova.
• A shared base model combined with node-private adapters, such as Pixal3D/WorldSculpt or DreamStyle3D.

Closes #343.

Motivation

Modly currently treats each model node as both a capability and the physical owner of its weights:

///

This causes every node to download its own copy of the same checkpoint, even when several nodes within an extension use an identical base model.

A simple node-owner alias is insufficient because some nodes need to combine a shared base checkpoint with additional private weights. It would also couple runtime identity and deletion behavior to a physical directory.

This PR separates:

• Capability identity.
• Shared weight ownership.
• Node-private weight ownership.

Manifest contract

Model extensions can declare top-level "weight_groups". Each group owns its own validated "model_sources" plan.

Nodes reference the shared groups they require and may continue declaring private "model_sources".

Example:

{
"id": "pixal3d",
"type": "model",
"weight_groups": [
{
"id": "pixal3d-base",
"model_sources": [
{
"id": "pixal3d",
"provider": "huggingface",
"repo_id": "TencentARC/Pixal3D",
"revision": "",
"destination": ".",
"checks": [
"pipeline.json"
]
}
]
}
],
"nodes": [
{
"id": "generate",
"weight_groups": [
"pixal3d-base"
]
},
{
"id": "worldsculpt",
"weight_groups": [
"pixal3d-base"
],
"model_sources": [
{
"id": "worldsculpt-adapter",
"provider": "huggingface",
"repo_id": "AlayaLab/WorldSculpt",
"revision": "",
"destination": "worldsculpt",
"checks": [
"worldsculpt/"
]
}
]
}
]
}

Storage layout

Shared groups are stored independently from node-private weights:

/
/
_shared/
/
...
/
...

Rules enforced by the implementation:

• Shared sources are confined to "_shared/".
• Node-private sources remain under "".
• Group IDs are local to their extension.
• "_shared" is reserved and cannot be used as a node ID.
• Absolute paths, "..", unsafe IDs, path separators inside IDs and escaping checks are rejected.
• Cross-extension weight-group references are not supported.
• No symlinks or hard links are required.

Download behavior

Installing a node builds an effective download plan containing:

1. Every shared weight group referenced by the node.
2. The node’s private "model_sources".

The download lifecycle now:

• Skips shared groups that are already complete.
• Deduplicates shared work during “Install all”.
• Locks downloads by canonical physical target.
• Prevents concurrent node installations from writing to the same shared group.
• Preserves existing pause, resume, cancellation and repair behavior.
• Keeps a completed shared group valid if a node-private source fails.
• Refreshes all dependent node states when a shared group changes.

Readiness behavior

A node is ready only when:

• Every referenced shared group passes its checks.
• Every node-private source passes its checks.

This allows states such as:

• Pixal3D base installed → the regular Pixal3D node is ready.
• Pixal3D base installed but WorldSculpt adapter missing → WorldSculpt is not ready.
• Installing WorldSculpt reuses the existing Pixal3D base and downloads only its additional weights.
• One completed SenseNova group makes every dependent node without private sources ready.

Runtime contract

Capability identity is now explicitly separated from storage identity.

The runtime no longer depends on "MODEL_DIR.name" to determine which node is running.

Direct and subprocess extensions receive:

MODEL_ID=/
MODEL_NODE_ID=
MODEL_DIR=
SHARED_MODEL_DIRS={"":""}

The Python generator context also exposes the resolved "shared_model_dirs" mapping.

"MODEL_DIR" keeps its existing node-private meaning for backward compatibility.

When the configured model storage path changes, both private and shared directories are resolved again from the host configuration.

Deletion and uninstall behavior

Shared groups have an independent lifecycle:

• Removing node-private weights deletes only the node directory.
• Removing a shared group requires a separate explicit action.
• The UI warns that every dependent node will become unavailable.
• Loaded generators referencing the group are unloaded before deletion.
• Removing a shared group refreshes every affected node.
• Deleting one capability never silently removes weights required by sibling capabilities.
• Uninstalling an extension with model-weight deletion enabled removes the extension model root once, including shared and private data.

UI changes

The Models UI and extension drawer now expose:

• Shared weight-group installation state.
• Shared versus node-private weight requirements.
• Nodes requiring additional private weights.
• Explicit shared-group installation and removal actions.
• Warnings listing the nodes affected by shared-group removal.

Manifest validation

The Electron and Python implementations apply equivalent normalization and validation rules.

Validation rejects:

• Duplicate group IDs.
• Unknown group references.
• Unsafe group or node IDs.
• "_shared" as a node ID.
• Unsafe destinations or checks.
• Sources that collide within the same physical root.
• Shared-weight metadata on non-model extensions.
• Attempts to reference groups outside the current extension.

Backward compatibility

Extensions that do not declare "weight_groups" retain their existing behavior and directory layout.

The following remain node-private and fully supported:

• Legacy "hf_repo".
• Legacy "download_check".
• Existing node-level "model_sources".

This PR does not migrate or deduplicate existing downloaded files automatically. Extension authors can provide migration guidance when adopting the new layout.

Test coverage

Regression coverage includes:

• Multiple nodes using one fully shared checkpoint.
• A shared base combined with a private adapter.
• Readiness propagation across dependent nodes.
• Install-all download deduplication.
• Canonical physical-target locking.
• Explicit runtime node identity.
• Direct and subprocess shared-directory propagation.
• Shared-group removal and dependent-node invalidation.
• Preservation of shared weights when private weights are removed.
• Manifest validation and path confinement.
• Compatibility with legacy manifests and node-owned directories.

Validation results

• Python: 118 tests passed.
• Node: 178 tests passed.
• Lint passed.
• Production build passed.

Out of scope

This first phase intentionally does not add:

• Weight sharing between unrelated extensions.
• A global or content-addressed model cache.
• Automatic migration of existing duplicate weights.
• Shared Python environments or extension processes.
• Arbitrary local filesystem paths.
• Non-Hugging-Face providers.

Cross-extension sharing can be designed separately once ownership, versioning, trust, reference counting and garbage-collection semantics have been defined.

[iammojogo-sudo]

Is this possible with different venv's though? Each venv has its own dependencies that are weight based for calculations in many cases. I would LOVE if this would work as it would reduce some memory and space on my laptop. Very curious how this plays out @DrHepa

[DrHepa]

Thanks, @iammojogo-sudo! Sharing checkpoint files doesn’t mean sharing a Python environment. This PR only reuses weights within one extension, whose nodes already share a venv. I checked our extensions: SD15/SDXL have matching dependency versions, but others need different Torch/CUDA or Transformers versions, so one shared venv isn’t a safe default. I’ve opened a separate proposal to keep venvs isolated while reusing verified package files where possible. That could save disk space—not RAM/VRAM.

[iammojogo-sudo]

" so one shared venv isn’t a safe default" exactly! That's why I was wondering. But it is good that "keep venvs isolated while reusing verified package files" because that right there could save a lot of re-downloading python dependencies at the least. Good stuff! If it can be made internal (in the .asar), that would be helpful for extension creators too, so that they don't have to code a shared communication across extensions. Could be a big deal! I love this.