Movexa

Migrate anything → anywhere → securely.

Movexa is a universal migration orchestrator — a planning + transfer engine that sits above mature point tools (rclone, croc, rsync) instead of replacing them. Give it a source, a destination, and a policy; it discovers what needs to move, picks the right transport, transfers with resumable checkpoints, verifies with Merkle-tree checksums, and rolls back cleanly on failure. Think Terraform for data migrations: movexa plan is a dry-run diff, movexa run applies it.

Local disks, remote hosts over SSH, S3-compatible object stores, SMB/CIFS shares, WebDAV, FTP, NFS exports, and Azure Blob/GCS/Google Drive/Dropbox/OneDrive are all supported today. The architecture is built to grow into containers, VMs, and databases without a rewrite — see ROADMAP.md.

Why

Point tools are excellent at one job each: rclone moves bytes to cloud backends, croc does secure ad-hoc P2P transfer, rsync does efficient POSIX sync. None of them plan, verify content end-to-end with a repairable Merkle tree, resume a killed multipart upload by reconciling against the real destination state, or expose the same job as a CLI command, a REST API call, and a live dashboard. Movexa is the orchestration layer that was missing — it calls into the same transports under the hood rather than reinventing them.

How it works

flowchart LR
    subgraph Engine["Engine — one Run per execution"]
        direction LR
        D[Discover] --> P[Plan] --> T[Transfer] --> V[Verify]
        V -.fatal / --atomic.-> R[Rollback]
        T -.crash.-> RS[Resume]
        RS --> T
    end
    Src[(Source\nfile / sftp / s3 / smb\nwebdav / ftp / nfs / cloud)] --> D
    T --> Dst[(Destination\nfile / sftp / s3 / smb\nwebdav / ftp / nfs / cloud)]
    Sel{{Transport\nselector}} -.chooses.-> T
    Store[(Postgres\nruns · items · chunks)] <-.state.-> Engine

Loading

Every stage transition is a row write in the same Postgres store as the job queue — so resume is "re-read the rows," not a separate subsystem. The destination is always treated as the source of truth on resume (S3 ListParts, or a re-hashed local partial file); Postgres is a reconciled hint, never trusted blindly. See docs/ENGINE.md for the full resume/verify design.

Source and destination connectors are a small, frozen interface (pkg/connector) that a compile-time registry wires up — no dynamic .so loading, one static binary. See docs/CONNECTOR_GUIDE.md to add a new one.

Quickstart

Option A — join the shared local_stack (recommended if you already run it)

# in your local_stack checkout:
docker compose --profile messaging up -d      # postgres, redis, minio, nats

# in this repo:
task stack-up                                  # builds + starts movexad, joined to local_stack_backend
docker compose exec movexad movexa config init
docker compose exec movexad movexa connectors  # → file, sftp, s3, smb, webdav, ftp, nfs with capability matrix

Option B — standalone (bundled Postgres + MinIO, no local_stack required)

docker compose -f docker-compose.yml up -d --build
docker compose exec movexad movexa connectors

Run your first migration

# job.yaml
version: "1"
name: demo
source: { uri: "file:///tmp/seed" }
dest:   { uri: "file:///tmp/dst" }
mode: oneshot
verify: { mode: checksum, hash: blake3 }

movexa plan job.yaml         # dry-run: what would move, and why this transport
movexa run  job.yaml --watch # do it
movexa verify --last         # re-check content + metadata

Job specs can also be written in HCL — movexa plan job.hcl / movexa run job.hcl dispatch automatically on a .hcl extension, decoding into the exact same job spec as YAML (see internal/spec/hcl.go). The REST API's POST /jobs still accepts YAML/JSON bodies only.

CLI reference

Command	Purpose
movexa plan <spec.yaml>	Discover + diff + select transport; print the plan. No data moves.
movexa run <spec.yaml>	Execute a plan. --detach --parallelism --bwlimit --transport --watch
movexa resume <run-id|--last>	Reconcile against the real destination and continue a killed run.
movexa verify <run-id>	--level L0..L3 --algo blake3|sha256 --repair
movexa status [run-id]	Live SSE progress table.
movexa ls <uri> / cp <src> <dst>	Ad-hoc listing / one-off copy.
movexa cancel <run-id>	Cooperative cancel, rollback if --atomic.
movexa connectors / transports	Capability matrices.
movexa connectors config add/list/get/rm	Manage named, reusable connector configs (the connector_config table) — required for drive:///dropbox:///onedrive://, optional elsewhere.
movexa auth <provider> <config-name>	Authorize an OAuth-based cloud connector by shelling out to the real rclone authorize; saves the resulting token into that connector config.
movexa serve	REST API + embedded worker.
movexa worker	Standalone pull-only worker (horizontal scale-out).
movexa audit	Query audit log entries for mutating operations (--target, --limit, --json).
movexa advise <job.yaml>	AI Migration Advisor: analyze job spec & recommend optimal transport, chunking, and concurrency (--json).
movexa gc	Sweep abandoned partials / multipart uploads.

Full flag reference: movexa <command> --help.

Connector matrix

Scheme	Status	Transports used
file://	✅	stream, same-fs reflink copy
sftp://	✅	stream; rsync shells out to the real rsync binary as a LAN accelerator, reachable only via an explicit transport: rsync pin (never auto-selected — see internal/selector)
s3://	✅	multipart, server-side copy
smb://	✅	stream
webdav:// / webdavs://	✅	stream, server-side copy (native WebDAV COPY)
ftp:// / ftps://	✅	stream (one data connection at a time, per the FTP protocol — see connectors/ftp)
nfs://	✅ code-complete	stream — live end-to-end verification against a real NFSv3 server is an open gap, see docs/DEFERRED.md
azureblob://	✅	stream, server-side copy (Copy Blob) — tested against Azurite
gcs://	✅	stream, server-side copy — tested against fake-gcs-server
drive:// / dropbox:// / onedrive://	✅ code-complete	stream, server-side copy — OAuth via movexa auth <provider> <config-name> (shells out to the real rclone authorize, see internal/cli/auth.go); no open-source emulator exists for any of the three, so live end-to-end verification against a real authorized account is an open gap, see docs/DEFERRED.md
docker://	✅	stream — helper container mount over Docker Engine API (github.com/moby/moby/client)
pvc://	✅	stream — helper pod mount over Kubernetes API (k8s.io/client-go)
qemu:// / kvm://	✅	stream, ranged read/write — QEMU/KVM disk image migration (.qcow2, .vmdk, .raw)
proxmox:// / pve://	✅	stream — Proxmox VE REST API VM drive & config migration
vmware:// / vsphere://	✅	stream, ranged read — VMware ESXi & vSphere datastore VM disk migration
postgres:// / postgresql://	✅	stream — PostgreSQL schema & table data migration (pgx/v5, pg_dump fallback)
mysql:// / mariadb://	✅	stream — MySQL table & schema data migration (go-sql-driver/mysql, mysqldump fallback)
mongodb:// / mongodb+srv://	✅	stream — MongoDB BSON document & collection migration (mongo-driver/v2)
redis:// / rediss://	✅	stream — Redis key-space & DUMP/RESTORE migration (go-redis/v9)
elasticsearch:// / es://	✅	stream — Elasticsearch & OpenSearch index document bulk migration

croc (WAN/NAT-traversal transport, PAKE-secured relay) is also implemented with retry/backoff, relay-death detection, and disk-polled progress reporting — see transports/croc's package doc comment for its current one-process-holds-both-ends scope.

Repository layout

See docs/ARCHITECTURE.md for the full module map and ownership rules. Short version: pkg/ is the frozen, dependency-free contract third parties import; internal/engine owns the six-stage pipeline; connectors/* and transports/* are independent, registry-wired packages.

Contributing / development

See AGENTS.md for house rules (build/lint/test commands, migration conventions, commit conventions). CI runs go build, go vet, golangci-lint, go test -race, and an end-to-end suite including a kill-9-mid-transfer resume test.

License

GNU AGPL-3.0-or-later. Movexa is a network service by design (the API server, movexad); AGPL keeps hosted forks obligated to share source, same as the running code. See NOTICE for third-party license attributions.