[WIP] Replication pipeline

[rkistner]

Previously, replication would generally happen in "batches", both for snapshot and streaming replication:

1. Read a batch of data.
2. Read source record state.
3. Evaluate sync config.
4. Write to object storage (if enabled).
5. Write to storage database.
6. Repeat with next batch.

There are lots of nuances, but conceptually the above is what happens. That means that we have lots of idle time: source database reads, CPU-heavy work, storage database work and object storage work are executed sequentially, leaving the resources idle for large portions of time.

This changes to a "pipeline" approach: Every stage can execute concurrently with every other stage. Stages pass finished work on to the next stage, then immediately continues. Back-pressure is used to pause stages if others get too far behind, bounding memory usage.

This opens up large gains in replication throughput, without needing to split the work in complex ways.

AI Usage

Vibe-coded using Codex gpt-6-astra. Still needs lots of manual review and rework to get production-ready.

[changeset-bot]

🦋 Changeset detected

Latest commit: 3ccf8dc

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@powersync/service-module-mongodb-storage	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-core	Patch
@powersync/service-schema	Patch
@powersync/service-module-convex	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres	Patch
@powersync/service-image	Patch
@powersync/service-module-core	Patch
@powersync/service-module-postgres-storage	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR