From 7f40fdd803411379073843852ce58447cf11baca Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Thu, 10 Sep 2026 15:10:49 +0200 Subject: [PATCH 01/20] New Sync Streams nav --- .../app-backend/client-side-integration.mdx | 2 +- .../source-db/postgres-maintenance.mdx | 2 +- debugging/error-codes.mdx | 2 +- docs.json | 61 ++-- snippets/binary-type.mdx | 2 +- sync/advanced/overview.mdx | 2 +- sync/advanced/storage-version-4.mdx | 2 +- sync/grammar/sync-streams/index.mdx | 2 +- sync/overview.mdx | 90 ------ sync/rules/client-parameters.mdx | 2 +- .../migrate-to-sync-streams.mdx} | 42 +-- sync/rules/overview.mdx | 2 +- sync/streams/client-usage.mdx | 6 +- sync/streams/examples.mdx | 2 +- sync/streams/overview.mdx | 297 +++--------------- sync/streams/quickstart.mdx | 276 ++++++++++++++++ 16 files changed, 374 insertions(+), 418 deletions(-) delete mode 100644 sync/overview.mdx rename sync/{streams/migration.mdx => rules/migrate-to-sync-streams.mdx} (69%) create mode 100644 sync/streams/quickstart.mdx diff --git a/configuration/app-backend/client-side-integration.mdx b/configuration/app-backend/client-side-integration.mdx index 8768f56b6..c8c76618c 100644 --- a/configuration/app-backend/client-side-integration.mdx +++ b/configuration/app-backend/client-side-integration.mdx @@ -10,7 +10,7 @@ After you've [instantiated](/intro/setup-guide#instantiate-the-powersync-databas | Purpose | Description | |---------|-------------| -| **Uploading mutations to your backend:** | Mutations that are made to the client-side SQLite database are uploaded to your backend application, where you control how they're applied to your backend source database (Postgres, MongoDB, MySQL, SQL Server, or Convex). This is how PowerSync achieves bi-directional syncing of data: The [PowerSync Service](/architecture/powersync-service) provides the _server-to-client read path_ based on your [Sync Streams or Sync Rules (legacy)](/sync/overview), and the _client-to-server write path_ goes via your backend. | +| **Uploading mutations to your backend:** | Mutations that are made to the client-side SQLite database are uploaded to your backend application, where you control how they're applied to your backend source database (Postgres, MongoDB, MySQL, SQL Server, or Convex). This is how PowerSync achieves bi-directional syncing of data: The [PowerSync Service](/architecture/powersync-service) provides the _server-to-client read path_ based on your [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)), and the _client-to-server write path_ goes via your backend. | | **Authentication integration:** (optional) | PowerSync uses JWTs for authentication between the Client SDK and PowerSync Service. Some [authentication providers](/configuration/auth/overview#common-authentication-providers) generate JWTs for users which PowerSync can verify directly. For others, some code must be [added to your application backend](/configuration/auth/custom) to generate the JWTs. | diff --git a/configuration/source-db/postgres-maintenance.mdx b/configuration/source-db/postgres-maintenance.mdx index f5ea3e3ef..263486cc4 100644 --- a/configuration/source-db/postgres-maintenance.mdx +++ b/configuration/source-db/postgres-maintenance.mdx @@ -7,7 +7,7 @@ description: "Manage Postgres replication slots and WAL lag for reliable PowerSy Postgres logical replication slots are used to keep track of [replication](/architecture/powersync-service#replication-from-the-source-database) progress (recorded as a [LSN](https://www.postgresql.org/docs/current/datatype-pg-lsn.html)). -Every time a new version of [Sync Streams or Sync Rules](/sync/overview) is deployed, PowerSync creates a new replication slot. Once the new version is fully processed, PowerSync switches to use the new slot and deletes the old one. The Service logs these steps and, during a snapshot, how much WAL budget remains. See [Postgres Replication Slots and WAL Budget](/debugging/log-reference#postgres-replication-slots-and-wal-budget) in the Log Reference. +Every time a new version of [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)) is deployed, PowerSync creates a new replication slot. Once the new version is fully processed, PowerSync switches to use the new slot and deletes the old one. The Service logs these steps and, during a snapshot, how much WAL budget remains. See [Postgres Replication Slots and WAL Budget](/debugging/log-reference#postgres-replication-slots-and-wal-budget) in the Log Reference. The replication slots can be viewed using this query: diff --git a/debugging/error-codes.mdx b/debugging/error-codes.mdx index affbf86b9..4bf69b795 100644 --- a/debugging/error-codes.mdx +++ b/debugging/error-codes.mdx @@ -9,7 +9,7 @@ This reference documents PowerSync error codes organized by component, with trou ## PSYNC_Rxxxx: Sync Config issues - **PSYNC_R0001**: - Catch-all [Sync Config](/sync/overview) parsing error, if no more specific error is available + Catch-all [Sync Config](/sync/streams/quickstart#defining-streams) parsing error, if no more specific error is available - **PSYNC_R2201**: A table or schema wildcard (`%`) is not supported by the configured source connector. diff --git a/docs.json b/docs.json index c57a8c129..2d20dbcd9 100644 --- a/docs.json +++ b/docs.json @@ -183,42 +183,23 @@ ] }, { - "group": "Sync Streams & Rules", + "group": "Sync Streams", "icon": "arrows-rotate", "pages": [ - "sync/overview", - { - "group": "Sync Streams", - "pages": [ - "sync/streams/overview", - "sync/streams/parameters", - "sync/streams/queries", - "sync/streams/ctes", - "sync/streams/bucket-count", - "sync/streams/examples", - "sync/streams/client-usage", - "sync/streams/migration" - ] - }, - { - "group": "Sync Rules (Legacy)", - "pages": [ - "sync/rules/overview", - "sync/rules/organize-data-into-buckets", - "sync/rules/global-buckets", - "sync/rules/parameter-queries", - "sync/rules/data-queries", - "sync/rules/many-to-many-join-tables", - "sync/rules/client-parameters" - ] - }, + "sync/streams/overview", + "sync/streams/quickstart", + "sync/streams/parameters", + "sync/streams/queries", + "sync/streams/ctes", + "sync/streams/bucket-count", + "sync/streams/client-usage", "sync/types", + "sync/streams/examples", { "group": "Supported SQL", "pages": [ "sync/supported-sql", - "sync/grammar/sync-streams/index", - "sync/grammar/sync-rules/index" + "sync/grammar/sync-streams/index" ] }, { @@ -237,6 +218,20 @@ "sync/advanced/partitioned-tables", "sync/advanced/sharded-databases" ] + }, + { + "group": "Sync Rules (Legacy)", + "pages": [ + "sync/rules/migrate-to-sync-streams", + "sync/rules/overview", + "sync/rules/organize-data-into-buckets", + "sync/rules/global-buckets", + "sync/rules/parameter-queries", + "sync/rules/data-queries", + "sync/rules/many-to-many-join-tables", + "sync/rules/client-parameters", + "sync/grammar/sync-rules/index" + ] } ] }, @@ -784,6 +779,14 @@ "source": "/usage/sync-streams", "destination": "/sync/streams/overview" }, + { + "source": "/sync/overview", + "destination": "/sync/streams/overview" + }, + { + "source": "/sync/streams/migration", + "destination": "/sync/rules/migrate-to-sync-streams" + }, { "source": "/usage/sync-rules/types", "destination": "/sync/types" diff --git a/snippets/binary-type.mdx b/snippets/binary-type.mdx index ace29f838..121b61735 100644 --- a/snippets/binary-type.mdx +++ b/snippets/binary-type.mdx @@ -1,3 +1,3 @@ - Binary data can be accessed in the Sync Streams / Sync Rules, but cannot be used as [parameters](/sync/overview#how-it-works). To sync binary columns/fields to clients, those columns need to be converted to hex or base64 representation using the relevant [functions](/sync/supported-sql#functions). + Binary data can be accessed in Sync Streams, but cannot be used as [parameters](/sync/streams/parameters). To sync binary columns/fields to clients, those columns need to be converted to hex or base64 representation using the relevant [functions](/sync/supported-sql#functions). \ No newline at end of file diff --git a/sync/advanced/overview.mdx b/sync/advanced/overview.mdx index de09cbf17..74a5d0707 100644 --- a/sync/advanced/overview.mdx +++ b/sync/advanced/overview.mdx @@ -1,6 +1,6 @@ --- title: "Advanced Topics" -description: "Advanced Sync Streams and Sync Rules topics." +description: "Advanced Sync Streams topics." sidebarTitle: Overview --- diff --git a/sync/advanced/storage-version-4.mdx b/sync/advanced/storage-version-4.mdx index 56d5922ba..2b6b4b9ed 100644 --- a/sync/advanced/storage-version-4.mdx +++ b/sync/advanced/storage-version-4.mdx @@ -23,7 +23,7 @@ The PowerSync Cloud and self-hosted columns below apply during the Beta only. On | Incremental reprocessing | MongoDB | Sync Streams | Included with version 4 | Included with version 4 | | S3 object storage | Any | Sync Streams or Sync Rules | Enabled per instance by PowerSync on request | [Set up S3 object storage](#self-hosted-s3-setup) | -Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/streams/migration). +Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/rules/migrate-to-sync-streams). ## Opt In diff --git a/sync/grammar/sync-streams/index.mdx b/sync/grammar/sync-streams/index.mdx index cc3e7f591..a4aab0fcb 100644 --- a/sync/grammar/sync-streams/index.mdx +++ b/sync/grammar/sync-streams/index.mdx @@ -1,5 +1,5 @@ --- -title: "Grammar Reference (Sync Streams)" +title: "Grammar Reference" description: "Railroad diagram reference for the SQL grammar supported in Sync Streams queries." --- diff --git a/sync/overview.mdx b/sync/overview.mdx deleted file mode 100644 index 8754bd87c..000000000 --- a/sync/overview.mdx +++ /dev/null @@ -1,90 +0,0 @@ ---- -title: "Sync Streams and Sync Rules" -sidebarTitle: "Overview" -description: "PowerSync Sync Streams and the legacy Sync Rules allow developers to control which data syncs to which clients/devices (i.e. they enable partial sync)." ---- - - -## Sync Streams — Recommended - -With [Sync Streams](/sync/streams/overview), you write SQL-like queries to define streams of data. Clients subscribe to the streams they need, either on-demand or automatically on connect. Sync Streams are the recommended path to achieve partial sync for both new and existing projects. - -Key improvements in Sync Streams over legacy Sync Rules include: -- **On-demand syncing**: You define Sync Streams on the PowerSync Service, and a client can then subscribe to them one or more times with different parameters, on-demand. You still have the option of auto-subscribing streams when a client connects, for "sync data upfront" behavior. -- **Temporary caching-like behavior**: Each subscription includes a configurable TTL that keeps data active after the client unsubscribes, acting as a warm cache for re-subscribing. -- **Simpler developer experience**: Simplified syntax and mental model, and capabilities such as your UI components automatically managing subscriptions (for example, React hooks). - -If you're on Sync Rules, you can migrate in a few clicks. Click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to generate a draft from your current config. See the [migration guide](/sync/streams/migration) for details. - - - - -## Sync Rules (Legacy) - -Sync Rules is the legacy approach for controlling data sync. It remains available and supported for existing projects: - - - -If you're currently using Sync Rules and want to migrate to Sync Streams, see our [migration docs](/sync/streams/migration). - - -## How It Works - -You may also find it useful to look at the [PowerSync Service architecture](/architecture/powersync-service) for background. - -Each [PowerSync Service](/architecture/powersync-service) instance has a deployed _Sync Streams_ (or legacy _Sync Rules_) configuration. This takes the form of a YAML file which contains: -- **In the case of Sync Streams:** Definitions of the streams that exist, with a SQL-like query (which can also contain limited subqueries), which defines the data in the stream, and references the necessary parameters. -- **In the case of Sync Rules:** Definitions of the different [buckets](/architecture/powersync-service#bucket-system) that exist, with SQL-like queries to specify the parameters used by each bucket (if any), as well as the data contained in each bucket. - - -A _parameter_ is a value that can be used in Sync Streams (or legacy Sync Rules) to create dynamic sync behavior for each user/client. Each client syncs only the relevant [_buckets_](/architecture/powersync-service#bucket-system) based on the parameters for that client. -* Sync Streams can make use of _authentication parameters_ from the JWT token (such as the user ID or other JWT claims), _connection parameters_ (specified at connection), and _subscription parameters_ (specified by the client when it subscribes to a stream at any time). See [Using Parameters](/sync/streams/parameters). -* Sync Rules can make use of _authentication parameters_ from the JWT token, as well as [_client parameters_](/sync/rules/client-parameters) (passed directly from the client when it connects to the PowerSync Service). - -It is also possible to have buckets/streams with no parameters. In the case of Sync Rules, these buckets sync to all users/clients automatically. - - - -The concept of _buckets_ is core to PowerSync and key to its performance and scalability. The [PowerSync Service architecture overview](/architecture/powersync-service) provides more background on this. -* In _Sync Streams_, buckets and parameters are implicit — they are automatically created based on the streams, their queries and subqueries. You don't need to explicitly define the buckets that exist. -* In legacy _Sync Rules_, buckets and their parameters are [explicitly defined](/sync/rules/overview#bucket-definition). - - -There are limitations on the SQL syntax and functionality that is supported in Sync Streams and Sync Rules. See [Supported SQL](/sync/supported-sql) for details and limitations. - -In addition to filtering data based on parameters, Sync Streams and Sync Rules also enable: - -* Selecting only specific tables/collections and columns/fields to sync. -* Filtering data based on static conditions. -* Transforming column/field names and values. - - -### Sync Streams/Rules Determine Replication From the Source Database - -A PowerSync Service instance [replicates and transforms](/architecture/powersync-service#replication-from-the-source-database) relevant data from your backend source database according to your Sync Streams/Sync Rules. During replication, data and metadata are persisted in [buckets](/architecture/powersync-service#bucket-system) on the PowerSync Service. Buckets are incrementally updated so that they contain the latest state as well as a history of changes (operations). This is key to how PowerSync achieves efficient delta syncing — having the operation history for each bucket allows clients to sync only the deltas that they need to get up to date (see [Protocol](/architecture/powersync-protocol#protocol) for more details). - - -As a practical example, let's say you have a bucket named `user_todo_lists` that contains the to-do lists for a user, and that bucket utilizes a `user_id` parameter (which will be embedded in the JWT). Now let's say users with IDs `A` and `B` exist in the source database. PowerSync will then replicate data from the source database and create individual buckets with IDs `user_todo_lists["A"]` and `user_todo_lists["B"]`. When the user with ID `A` connects, they can efficiently sync just the bucket with ID `user_todo_lists["A"]`. - - - - - - - -### Sync Streams/Rules Determine Real-Time Streaming Sync to Clients - -Whenever buckets are updated (buckets added or removed, or operations added to existing buckets), these changes are [streamed in real-time](/architecture/powersync-service#streaming-sync) to clients based on the Sync Streams/Sync Rules. - -This syncing behavior can be highly dynamic: in the case of Sync Streams, syncing will dynamically adjust based on the stream subscriptions (which can make use of _subscription parameters_), as well as _connection parameters_ and _authentication parameters_ (from the JWT). In the case of Sync Rules, syncing will dynamically adjust based on changes in _client parameters_ and _authentication parameters_. - -The bucket data is persisted in SQLite on the client-side, where it is easily queryable based on the [client-side schema](/intro/setup-guide#define-your-client-side-schema), which corresponds to the Sync Streams/Rules. - -For more information on the client-side SQLite database structure, see [Client Architecture](/architecture/client-architecture#client-side-schema-and-sqlite-database-structure). - - - - - - - diff --git a/sync/rules/client-parameters.mdx b/sync/rules/client-parameters.mdx index 08a814109..cdd33224b 100644 --- a/sync/rules/client-parameters.mdx +++ b/sync/rules/client-parameters.mdx @@ -16,7 +16,7 @@ PowerSync already supports using **token parameters** in parameter queries. An e [Sync Streams](/sync/streams/overview) make it easier to manage dynamic parameters, especially for apps where parameters are managed across different UI components and tabs. Sync Streams offer _subscription parameters_ (specified when subscribing to a stream) and _connection parameters_ (the equivalent of client parameters). - We recommend Sync Streams for new projects, and [migrating](/sync/streams/migration) existing projects. + We recommend Sync Streams for new projects, and [migrating](/sync/rules/migrate-to-sync-streams) existing projects. ### Usage diff --git a/sync/streams/migration.mdx b/sync/rules/migrate-to-sync-streams.mdx similarity index 69% rename from sync/streams/migration.mdx rename to sync/rules/migrate-to-sync-streams.mdx index 365cebf2b..31bbc0a0c 100644 --- a/sync/streams/migration.mdx +++ b/sync/rules/migrate-to-sync-streams.mdx @@ -1,41 +1,27 @@ --- -title: "Migrating from Sync Rules" -description: "Migrate existing projects from legacy Sync Rules to Sync Streams." +title: "Migrate to Sync Streams" +description: "Migrate an existing project from legacy Sync Rules to Sync Streams." --- import StreamDefinitionReference from '/snippets/stream-definition-reference.mdx'; -## Why Migrate? - -PowerSync's original Sync Rules system was optimized for offline-first use cases where you want to "sync everything upfront" when the client connects, so data is available locally if the user goes offline. - -However, many developers are building apps where users are mostly online, and you don't want to make users wait to sync a lot of data upfront. This is especially true for **web apps**: users are mostly online, you often want to sync only the data needed for the current page, and users frequently have multiple browser tabs open — each needing different subsets of data. - -### The Problem with Client Parameters - -[Client Parameters](/sync/rules/client-parameters) in Sync Rules partially support on-demand syncing — for example, using a `project_ids` array to sync only specific projects. However, manually managing these arrays across different browser tabs becomes painful: +Sync Streams do everything Sync Rules do, and more. A stream with `auto_subscribe: true` syncs when the client connects, the same way a bucket definition does, so apps that sync all relevant data upfront for offline use keep working the same way after migrating. The [migration tool](#migration-tool) sets `auto_subscribe: true` on every generated stream, so no client-side changes are required when you first deploy. -- You need to aggregate IDs across all open tabs -- You need additional logic for different data types (tables) -- If you want to keep data around after a tab closes (caching), you need even more management - -### How Sync Streams Solve This - -Sync Streams address these limitations: +## Why Migrate? -1. **On-demand syncing**: Define streams once, then subscribe from your app one or more times with different parameters. No need to manage arrays of IDs — each subscription is independent. +Beyond matching Sync Rules, Sync Streams add: -2. **Multi-tab support**: Each subscription manages its own lifecycle. Open the same list in two tabs? Each tab subscribes independently. Close one? The other keeps working. +1. **More expressive queries**: Stream queries support JOINs, [CTEs](/sync/streams/ctes), subqueries, and [multiple queries per stream](/sync/streams/queries#multiple-queries-per-stream), with syntax closer to plain SQL. [Parameter queries become inline subqueries](#data-with-subqueries-replaces-parameter-queries), so you write one query instead of separate `parameters:` and `data:` blocks. -3. **Built-in caching**: Each subscription has a configurable `ttl` that keeps data cached after unsubscribing. When users return to a screen, data may already be available — no loading state needed. +2. **On-demand syncing**: Define a stream once, then subscribe from your app one or more times with different parameters. Each subscription has its own lifecycle, so two screens or browser tabs can subscribe to the same stream independently. With Sync Rules, [Client Parameters](/sync/rules/client-parameters) approximate this, but you have to aggregate the parameter values yourself across screens and tabs, and remove them when they are no longer needed. -4. **Simpler, more powerful syntax**: Stream queries support JOINs, CTEs, subqueries, and multiple queries per stream, and the syntax is closer to plain SQL. For example, [parameter queries become inline subqueries](#data-with-subqueries-replaces-parameter-queries), so you write a single query instead of separate `parameters:` and `data:` blocks. +3. **Built-in caching**: Each subscription has a configurable `ttl` that keeps data on the device after unsubscribing. When users return to a screen, the data is often already available. -5. **Framework integration**: [React hooks, Vue composables, TanStack Query, and Kotlin Compose extensions](/sync/streams/client-usage#framework-integrations) let your UI components automatically manage subscriptions based on what's rendered. +4. **Framework integration**: [React hooks, Vue composables, TanStack Query, and Kotlin Compose extensions](/sync/streams/client-usage#framework-integrations) let UI components manage subscriptions based on what is rendered. -### Still Need Offline-First? +5. **Access to new features**: Newer PowerSync Service features such as [wildcard schemas](/sync/advanced/schemas-and-connections) and [incremental reprocessing](/sync/advanced/storage-version-4) require Sync Streams. -If you want "sync everything upfront" behavior (like Sync Rules), set [`auto_subscribe: true`](/sync/streams/overview#using-auto-subscribe) on your Sync Streams and clients will subscribe automatically when they connect. +You can migrate incrementally. Deploy the generated streams with `auto_subscribe: true` first, then convert individual streams to on-demand subscriptions where that benefits your app. ## Requirements @@ -117,7 +103,7 @@ The output uses `auto_subscribe: true` by default, preserving your existing sync ### Global Data (No Parameters) -In Sync Rules, a ["global" bucket](/sync/rules/global-buckets) syncs the same data to all users. In Sync Streams, you achieve this with queries that have no parameters. Add [`auto_subscribe: true`](/sync/streams/overview#using-auto-subscribe) to maintain the Sync Rules behavior where data syncs automatically on connect. +In Sync Rules, a ["global" bucket](/sync/rules/global-buckets) syncs the same data to all users. In Sync Streams, you achieve this with queries that have no parameters. Add [`auto_subscribe: true`](/sync/streams/quickstart#using-auto-subscribe) to maintain the Sync Rules behavior where data syncs automatically on connect. **Sync Rules:** ```yaml @@ -236,7 +222,7 @@ const page2 = await db.syncStream('posts', { page_number: 2 }).subscribe(); ## Client-Side Changes -After updating your Sync Config, update your client code to use subscriptions: +Streams generated by the migration tool with `auto_subscribe: true` need no client changes. When you convert a stream to on-demand syncing, replace connect-time parameters with a subscription: ```js // Before (Sync Rules with Client Parameters) @@ -249,4 +235,6 @@ await db.connect(connector); const sub = await db.syncStream('project_data', { project_id: projectId }).subscribe(); ``` +If you want to keep passing values at connect time instead, use [connection parameters](/sync/streams/parameters#connection-parameters). + See [Client-Side Usage](/sync/streams/client-usage) for detailed examples. diff --git a/sync/rules/overview.mdx b/sync/rules/overview.mdx index 6d85e0bcc..17921d0e2 100644 --- a/sync/rules/overview.mdx +++ b/sync/rules/overview.mdx @@ -11,7 +11,7 @@ Sync Rules are PowerSync's original system for results in fewer sync buckets. +If multiple streams share the same filtering logic, consider using [CTEs](/sync/streams/ctes) to avoid repetition and [multiple queries per stream](/sync/streams/queries#multiple-queries-per-stream) so the client only needs to manage one subscription instead of multiple. This is more efficient and results in fewer sync buckets. ### User's Default or Primary Item diff --git a/sync/streams/overview.mdx b/sync/streams/overview.mdx index 3ec4013ca..096b8fc3a 100644 --- a/sync/streams/overview.mdx +++ b/sync/streams/overview.mdx @@ -1,287 +1,66 @@ --- title: "Sync Streams" -description: "Introduction to Sync Streams, the recommended way to define which data syncs to each client with SQL-based stream definitions." -sidebarTitle: "Quickstart" +description: "Sync Streams define which data syncs to each client. Learn what a stream is and how PowerSync replicates and streams the data." +sidebarTitle: "Overview" --- -import StreamDefinitionReference from '/snippets/stream-definition-reference.mdx'; +With Sync Streams, you write SQL-like queries to define streams of data, and your client app subscribes to the streams it needs. This enables _partial sync_: each client syncs only the relevant subset of data, instead of the entire database. PowerSync keeps subscribed data synced in real-time to a client-side SQLite database, where it stays available when the device is offline. -With Sync Streams, you write simple SQL-like queries to define streams of data, and your client app subscribes to the streams it needs. This enables _partial sync_: each client syncs only the relevant subset of data, instead of the entire database. PowerSync handles the rest, keeping subscribed data synced in real-time to a client-side SQLite database, where it stays available even when the device is offline. - -For example, you might define a stream that syncs only the current user's to-do items, another for shared projects they have access to, and another for reference data that everyone needs. Your app subscribes to these streams on demand, and only that data syncs to the device. Offline-first apps that need all relevant data available upfront can use `auto_subscribe: true` so streams sync automatically when clients connect. +For example, you might define a stream that syncs only the current user's to-do items, another for shared projects they have access to, and another for reference data that everyone needs. Your app subscribes to these streams on demand, and only that data syncs to the device. Apps that need all relevant data available upfront can set `auto_subscribe: true` so streams sync automatically when clients connect. **Are you still using Sync Rules?** Sync Streams support everything Sync Rules do, plus more expressive queries (including JOIN support), on-demand syncing, and a simpler developer experience (e.g. React hooks that manage subscriptions automatically). -You can migrate in a few clicks. Click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to generate a draft from your current config. See the [migration guide](/sync/streams/migration) for details. +You can migrate in a few clicks. Click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to generate a draft from your current config. See [Migrate to Sync Streams](/sync/rules/migrate-to-sync-streams) for details. -## Defining Streams - -Streams are defined in a YAML configuration file. Each stream has a **name** and a **query** that specifies which rows to sync using SQL-like syntax. The query can reference [parameters](/sync/overview#how-it-works) like the authenticated user's ID to personalize what each user receives. - - - -In the [PowerSync Dashboard](https://dashboard.powersync.com/): - -1. Select your project and instance -2. Go to **Sync Streams** -3. Edit the YAML directly in the dashboard -4. Click **Deploy** to validate and deploy - -```yaml -config: - edition: 3 - -streams: - todos: - query: SELECT * FROM todos WHERE owner_id = auth.user_id() -``` - - - -Add a `sync_config` section to your `service.yaml`. Using a **separate file** is recommended (e.g. `sync_config: path: sync-config.yaml`). Put the stream definition in that file: - -```yaml sync-config.yaml -config: - edition: 3 +## How It Works -streams: - todos: - query: SELECT * FROM todos WHERE owner_id = auth.user_id() -``` +Each PowerSync Service instance has a deployed Sync Streams configuration: a YAML file that defines the streams that exist. Each stream has a name and a SQL-like query that selects the tables and columns to sync, filters rows by static conditions or by parameters, and can rename or transform columns. The Service uses this configuration in two places: when it replicates data from your source database into buckets, and when it streams those buckets to clients. -You can also use inline `sync_config: content: |` with the YAML nested in your main config. See [Self-Hosted Instance Configuration](/configuration/powersync-service/self-hosted-instances#sync_config) for both options. - - +See the [PowerSync Service architecture](/architecture/powersync-service) for more background. -Available stream options: +### Buckets and Parameters - +PowerSync groups replicated data into [buckets](/architecture/powersync-service#bucket-system): partitions of data that are synced as a unit. A stream creates one bucket for each unique value of its filter, such as each user ID matched by `auth.user_id()` or each `list_id` that a client subscribes with. A stream without parameters creates a single bucket that syncs the same data to every subscriber. -## Basic Examples +Buckets are implicit in Sync Streams. The Service creates them from your stream queries, parameters, and subqueries, and you do not define or name them yourself. See [Bucket Count](/sync/streams/bucket-count) for how queries determine the number of buckets. -There are two independent concepts to understand: +### Replication From the Source Database -- _What_ data the stream returns. For example: - - *Global data*: No parameters. Same data for all users (e.g. reference tables like categories). - - *Filtered data*: Filters the data by a parameter value. This can make use of _auth parameters_ from the JWT token (such as the user ID or other JWT claims), _subscription parameters_ (specified by the client when it subscribes to a stream at any time), or _connection parameters_ (specified at connection). Different users will get different sets of data based on the parameters. See [Using Parameters](/sync/streams/parameters) for the full reference. -- _When_ the client syncs the data - - *Auto-subscribe*: Client automatically subscribes on connect (`auto_subscribe: true`) - - *On-demand*: Client explicitly subscribes when needed (default behavior) - -### Global Data - -Data without parameters is "global" data, meaning the same data goes to all users/clients. This is useful for reference tables: - -```yaml -config: - edition: 3 - -streams: - # Same categories for everyone - categories: - query: SELECT * FROM categories - - # Same active products for everyone - products: - query: SELECT * FROM products WHERE active = true -``` +The Service [replicates and transforms](/architecture/powersync-service#replication-from-the-source-database) data from your source database according to your stream queries, and persists the data and metadata in buckets. Buckets are updated incrementally, so they contain the latest state as well as a history of changes (operations). This operation history allows clients to sync only the deltas they need to get up to date. See [Protocol](/architecture/powersync-protocol#protocol) for details. -Global data streams still require clients to subscribe explicitly unless you set `auto_subscribe: true` +For example, a stream `user_lists` with the query `SELECT * FROM lists WHERE owner_id = auth.user_id()` creates one bucket per user. If users `A` and `B` exist in the source database, the Service creates a bucket for each of them. When user `A` connects and subscribes, they sync only their own bucket. -### Filtering Data by User - -Use `auth.user_id()` or other [JWT claims](/sync/streams/parameters#auth-parameters) to return different data per user: - -```yaml -config: - edition: 3 - -streams: - # Each user gets their own lists - my_lists: - query: SELECT * FROM lists WHERE owner_id = auth.user_id() - - # Each user gets their own orders - my_orders: - query: SELECT * FROM orders WHERE user_id = auth.user_id() -``` - -### Filtering Data Based on Subscription Parameters - -Use `subscription.parameter()` for data that clients subscribe to explicitly: - -```yaml -config: - edition: 3 - -streams: - # Sync todos for a specific list when the client subscribes with a list_id - list_todos: - query: | - SELECT * FROM todos - WHERE list_id = subscription.parameter('list_id') - AND list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) -``` - -```js -// Client subscribes with the list they want to view -const sub = await db.syncStream('list_todos', { list_id: 'abc123' }).subscribe(); -``` - -### Using Auto-Subscribe - -Set `auto_subscribe: true` to sync data automatically when clients connect. This is useful for: -- Reference data that all users need, or that are needed in many screens in the app. -- User data that should always be available offline -- Maintaining [Sync Rules](/sync/rules/overview) default behavior ("sync everything upfront") when migrating to Sync Streams - -```yaml -config: - edition: 3 - -streams: - # Global data, synced automatically - categories: - auto_subscribe: true - query: SELECT * FROM categories - - # User-scoped data, synced automatically - my_orders: - auto_subscribe: true - query: SELECT * FROM orders WHERE user_id = auth.user_id() - - # Parameterized data, subscribed on-demand (no auto_subscribe) - order_items: - query: | - SELECT * FROM order_items - WHERE order_id = subscription.parameter('order_id') - AND order_id IN (SELECT id FROM orders WHERE user_id = auth.user_id()) -``` - - -## Client-Side Usage - -Subscribe to streams from your client app: - - - -```js -const sub = await db.syncStream('list_todos', { list_id: 'abc123' }) - .subscribe({ ttl: 3600 }); - -// Wait for this subscription to have synced -await sub.waitForFirstSync(); - -// When the component needing the subscription is no longer active... -sub.unsubscribe(); -``` - -**React hooks:** - -```jsx -const stream = useSyncStream({ name: 'list_todos', parameters: { list_id: 'abc123' } }); -// Check download progress or subscription information -stream?.progress; -stream?.subscription.hasSynced; -``` - -The `useQuery` hook can wait for Sync Streams before running queries: - -```jsx -const { data } = useQuery( - 'SELECT * FROM todos WHERE list_id = ?', - [listId], - { streams: [{ name: 'list_todos', parameters: { list_id: listId }, waitForStream: true }] } -); -``` - - - -```dart -final sub = await db - .syncStream('list_todos', {'list_id': 'abc123'}) - .subscribe(ttl: const Duration(hours: 1)); - -// Wait for this subscription to have synced -await sub.waitForFirstSync(); - -// When the component needing the subscription is no longer active... -sub.unsubscribe(); -``` - - - -```kotlin -val sub = database.syncStream("list_todos", mapOf("list_id" to JsonParam.String("abc123"))) - .subscribe(ttl = 1.0.hours) - -// Wait for this subscription to have synced -sub.waitForFirstSync() - -// When the component needing the subscription is no longer active... -sub.unsubscribe() -``` - - - - -```swift -let sub = try await db.syncStream(name: "list_todos", params: ["list_id": JsonValue.string("abc123")]) - .subscribe(ttl: 60 * 60, priority: nil) // 1 hour - -// Wait for this subscription to have synced -try await sub.waitForFirstSync() - -// When the component needing the subscription is no longer active... -try await sub.unsubscribe() -``` - - - -```csharp -var sub = await db.SyncStream("list_todos", new() { ["list_id"] = "abc123" }) - .Subscribe(new SyncStreamSubscribeOptions { Ttl = TimeSpan.FromHours(1) }); - -// Wait for this subscription to have synced -await sub.WaitForFirstSync(); - -// When the component needing the subscription is no longer active... -sub.Unsubscribe(); -``` - - - -### TTL (Time-To-Live) - -Each subscription has a `ttl` that keeps data cached after unsubscribing. This enables warm cache behavior — when users return to a screen and you re-subscribe to relevant streams, data is already available on the client. Default TTL is 24 hours. See [Client-Side Usage](/sync/streams/client-usage) for details. - -```js -// Set TTL in seconds when subscribing -const sub = await db.syncStream('todos', { list_id: 'abc' }) - .subscribe({ ttl: 3600 }); // Cache for 1 hour after unsubscribe -``` -## Developer Notes - -- **SQL Syntax**: Stream queries use a SQL-like syntax with `SELECT` statements. You can use subqueries, `INNER JOIN`, and [CTEs](/sync/streams/ctes) for filtering. `GROUP BY`, `ORDER BY`, and `LIMIT` are not supported. See [Writing Queries](/sync/streams/queries) for details on joins, multiple queries per stream, and other features. - -- **Type Conversion**: Data types from your source database (Postgres, MongoDB, MySQL, SQL Server or Convex) are converted when synced to the client's SQLite database. SQLite has a limited type system, so most types become `text` and you may need to parse or cast values in your app code. See [Type Mapping](/sync/types) for details on how each type is handled. - -- **Primary Key**: PowerSync requires every synced table to have a primary key column named `id` of type `text`. If your backend uses a different column name or type, you'll need to map it. For MongoDB, collections use `_id` as the ID field; you must alias it in your stream queries (e.g. `SELECT *, _id as id FROM your_collection`). - -- **Case Sensitivity**: To avoid issues across different databases and platforms, use **lowercase identifiers** for all table and column names in your Sync Streams. If your backend uses mixed case, see [Case Sensitivity](/sync/advanced/case-sensitivity) for how to handle it. + + + -- **Bucket Limits**: PowerSync uses internal partitions called [buckets](/architecture/powersync-service#bucket-system) to efficiently sync data. Each user has a limit on how many buckets they can sync ([1,000 by default](/resources/performance-and-limits)), and the query pattern determines how many each stream creates. See [Bucket Count](/sync/streams/bucket-count) for how buckets are counted, and [Reducing Bucket Count](/sync/advanced/reducing-bucket-count) to resolve `PSYNC_S2305` errors. +### Streaming Sync to Clients -- **Troubleshooting**: If data isn't syncing as expected, the [Sync Diagnostics Client](/tools/diagnostics-client) helps you inspect what's happening for a specific user — you can see which buckets the user has and what data is being synced. +Whenever buckets change (buckets are added or removed, or operations are added to existing buckets), the Service [streams these changes in real-time](/architecture/powersync-service#streaming-sync) to the subscribed clients. The set of buckets a client receives adjusts as it subscribes to and unsubscribes from streams, and depends on its subscription, connection, and authentication parameters. -## Examples & Demos +On the client, bucket data is persisted in SQLite, where you query it through your [client-side schema](/intro/setup-guide#define-your-client-side-schema). See [Client Architecture](/architecture/client-architecture#client-side-schema-and-sqlite-database-structure) for the database structure. -See [Examples & Demos](/sync/streams/examples) for working demo apps and complete application patterns. + + + -## Migrating from Legacy Sync Rules +## Next Steps -If you have an existing project using legacy Sync Rules, see the [Migration Guide](/sync/streams/migration) for step-by-step instructions, syntax changes, and examples. + + + Define your first streams and subscribe to them from your app. + + + Filter data with auth, subscription, and connection parameters. + + + Query syntax, joins, subqueries, and multiple queries per stream. + + + Manage subscriptions in each SDK and UI framework. + + diff --git a/sync/streams/quickstart.mdx b/sync/streams/quickstart.mdx new file mode 100644 index 000000000..c8ae8cd68 --- /dev/null +++ b/sync/streams/quickstart.mdx @@ -0,0 +1,276 @@ +--- +title: "Sync Streams Quickstart" +description: "Define your first Sync Streams and subscribe to them from your client app." +sidebarTitle: "Quickstart" +--- + +import StreamDefinitionReference from '/snippets/stream-definition-reference.mdx'; + +This page shows how to define streams and subscribe to them from your app. For what streams are and how PowerSync replicates and syncs them, see the [Sync Streams overview](/sync/streams/overview). + +## Defining Streams + +Streams are defined in a YAML configuration file. Each stream has a **name** and a **query** that specifies which rows to sync using SQL-like syntax. The query can reference [parameters](/sync/streams/parameters) like the authenticated user's ID to personalize what each user receives. + + + +In the [PowerSync Dashboard](https://dashboard.powersync.com/): + +1. Select your project and instance +2. Go to **Sync Streams** +3. Edit the YAML directly in the dashboard +4. Click **Deploy** to validate and deploy + +```yaml +config: + edition: 3 + +streams: + todos: + query: SELECT * FROM todos WHERE owner_id = auth.user_id() +``` + + + +Add a `sync_config` section to your `service.yaml`. Using a **separate file** is recommended (e.g. `sync_config: path: sync-config.yaml`). Put the stream definition in that file: + +```yaml sync-config.yaml +config: + edition: 3 + +streams: + todos: + query: SELECT * FROM todos WHERE owner_id = auth.user_id() +``` + +You can also use inline `sync_config: content: |` with the YAML nested in your main config. See [Self-Hosted Instance Configuration](/configuration/powersync-service/self-hosted-instances#sync_config) for both options. + + + +Available stream options: + + + +## Basic Examples + +There are two independent concepts to understand: + +- _What_ data the stream returns. For example: + - *Global data*: No parameters. Same data for all users (e.g. reference tables like categories). + - *Filtered data*: Filters the data by a parameter value. This can make use of _auth parameters_ from the JWT token (such as the user ID or other JWT claims), _subscription parameters_ (specified by the client when it subscribes to a stream at any time), or _connection parameters_ (specified at connection). Different users will get different sets of data based on the parameters. See [Using Parameters](/sync/streams/parameters) for the full reference. +- _When_ the client syncs the data + - *Auto-subscribe*: Client automatically subscribes on connect (`auto_subscribe: true`) + - *On-demand*: Client explicitly subscribes when needed (default behavior) + +### Global Data + +Data without parameters is "global" data, meaning the same data goes to all users/clients. This is useful for reference tables: + +```yaml +config: + edition: 3 + +streams: + # Same categories for everyone + categories: + query: SELECT * FROM categories + + # Same active products for everyone + products: + query: SELECT * FROM products WHERE active = true +``` + + +Global data streams still require clients to subscribe explicitly unless you set `auto_subscribe: true` + + +### Filtering Data by User + +Use `auth.user_id()` or other [JWT claims](/sync/streams/parameters#auth-parameters) to return different data per user: + +```yaml +config: + edition: 3 + +streams: + # Each user gets their own lists + my_lists: + query: SELECT * FROM lists WHERE owner_id = auth.user_id() + + # Each user gets their own orders + my_orders: + query: SELECT * FROM orders WHERE user_id = auth.user_id() +``` + +### Filtering Data Based on Subscription Parameters + +Use `subscription.parameter()` for data that clients subscribe to explicitly: + +```yaml +config: + edition: 3 + +streams: + # Sync todos for a specific list when the client subscribes with a list_id + list_todos: + query: | + SELECT * FROM todos + WHERE list_id = subscription.parameter('list_id') + AND list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) +``` + +```js +// Client subscribes with the list they want to view +const sub = await db.syncStream('list_todos', { list_id: 'abc123' }).subscribe(); +``` + +### Using Auto-Subscribe + +Set `auto_subscribe: true` to sync data automatically when clients connect. This is useful for: +- Reference data that all users need, or that is needed in many screens in the app. +- User data that should always be available offline. +- Keeping the "sync everything upfront" behavior of legacy [Sync Rules](/sync/rules/overview) when migrating to Sync Streams. + +```yaml +config: + edition: 3 + +streams: + # Global data, synced automatically + categories: + auto_subscribe: true + query: SELECT * FROM categories + + # User-scoped data, synced automatically + my_orders: + auto_subscribe: true + query: SELECT * FROM orders WHERE user_id = auth.user_id() + + # Parameterized data, subscribed on-demand (no auto_subscribe) + order_items: + query: | + SELECT * FROM order_items + WHERE order_id = subscription.parameter('order_id') + AND order_id IN (SELECT id FROM orders WHERE user_id = auth.user_id()) +``` + + +## Client-Side Usage + +Subscribe to streams from your client app: + + + +```js +const sub = await db.syncStream('list_todos', { list_id: 'abc123' }) + .subscribe({ ttl: 3600 }); + +// Wait for this subscription to have synced +await sub.waitForFirstSync(); + +// When the component needing the subscription is no longer active... +sub.unsubscribe(); +``` + +**React hooks:** + +```jsx +const stream = useSyncStream({ name: 'list_todos', parameters: { list_id: 'abc123' } }); +// Check download progress or subscription information +stream?.progress; +stream?.subscription.hasSynced; +``` + +The `useQuery` hook can wait for Sync Streams before running queries: + +```jsx +const { data } = useQuery( + 'SELECT * FROM todos WHERE list_id = ?', + [listId], + { streams: [{ name: 'list_todos', parameters: { list_id: listId }, waitForStream: true }] } +); +``` + + + +```dart +final sub = await db + .syncStream('list_todos', {'list_id': 'abc123'}) + .subscribe(ttl: const Duration(hours: 1)); + +// Wait for this subscription to have synced +await sub.waitForFirstSync(); + +// When the component needing the subscription is no longer active... +sub.unsubscribe(); +``` + + + +```kotlin +val sub = database.syncStream("list_todos", mapOf("list_id" to JsonParam.String("abc123"))) + .subscribe(ttl = 1.0.hours) + +// Wait for this subscription to have synced +sub.waitForFirstSync() + +// When the component needing the subscription is no longer active... +sub.unsubscribe() +``` + + + + +```swift +let sub = try await db.syncStream(name: "list_todos", params: ["list_id": JsonValue.string("abc123")]) + .subscribe(ttl: 60 * 60, priority: nil) // 1 hour + +// Wait for this subscription to have synced +try await sub.waitForFirstSync() + +// When the component needing the subscription is no longer active... +try await sub.unsubscribe() +``` + + + +```csharp +var sub = await db.SyncStream("list_todos", new() { ["list_id"] = "abc123" }) + .Subscribe(new SyncStreamSubscribeOptions { Ttl = TimeSpan.FromHours(1) }); + +// Wait for this subscription to have synced +await sub.WaitForFirstSync(); + +// When the component needing the subscription is no longer active... +sub.Unsubscribe(); +``` + + + +### TTL (Time-To-Live) + +Each subscription has a `ttl` that keeps data cached after unsubscribing. This enables warm cache behavior: when users return to a screen and you re-subscribe to relevant streams, data is already available on the client. Default TTL is 24 hours. See [Client-Side Usage](/sync/streams/client-usage) for details. + +```js +// Set TTL in seconds when subscribing +const sub = await db.syncStream('todos', { list_id: 'abc' }) + .subscribe({ ttl: 3600 }); // Cache for 1 hour after unsubscribe +``` + +## Developer Notes + +- **SQL Syntax**: Stream queries use a SQL-like syntax with `SELECT` statements. You can select specific columns, filter on static conditions, rename or transform columns, and use subqueries, `INNER JOIN`, and [CTEs](/sync/streams/ctes) for filtering. `GROUP BY`, `ORDER BY`, and `LIMIT` are not supported. See [Writing Queries](/sync/streams/queries) for details and [Supported SQL](/sync/supported-sql) for the operators and functions you can use. + +- **Type Conversion**: Data types from your source database (Postgres, MongoDB, MySQL, SQL Server or Convex) are converted when synced to the client's SQLite database. SQLite has a limited type system, so most types become `text` and you may need to parse or cast values in your app code. See [Type Mapping](/sync/types) for details on how each type is handled. + +- **Primary Key**: PowerSync requires every synced table to have a primary key column named `id` of type `text`. If your backend uses a different column name or type, you'll need to map it. For MongoDB, collections use `_id` as the ID field; you must alias it in your stream queries (e.g. `SELECT *, _id as id FROM your_collection`). + +- **Case Sensitivity**: To avoid issues across different databases and platforms, use **lowercase identifiers** for all table and column names in your Sync Streams. If your backend uses mixed case, see [Case Sensitivity](/sync/advanced/case-sensitivity) for how to handle it. + +- **Bucket Limits**: Each user has a limit on how many buckets they can sync ([1,000 by default](/resources/performance-and-limits)), and the query pattern determines how many each stream creates. See [Bucket Count](/sync/streams/bucket-count) for how buckets are counted, and [Reducing Bucket Count](/sync/advanced/reducing-bucket-count) to resolve `PSYNC_S2305` errors. + +- **Troubleshooting**: If data isn't syncing as expected, the [Sync Diagnostics Client](/tools/diagnostics-client) helps you inspect what's happening for a specific user. You can see which buckets the user has and what data is being synced. + +## Examples & Demos + +See [Examples & Demos](/sync/streams/examples) for working demo apps and complete application patterns. From f36b02758158410e303d20b0245c8f9ec6ea15d0 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Wed, 16 Sep 2026 09:25:45 +0200 Subject: [PATCH 02/20] Update image directory --- sync/streams/overview.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sync/streams/overview.mdx b/sync/streams/overview.mdx index 096b8fc3a..5fbe3ba2a 100644 --- a/sync/streams/overview.mdx +++ b/sync/streams/overview.mdx @@ -35,7 +35,7 @@ For example, a stream `user_lists` with the query `SELECT * FROM lists WHERE own - + ### Streaming Sync to Clients @@ -45,7 +45,7 @@ Whenever buckets change (buckets are added or removed, or operations are added t On the client, bucket data is persisted in SQLite, where you query it through your [client-side schema](/intro/setup-guide#define-your-client-side-schema). See [Client Architecture](/architecture/client-architecture#client-side-schema-and-sqlite-database-structure) for the database structure. - + ## Next Steps From 23be733ad56708b9926612af4f286a659c50a3e8 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Wed, 16 Sep 2026 13:07:06 +0200 Subject: [PATCH 03/20] Improve migration guide - better flow, more practical examples --- sync/rules/migrate-to-sync-streams.mdx | 260 +++++++++++++++---------- 1 file changed, 153 insertions(+), 107 deletions(-) diff --git a/sync/rules/migrate-to-sync-streams.mdx b/sync/rules/migrate-to-sync-streams.mdx index 31bbc0a0c..ba32ac46d 100644 --- a/sync/rules/migrate-to-sync-streams.mdx +++ b/sync/rules/migrate-to-sync-streams.mdx @@ -1,33 +1,36 @@ --- title: "Migrate to Sync Streams" -description: "Migrate an existing project from legacy Sync Rules to Sync Streams." +description: "Convert legacy Sync Rules to Sync Streams without changing what your app syncs, then adopt on-demand syncing over time." --- import StreamDefinitionReference from '/snippets/stream-definition-reference.mdx'; -Sync Streams do everything Sync Rules do, and more. A stream with `auto_subscribe: true` syncs when the client connects, the same way a bucket definition does, so apps that sync all relevant data upfront for offline use keep working the same way after migrating. The [migration tool](#migration-tool) sets `auto_subscribe: true` on every generated stream, so no client-side changes are required when you first deploy. -## Why Migrate? - -Beyond matching Sync Rules, Sync Streams add: +Sync Streams support everything Sync Rules do (and more), and migrating does not change what your app syncs. The [migration tool](#migrate-with-the-migration-tool) converts your bucket definitions into streams with the same behavior: -1. **More expressive queries**: Stream queries support JOINs, [CTEs](/sync/streams/ctes), subqueries, and [multiple queries per stream](/sync/streams/queries#multiple-queries-per-stream), with syntax closer to plain SQL. [Parameter queries become inline subqueries](#data-with-subqueries-replaces-parameter-queries), so you write one query instead of separate `parameters:` and `data:` blocks. +- Every generated stream has `auto_subscribe: true`, so clients keep syncing all their data when they connect, exactly as they do with Sync Rules. +- [Client Parameters](/sync/rules/client-parameters) become [connection parameters](/sync/streams/parameters#connection-parameters). Your app passes them the same way when it connects. +- Once your SDKs meet the [minimum versions](#requirements), no client-side code changes are needed. -2. **On-demand syncing**: Define a stream once, then subscribe from your app one or more times with different parameters. Each subscription has its own lifecycle, so two screens or browser tabs can subscribe to the same stream independently. With Sync Rules, [Client Parameters](/sync/rules/client-parameters) approximate this, but you have to aggregate the parameter values yourself across screens and tabs, and remove them when they are no longer needed. +In most cases you migrate to stay compatible first, then [adopt Sync Streams features](#adopt-sync-streams-features-over-time) such as on-demand syncing one stream at a time. -3. **Built-in caching**: Each subscription has a configurable `ttl` that keeps data on the device after unsubscribing. When users return to a screen, the data is often already available. +If your Sync Config has a `bucket_definitions:` section, you use Sync Rules and this guide applies to you. If it only has `streams:`, you already use Sync Streams and no action is needed. -4. **Framework integration**: [React hooks, Vue composables, TanStack Query, and Kotlin Compose extensions](/sync/streams/client-usage#framework-integrations) let UI components manage subscriptions based on what is rendered. +## Why Migrate? -5. **Access to new features**: Newer PowerSync Service features such as [wildcard schemas](/sync/advanced/schemas-and-connections) and [incremental reprocessing](/sync/advanced/storage-version-4) require Sync Streams. +Beyond matching Sync Rules, Sync Streams add: -You can migrate incrementally. Deploy the generated streams with `auto_subscribe: true` first, then convert individual streams to on-demand subscriptions where that benefits your app. +- **More expressive queries:** Stream queries support JOINs, [CTEs](/sync/streams/ctes), subqueries, and [multiple queries per stream](/sync/streams/queries#multiple-queries-per-stream), with syntax closer to plain SQL. You write one query instead of separate `parameters:` and `data:` blocks. +- **On-demand syncing:** Define a stream once, then subscribe from your app one or more times with different parameters. Each subscription has its own lifecycle, so two screens or browser tabs can subscribe to the same stream independently. With Sync Rules, Client Parameters approximate this. You have to aggregate the parameter values yourself across screens and tabs, and remove them when they are no longer needed. +- **Built-in caching:** Each subscription has a configurable `ttl` that keeps data on the device after unsubscribing. When users return to a screen, the data is often already available. +- **Framework integration:** [React hooks, Vue composables, TanStack Query, and Kotlin Compose extensions](/sync/streams/client-usage#framework-integrations) let UI components manage subscriptions based on what is rendered. +- **Access to new features:** Newer PowerSync Service features such as [incremental reprocessing](/sync/advanced/storage-version-4) require Sync Streams. ## Requirements - PowerSync Service v1.20.0+ (Cloud instances already meet this) -- Latest SDK versions with [Rust-based sync client](https://releases.powersync.com/announcements/improved-sync-performance-in-our-client-sdks) (enabled by default on latest SDKs) -- `config: edition: 3` in your Sync Config +- An SDK version that supports Sync Streams (see table). Streams run on the [Rust-based sync client](https://releases.powersync.com/announcements/improved-sync-performance-in-our-client-sdks), which is the default in current SDKs. If your version is between the two columns, enable it manually. +- `config: edition: 3` in your Sync Config (the migration tool sets this) @@ -83,132 +86,188 @@ try await db.connect(connector: connector, options: ConnectOptions( -## Migration Tool - -You can generate a Sync Streams draft from your existing Sync Rules in two ways: - -1. **Dashboard:** In the [PowerSync Dashboard](https://dashboard.powersync.com/), use the **Migrate to Sync Streams** button. It converts your Sync Rules into a Sync Streams draft that you can review before deploying. - -2. **CLI:** Run `powersync migrate sync-rules` to produce a Sync Streams draft from your current Sync Config. - -The output uses `auto_subscribe: true` by default, preserving your existing sync-everything-upfront behavior so no client-side changes are required when you first deploy. - -**Next steps:** Review the draft, then deploy it (via the Dashboard or `powersync deploy sync-config`). After that, you can optionally migrate individual streams to on-demand subscriptions over time — remove `auto_subscribe: true` from specific streams and update client code to use the `syncStream()` API where it makes sense for your app. - -## Stream Definition Reference +## Migrate With the Migration Tool - + + + Use one of the following: -## Migration Examples + - **PowerSync Dashboard:** Click **Migrate to Sync Streams**. The Dashboard converts the instance's deployed Sync Rules and opens the result as a draft for you to review. + - **CLI:** Run `powersync migrate sync-rules`. By default the command reads `sync-config.yaml` in your linked project directory and overwrites it with the result. Use `--input-file` and `--output-file` to read from and write to other paths. See the [CLI reference](/tools/cli). + + + Compare the draft with your Sync Rules. See [What the Tool Generates](#what-the-tool-generates) for how the output maps to your bucket definitions, and [What to Check Before You Deploy](#what-to-check-before-you-deploy) for the items that need your attention. + + + Deploy the draft from the Dashboard or with `powersync deploy sync-config`. This works like any other Sync Config deploy: the Service reprocesses your data in the background while the current version keeps serving clients, then switches over without downtime. After the switch, each client does a one-time full re-sync. + + -### Global Data (No Parameters) +### What the Tool Generates -In Sync Rules, a ["global" bucket](/sync/rules/global-buckets) syncs the same data to all users. In Sync Streams, you achieve this with queries that have no parameters. Add [`auto_subscribe: true`](/sync/streams/quickstart#using-auto-subscribe) to maintain the Sync Rules behavior where data syncs automatically on connect. +The following Sync Rules define global data, user-scoped data, a parameter query that reads from a table, and a Client Parameter: -**Sync Rules:** ```yaml bucket_definitions: global: data: - - SELECT * FROM todos - - SELECT * FROM lists WHERE archived = false + - SELECT * FROM categories + user_lists: + parameters: SELECT request.user_id() as user_id + data: + - SELECT * FROM lists WHERE owner_id = bucket.user_id + list_todos: + parameters: SELECT id as list_id FROM lists WHERE owner_id = request.user_id() + data: + - SELECT * FROM todos WHERE list_id = bucket.list_id + page_posts: + parameters: SELECT request.parameters() ->> 'page_number' as page_number + data: + - SELECT * FROM posts WHERE page_number = bucket.page_number ``` -**Sync Streams:** +The migration tool converts them to: + ```yaml config: edition: 3 - streams: - shared_data: - auto_subscribe: true # Sync automatically like Sync Rules + migrated_to_streams: + auto_subscribe: true + with: + list_todos_param: SELECT id AS list_id FROM lists WHERE owner_id = auth.user_id() queries: - - SELECT * FROM todos - - SELECT * FROM lists WHERE archived = false + # Translated from "global" bucket definition. + - SELECT * FROM categories + # Translated from "user_lists" bucket definition. + - SELECT * FROM lists WHERE owner_id = auth.user_id() + # Translated from "list_todos" bucket definition. + - "SELECT todos.* FROM todos,list_todos_param AS bucket WHERE todos.list_id = bucket.list_id" + # Translated from "page_posts" bucket definition. + - SELECT * FROM posts WHERE page_number = connection.parameter('page_number') ``` - -Without `auto_subscribe: true`, clients would need to explicitly subscribe to these streams. This gives you flexibility to migrate incrementally or switch to on-demand syncing later. - +The tool applies these rules: -### User-Scoped Data +- **Compatibility edition:** It sets `config: edition: 3`, which Sync Streams require, and keeps any other options in your `config` block. +- **One stream per priority:** Bucket definitions with the same [priority](/sync/advanced/prioritized-sync) are merged into one stream named `migrated_to_streams`. Comments mark which bucket definition each group of queries came from. If your bucket definitions use different priorities, the tool creates one stream per priority, named `migrated_to_streams_prio_`. +- **Same sync behavior:** Every stream has `auto_subscribe: true`. Queries are always written as a `queries:` list so that you can add more. +- **Parameters:** `request.*` functions become `auth.*` and `connection.*` functions. See [Parameter Syntax Changes](#parameter-syntax-changes) for the full mapping. Parameter queries that only select request values, such as `SELECT request.user_id() as user_id`, are replaced by those values in the data queries: `bucket.user_id` becomes `auth.user_id()`. Parameter queries that read from a table become CTEs in a `with:` block, named `_param`, and the data queries join them under the alias `bucket`. +- **Cleanup:** The `bucket_definitions:` section is removed. -**Sync Rules:** -```yaml -bucket_definitions: - user_lists: - priority: 1 - parameters: SELECT request.user_id() as user_id - data: - - SELECT * FROM lists WHERE owner_id = bucket.user_id -``` +### What to Check Before You Deploy -**Sync Streams:** -```yaml -config: - edition: 3 +- **Compatibility edition:** If your Sync Rules had no `edition` set, `edition: 3` also turns on the edition 2 fixes, such as ISO 8601 timestamp formatting and custom Postgres type handling. These change how some values look in the client database. See [Compatibility](/sync/advanced/compatibility) for the full list. To keep the old behavior for a fix, set its option to `false` next to the edition: + + ```yaml + config: + edition: 3 + timestamps_iso8601: false + ``` +- **Queries the tool cannot convert:** This is rare. When it happens, the tool stops and reports the query it could not parse. The Dashboard shows the error and its line in the validation panel, and the CLI prints it. Convert that bucket definition by hand using [Parameter Syntax Changes](#parameter-syntax-changes), or ask on [Discord](https://discord.gg/powersync). + +## Adopt Sync Streams Features + +After the deploy, the generated streams behave like your bucket definitions did. You can then make the following changes one stream at a time. Changes that keep `auto_subscribe: true` need no client changes. Changes that remove it or change the parameter type need an app update, because clients only receive that data once they subscribe. + +| Change | Client changes | +|--------|----------------| +| [Split the merged stream](#split-the-merged-stream) into named streams | None | +| [Replace parameter CTEs with subqueries](#replace-parameter-ctes-with-subqueries) or JOINs | None | +| [Sync data on demand](#sync-data-on-demand) instead of on connect | Subscribe to the stream from the app | +| [Convert connection parameters to subscription parameters](#convert-connection-parameters-to-subscription-parameters) | Replace connect-time `params` with subscriptions | + +When old and new app versions coexist, keep the old stream and add the changed one under a new name. Newer app versions [opt out of auto-subscribed streams](/sync/streams/client-usage#opting-out-of-auto-subscribed-streams) and subscribe explicitly. Remove the old stream when the older app versions are retired. + +### Split the Merged Stream + +The tool merges your bucket definitions into one stream. Splitting them into named streams makes each stream's purpose visible and lets you change each one independently later. Global data, which syncs the same rows to every user, and user-scoped data both keep `auto_subscribe: true`. Set [`priority`](/sync/advanced/prioritized-sync) per stream where needed: + +```yaml streams: + categories: + auto_subscribe: true + query: SELECT * FROM categories user_lists: auto_subscribe: true priority: 1 query: SELECT * FROM lists WHERE owner_id = auth.user_id() ``` -### Data with Subqueries (Replaces Parameter Queries) +Both streams still sync on connect, so no client changes are needed. + +### Replace Parameter CTEs With Subqueries + +A parameter query that read from a table becomes a CTE that the data query joins. A subquery expresses the same filter in one statement. The generated `list_todos` queries above become: -**Sync Rules:** ```yaml -bucket_definitions: - owned_lists: - parameters: | - SELECT id as list_id FROM lists WHERE owner_id = request.user_id() - data: - - SELECT * FROM lists WHERE lists.id = bucket.list_id - - SELECT * FROM todos WHERE todos.list_id = bucket.list_id +streams: + list_todos: + auto_subscribe: true + query: SELECT * FROM todos WHERE list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) ``` -**Sync Streams:** -```yaml -config: - edition: 3 +The same rows sync, so no client changes are needed. See [Writing Queries](/sync/streams/queries) for JOINs, nested subqueries, and multiple queries per stream. + +### Sync Data On Demand + +A stream without `auto_subscribe: true` syncs only while the app is subscribed to it. Use this for data that a user needs on one screen, such as the todos of the list they opened. Add a [subscription parameter](/sync/streams/parameters#subscription-parameters) for the value the screen provides, and keep an `auth.*` filter so that clients can only subscribe to data they may access: +```yaml streams: - owned_lists: - auto_subscribe: true - query: SELECT * FROM lists WHERE owner_id = auth.user_id() list_todos: query: | - SELECT * FROM todos - WHERE list_id = subscription.parameter('list_id') + SELECT * FROM todos + WHERE list_id = subscription.parameter('list_id') AND list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) ``` -### Client Parameters → Subscription Parameters +The app subscribes when the screen opens and unsubscribes when it closes. The subscription's [TTL](/sync/streams/client-usage#ttl-time-to-live) keeps the data on the device afterwards, so returning to the screen is instant: -**Sync Rules** used global [Client Parameters](/sync/rules/client-parameters): -```yaml -bucket_definitions: - posts: - parameters: SELECT (request.parameters() ->> 'current_page') as page_number - data: - - SELECT * FROM posts WHERE page_number = bucket.page_number +```js +const sub = await db.syncStream('list_todos', { list_id: listId }).subscribe(); +await sub.waitForFirstSync(); + +// When the screen closes +sub.unsubscribe(); ``` -**Sync Streams** use Subscription Parameters, which are more flexible — you can subscribe multiple times with different values: +See [Client-Side Usage](/sync/streams/client-usage) for each SDK and for [framework integrations](/sync/streams/client-usage#framework-integrations) that manage subscriptions from UI components. + +### Convert Connection Parameters to Subscription Parameters + +The tool converts Client Parameters to connection parameters because they behave the same way: the app passes them in `connect()`, they apply to the whole connection, and the app has to reconnect to change them. This keeps your existing behavior, but it is not the best fit for on-demand syncing. Subscription parameters let the app subscribe to the same stream several times with different values, without reconnecting, and each subscription has its own lifecycle. If you prefer to keep passing values at connect time, keep the connection parameters. They need no client changes. + +Before, the migrated `page_posts` query syncs one page per connection: + ```yaml -config: - edition: 3 +streams: + page_posts: + auto_subscribe: true + query: SELECT * FROM posts WHERE page_number = connection.parameter('page_number') +``` + +```js +await db.connect(connector, { + params: { page_number: 1 } +}); +``` + +After, the app subscribes to the pages it needs: +```yaml streams: - posts: + page_posts: query: SELECT * FROM posts WHERE page_number = subscription.parameter('page_number') ``` ```js +await db.connect(connector); + // Subscribe to multiple pages simultaneously -const page1 = await db.syncStream('posts', { page_number: 1 }).subscribe(); -const page2 = await db.syncStream('posts', { page_number: 2 }).subscribe(); +const page1 = await db.syncStream('page_posts', { page_number: 1 }).subscribe(); +const page2 = await db.syncStream('page_posts', { page_number: 2 }).subscribe(); ``` ## Parameter Syntax Changes @@ -217,24 +276,11 @@ const page2 = await db.syncStream('posts', { page_number: 2 }).subscribe(); |------------|--------------| | `request.user_id()` | `auth.user_id()` | | `request.jwt() ->> 'claim'` | `auth.parameter('claim')` | -| `request.parameters() ->> 'key'` | `subscription.parameter('key')` ([subscription parameter](/sync/streams/parameters#subscription-parameters)) or `connection.parameter('key')` ([connection parameter](/sync/streams/parameters#connection-parameters)) | -| `bucket.param_name` | Use the parameter directly in the query e.g. `subscription.parameter('key')` | +| `request.jwt()` | `auth.parameters()` | +| `request.parameters() ->> 'key'` | `connection.parameter('key')` ([connection parameter](/sync/streams/parameters#connection-parameters)). Use `subscription.parameter('key')` ([subscription parameter](/sync/streams/parameters#subscription-parameters)) when you convert the stream to on-demand syncing. | +| `request.parameters()` | `connection.parameters()` | +| `bucket.param_name` | Use the parameter directly in the query, for example `auth.user_id()`, or a subquery. See [Using Subqueries](/sync/streams/queries#using-subqueries). | -## Client-Side Changes - -Streams generated by the migration tool with `auto_subscribe: true` need no client changes. When you convert a stream to on-demand syncing, replace connect-time parameters with a subscription: - -```js -// Before (Sync Rules with Client Parameters) -await db.connect(connector, { - params: { current_project: projectId } -}); - -// After (Sync Streams with Subscriptions) -await db.connect(connector); -const sub = await db.syncStream('project_data', { project_id: projectId }).subscribe(); -``` - -If you want to keep passing values at connect time instead, use [connection parameters](/sync/streams/parameters#connection-parameters). +## Stream Definition Reference -See [Client-Side Usage](/sync/streams/client-usage) for detailed examples. + From 77f54068ed08e4b296fd7ddf4e46836bd61b6188 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Wed, 16 Sep 2026 13:15:48 +0200 Subject: [PATCH 04/20] Add a usage sidebar section for Sync Streams --- docs.json | 25 +++++++++++++++---------- sync/advanced/overview.mdx | 6 ++++-- sync/supported-sql.mdx | 1 - 3 files changed, 19 insertions(+), 13 deletions(-) diff --git a/docs.json b/docs.json index 2d20dbcd9..21866842c 100644 --- a/docs.json +++ b/docs.json @@ -188,18 +188,24 @@ "pages": [ "sync/streams/overview", "sync/streams/quickstart", - "sync/streams/parameters", - "sync/streams/queries", - "sync/streams/ctes", - "sync/streams/bucket-count", - "sync/streams/client-usage", - "sync/types", - "sync/streams/examples", { - "group": "Supported SQL", + "group": "Usage", + "pages": [ + "sync/streams/queries", + "sync/streams/parameters", + "sync/streams/ctes", + "sync/streams/bucket-count", + "sync/streams/client-usage", + "sync/advanced/prioritized-sync", + "sync/streams/examples" + ] + }, + { + "group": "Reference", "pages": [ "sync/supported-sql", - "sync/grammar/sync-streams/index" + "sync/grammar/sync-streams/index", + "sync/types" ] }, { @@ -207,7 +213,6 @@ "pages": [ "sync/advanced/overview", "sync/advanced/reducing-bucket-count", - "sync/advanced/prioritized-sync", "sync/advanced/client-id", "sync/advanced/case-sensitivity", "sync/advanced/compatibility", diff --git a/sync/advanced/overview.mdx b/sync/advanced/overview.mdx index 74a5d0707..070471c18 100644 --- a/sync/advanced/overview.mdx +++ b/sync/advanced/overview.mdx @@ -1,15 +1,17 @@ --- title: "Advanced Topics" -description: "Advanced Sync Streams topics." +description: "Sync Streams topics you only need when a specific condition applies to your data, source database, or deployment." sidebarTitle: Overview --- +These pages cover situations that do not apply to every project. Use them when a specific condition applies to your data, source database, or deployment. For pages that every project needs, see [Writing Queries](/sync/streams/queries) and the other Usage pages. + - + diff --git a/sync/supported-sql.mdx b/sync/supported-sql.mdx index a99508ebe..9287c428c 100644 --- a/sync/supported-sql.mdx +++ b/sync/supported-sql.mdx @@ -1,7 +1,6 @@ --- title: "Supported SQL" description: "Reference for SQL syntax, operators, built-in functions, and type casting supported in Sync Streams/Sync Rules queries." -sidebarTitle: "Guide" --- This guide explains the SQL supported in [Sync Streams](/sync/streams/overview) and [Sync Rules (legacy)](/sync/rules/overview): what you can write, with examples and restrictions. From 66218b2ff50f10a060dd00088848493d4d93abd6 Mon Sep 17 00:00:00 2001 From: benitav Date: Wed, 16 Sep 2026 13:29:27 +0200 Subject: [PATCH 05/20] Update snippets/binary-type.mdx Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> --- snippets/binary-type.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/snippets/binary-type.mdx b/snippets/binary-type.mdx index 121b61735..530cededc 100644 --- a/snippets/binary-type.mdx +++ b/snippets/binary-type.mdx @@ -1,3 +1,3 @@ - Binary data can be accessed in Sync Streams, but cannot be used as [parameters](/sync/streams/parameters). To sync binary columns/fields to clients, those columns need to be converted to hex or base64 representation using the relevant [functions](/sync/supported-sql#functions). + Binary data can be accessed in Sync Streams (or legacy Sync Rules), but cannot be used as a parameter (see [Sync Streams parameters](/sync/streams/parameters) or [Sync Rules client parameters](/sync/rules/client-parameters)). To sync binary columns/fields to clients, those columns need to be converted to hex or base64 representation using the relevant [functions](/sync/supported-sql#functions). \ No newline at end of file From b89a5cd1aff97ffbe31134e1623a62d63113bc70 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Wed, 16 Sep 2026 13:34:02 +0200 Subject: [PATCH 06/20] Fix broken anchor link --- sync/rules/migrate-to-sync-streams.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sync/rules/migrate-to-sync-streams.mdx b/sync/rules/migrate-to-sync-streams.mdx index ba32ac46d..0ad3a05a8 100644 --- a/sync/rules/migrate-to-sync-streams.mdx +++ b/sync/rules/migrate-to-sync-streams.mdx @@ -12,7 +12,7 @@ Sync Streams support everything Sync Rules do (and more), and migrating does not - [Client Parameters](/sync/rules/client-parameters) become [connection parameters](/sync/streams/parameters#connection-parameters). Your app passes them the same way when it connects. - Once your SDKs meet the [minimum versions](#requirements), no client-side code changes are needed. -In most cases you migrate to stay compatible first, then [adopt Sync Streams features](#adopt-sync-streams-features-over-time) such as on-demand syncing one stream at a time. +In most cases you migrate to stay compatible first, then [adopt Sync Streams features](#adopt-sync-streams-features) such as on-demand syncing one stream at a time. If your Sync Config has a `bucket_definitions:` section, you use Sync Rules and this guide applies to you. If it only has `streams:`, you already use Sync Streams and no action is needed. From b7429c1542a36ac8f01795e340b56bee6909b68f Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Wed, 16 Sep 2026 13:41:37 +0200 Subject: [PATCH 07/20] Better link for input-file and output-file --- sync/rules/migrate-to-sync-streams.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sync/rules/migrate-to-sync-streams.mdx b/sync/rules/migrate-to-sync-streams.mdx index 0ad3a05a8..92ba9892a 100644 --- a/sync/rules/migrate-to-sync-streams.mdx +++ b/sync/rules/migrate-to-sync-streams.mdx @@ -93,7 +93,7 @@ try await db.connect(connector: connector, options: ConnectOptions( Use one of the following: - **PowerSync Dashboard:** Click **Migrate to Sync Streams**. The Dashboard converts the instance's deployed Sync Rules and opens the result as a draft for you to review. - - **CLI:** Run `powersync migrate sync-rules`. By default the command reads `sync-config.yaml` in your linked project directory and overwrites it with the result. Use `--input-file` and `--output-file` to read from and write to other paths. See the [CLI reference](/tools/cli). + - **CLI:** Run `powersync migrate sync-rules`. By default the command reads `sync-config.yaml` in your `powersync` config directory and overwrites it with the result. Use `--input-file` and `--output-file` to read from and write to other paths. See the [command reference](https://github.com/powersync-ja/powersync-cli/blob/main/cli/README.md#powersync-migrate-sync-rules) for all flags. Compare the draft with your Sync Rules. See [What the Tool Generates](#what-the-tool-generates) for how the output maps to your bucket definitions, and [What to Check Before You Deploy](#what-to-check-before-you-deploy) for the items that need your attention. From 2831a32a3f75ae976ea374d9e7ba6e91fa672802 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Fri, 18 Sep 2026 16:57:53 +0200 Subject: [PATCH 08/20] Reorganize the Prioritized Sync page --- sync/advanced/prioritized-sync.mdx | 262 ----------------------------- 1 file changed, 262 deletions(-) delete mode 100644 sync/advanced/prioritized-sync.mdx diff --git a/sync/advanced/prioritized-sync.mdx b/sync/advanced/prioritized-sync.mdx deleted file mode 100644 index bc5a1340e..000000000 --- a/sync/advanced/prioritized-sync.mdx +++ /dev/null @@ -1,262 +0,0 @@ ---- -title: "Prioritized Sync" -description: "Prioritize which tables sync first so users can start working immediately while remaining data continues loading in the background." ---- - -## Overview - -PowerSync supports defining sync priorities, which allows you to control the sync order for different data. This is particularly useful when certain data should be available sooner than others. - -In Sync Streams, priorities are assigned to streams and PowerSync manages the underlying buckets internally. (In legacy Sync Rules, priorities were assigned to buckets explicitly.) - - -**Availability** - -This feature was introduced in version **1.7.1** of the PowerSync Service, and in the following SDK versions: -- [Flutter v1.12.0](/client-sdks/reference/flutter) -- [React Native v1.18.1](/client-sdks/reference/react-native-and-expo) -- [JavaScript Web v1.14.2](/client-sdks/reference/javascript-web) -- [Kotlin v1.0.0-BETA26](/client-sdks/reference/kotlin) -- [Swift v1.0.0-Beta.8](/client-sdks/reference/swift) -- [.NET v0.0.6-alpha.1](/client-sdks/reference/dotnet) - - - -## Why Use Sync Priorities? - -PowerSync's standard sync protocol ensures that: -- The local data view is only updated when a fully consistent checkpoint is available. -- All pending local changes must be uploaded, acknowledged, and synced back before new data is applied. - -While this guarantees consistency, it can lead to delays, especially for large datasets or continuous client-side updates. Sync priorities provide a way to speed up syncing of high-priority data while still maintaining overall integrity. - -## How It Works - -Each bucket is assigned a priority value between 0 and 3, where: - -- 0 is the highest priority and has special behavior (detailed below). -- 3 is the default and lowest priority. -- Lower numbers indicate higher priority. - -Higher-priority data syncs first, and lower-priority data syncs later. If you only use a single priority, there is no difference between priorities 1-3. The difference only comes in when you use multiple different priorities. - - - -In Sync Streams, you assign priorities directly to streams. PowerSync manages buckets internally, so you don't need to think about bucket structure. Each stream with a given priority will have its data synced at that priority level. - -```yaml -streams: - lists: - auto_subscribe: true - query: SELECT * FROM lists WHERE owner_id = auth.user_id() - priority: 1 # Syncs first - - todos: - auto_subscribe: true - query: SELECT * FROM todos WHERE list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) - priority: 2 # Syncs after lists -``` - -Clients can also override the priority when subscribing: - -```js -// Override the stream's default priority for this subscription -const sub = await db.syncStream('todos', { list_id: 'abc' }).subscribe({ priority: 1 }); -``` - -When different components subscribe to the same stream with the same parameters but different priorities, PowerSync uses the highest priority for syncing. That higher priority is kept until the subscription ends (or its TTL expires). Subscriptions with different parameters are independent and do not conflict. - - -In Sync Rules, you assign priorities to bucket definitions. The priority determines when data in that bucket syncs relative to other buckets. - -```yaml -bucket_definitions: - user_lists: - priority: 1 # Syncs first - parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() - data: - - SELECT * FROM lists WHERE id = bucket.list_id - - user_todos: - priority: 2 # Syncs after lists - parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() - data: - - SELECT * FROM todos WHERE list_id = bucket.list_id -``` - - - -## Syntax and Configuration - - - -In Sync Streams, set the `priority` option on the stream definition: - -```yaml -streams: - high_priority_data: - auto_subscribe: true - query: SELECT * FROM important_table WHERE user_id = auth.user_id() - priority: 1 - - low_priority_data: - auto_subscribe: true - query: SELECT * FROM background_table WHERE user_id = auth.user_id() - priority: 2 -``` - - -In Sync Rules, priorities can be defined using the `priority` YAML key on bucket definitions, or with the `_priority` attribute inside parameter queries: - -```yaml -bucket_definitions: - # Using the `priority` YAML key - user_data: - priority: 1 - parameters: SELECT request.user_id() AS id WHERE ... - data: - # ... - - # Using the `_priority` attribute (useful for multiple parameter queries with different priorities) - project_data: - parameters: SELECT id AS project_id, 2 AS _priority FROM projects WHERE ... - data: - # ... -``` - - - - -Priorities must be static and cannot depend on row values within a parameter query. - - -## Example: Syncing Lists Before Todos - -Consider a scenario where you want to display lists immediately while loading todos in the background. This approach allows users to view and interact with lists right away without waiting for todos to sync. - - - -```yaml -config: - edition: 3 - -streams: - lists: - auto_subscribe: true - query: SELECT * FROM lists WHERE owner_id = auth.user_id() - priority: 1 # Syncs first - - todos: - auto_subscribe: true - query: | - SELECT * FROM todos - WHERE list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) - priority: 2 # Syncs after lists -``` - -The `lists` stream syncs first (priority 1), allowing users to see and interact with their lists immediately. The `todos` stream syncs afterward (priority 2), loading in the background. - - -```yaml -bucket_definitions: - user_lists: - priority: 1 # Syncs first - parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() - data: - - SELECT * FROM lists WHERE id = bucket.list_id - - user_todos: - priority: 2 # Syncs after lists - parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() - data: - - SELECT * FROM todos WHERE list_id = bucket.list_id -``` - -The `user_lists` bucket syncs first (priority 1), allowing users to see and interact with their lists immediately. The `user_todos` bucket syncs afterward (priority 2), loading in the background. - - - - -## Behavioral Considerations - -- **Interruption for Higher Priority Data**: Syncing lower-priority data _may_ be interrupted if new data for higher-priority streams/buckets arrives. -- **Local Changes & Consistency**: If local writes fail due to validation or permission issues, they are only reverted after _all_ data has synced. -- **Deleted Data**: Deleted data may only be removed after _all_ priorities have completed syncing. Future updates may improve this behavior. -- **Data Ordering**: Lower-priority data will never appear before higher-priority data. - -## Special Case: Priority 0 - -Priority 0 buckets sync regardless of pending uploads. - -For example, in a collaborative document editing app (e.g., using Yjs), each change is stored as a separate row. Since out-of-order updates don’t affect document integrity, Priority 0 can ensure immediate availability of updates. - -Caution: If misused, Priority 0 may cause flickering or inconsistencies, as updates could arrive out of order. - -## Consistency Considerations - -PowerSync's full consistency guarantees only apply once all priorities have completed syncing. - -When higher-priority data is synced, all inserts and updates at that priority level will be consistent. However, deletes are only applied when the full sync completes, so you may still have some stale data at those priority levels. - -Consider the following example: - -Imagine a task management app where users create lists and todos. Some users have millions of todos. To improve first-load speed: - -- Lists are assigned Priority 1, syncing first to allow UI rendering. -- Todos are assigned Priority 2, loading in the background. - -Now, if another user adds new todos, it’s possible for the list count (synced at Priority 1) to temporarily not match the actual todos (synced at Priority 2). If real-time accuracy is required, both lists and todos should use the same priority. - -## Client-Side Considerations - -PowerSync's client SDKs provide APIs to allow applications to track sync status at different priority levels. Developers can leverage these to ensure critical data is available before proceeding with UI updates or background processing. This includes: - -1. `waitForFirstSync(priority: int)`. When passing the optional `priority` parameter to this method, it will wait for specific priority level to complete syncing. -2. `SyncStatus.priorityStatusEntries()` A list containing sync information for each priority that was seen by the PowerSync Service. -3. `SyncStatus.statusForPriority(priority: int)` This method takes a fixed priority and returns the sync state for that priority by looking it up in `priorityStatusEntries`. - -## Example -Using the above we can render a lists component only once the user's lists (with priority 1) have completed syncing, else display a message indicating that the sync is still in progress: - -```dart - // Define the priority level for lists - static final _listsPriority = BucketPriority(1); - - @override - Widget build(BuildContext context) { - // Use FutureBuilder to wait for the first sync of the specified priority to complete - return FutureBuilder( - future: db.waitForFirstSync(priority: _listsPriority), - builder: (context, snapshot) { - if (snapshot.connectionState == ConnectionState.done) { - // Use StreamBuilder to render the lists once the sync completes - return StreamBuilder( - stream: TodoList.watchListsWithStats(), - builder: (context, snapshot) { - if (snapshot.data case final todoLists?) { - return ListView( - padding: const EdgeInsets.symmetric(vertical: 8.0), - children: todoLists.map((list) { - return ListItemWidget(list: list); - }).toList(), - ); - } else { - return const CircularProgressIndicator(); - } - }, - ); - } else { - return const Text('Busy with sync...'); - } - }, - ); - } - -``` - -Example implementations of prioritized sync are also available in the following apps: -- Flutter: [Supabase To-Do List](https://github.com/powersync-ja/powersync.dart/tree/main/demos/supabase-todolist) -- Kotlin: - - [Supabase To-Do List (KMP)](https://github.com/powersync-ja/powersync-kotlin/blob/main/demos/supabase-todolist/shared/src/commonMain/kotlin/com/powersync/demos/App.kt#L46) - - [Supabase To-Do List (Android)](https://github.com/powersync-ja/powersync-kotlin/blob/main/demos/android-supabase-todolist/src/main/java/com/powersync/androidexample/screens/HomeScreen.kt#L69) -- Swift: [Supabase To-Do List](https://github.com/powersync-ja/powersync-swift/tree/main/Demos/PowerSyncExample) \ No newline at end of file From e16fa89d4692c2a193e0505a5f4b0d58dbb86f8f Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Fri, 18 Sep 2026 16:58:10 +0200 Subject: [PATCH 09/20] Update refs and add redirect --- architecture/consistency.mdx | 2 +- client-sdks/advanced/raw-tables.mdx | 2 +- debugging/troubleshooting.mdx | 2 +- docs.json | 8 +- snippets/stream-definition-reference.mdx | 2 +- sync/rules/migrate-to-sync-streams.mdx | 4 +- sync/streams/client-usage.mdx | 4 +- sync/streams/prioritized-sync.mdx | 249 +++++++++++++++++++++++ 8 files changed, 263 insertions(+), 10 deletions(-) create mode 100644 sync/streams/prioritized-sync.mdx diff --git a/architecture/consistency.mdx b/architecture/consistency.mdx index f717cdad3..756d09a72 100644 --- a/architecture/consistency.mdx +++ b/architecture/consistency.mdx @@ -24,7 +24,7 @@ While mutations are present in the upload queue, the client does not advance to Only once all the client-side mutations have been acknowledged by the server, and the data for that new checkpoint is downloaded by the client, does the client advance to the next checkpoint. This ensures that the operations are always ordered correctly on the client. -There is one nuanced case here, which is buckets with [Priority 0](/sync/advanced/prioritized-sync#special-case-priority-0) if you are using [Prioritized Syncing](/sync/advanced/prioritized-sync). +There is one nuanced case here, which is buckets with [Priority 0](/sync/streams/prioritized-sync#special-case-priority-0) if you are using [Prioritized Syncing](/sync/streams/prioritized-sync). ## Types of Client-Side Mutations/Operations diff --git a/client-sdks/advanced/raw-tables.mdx b/client-sdks/advanced/raw-tables.mdx index de17713bd..ebe568c96 100644 --- a/client-sdks/advanced/raw-tables.mdx +++ b/client-sdks/advanced/raw-tables.mdx @@ -507,7 +507,7 @@ Raw tables support advanced table constraints including foreign keys. When enabl 1. While PowerSync will always apply synced data in a transaction, there is no way to control the order in which rows get applied. For this reason, foreign keys need to be configured with `DEFERRABLE INITIALLY DEFERRED`. -2. When using [stream priorities](/sync/advanced/prioritized-sync), you need to ensure you don't have foreign keys from high-priority +2. When using [stream priorities](/sync/streams/prioritized-sync), you need to ensure you don't have foreign keys from high-priority rows to lower-priority data. PowerSync applies data in one transaction per priority, so these foreign keys would not work. 3. As usual when using foreign keys, note that they need to be explicitly enabled with `pragma foreign_keys = on`. diff --git a/debugging/troubleshooting.mdx b/debugging/troubleshooting.mdx index 89605bdb7..7967620d1 100644 --- a/debugging/troubleshooting.mdx +++ b/debugging/troubleshooting.mdx @@ -131,6 +131,6 @@ Sync & API logs in the [PowerSync Dashboard](https://dashboard.powersync.com/) r #### Common Causes of Latency * **Large initial sync**: if your Sync Streams/Rules result in a large dataset, the first sync after connecting will be slow. Inspect bucket sizes and sync state with the [Sync Diagnostics Client](/tools/diagnostics-client). -* **Upload queue blocking downloads**: by default, uploads are processed before downloads, so a backlogged upload queue delays receiving new data. Buckets and streams at [priority 0](/sync/advanced/prioritized-sync) are not blocked by uploads, but come with the trade-off of potential sync inconsistencies. +* **Upload queue blocking downloads**: by default, uploads are processed before downloads, so a backlogged upload queue delays receiving new data. Buckets and streams at [priority 0](/sync/streams/prioritized-sync) are not blocked by uploads, but come with the trade-off of potential sync inconsistencies. * **Replication lag on the source database**: high write volume, long-running transactions, bulk updates, or backfills can cause replication to fall behind faster than the service can drain it. See [Replication Lag](/maintenance-ops/replication-lag) for source-specific causes and fixes. * **Too many buckets per user**: incremental sync overhead scales roughly linearly with the number of buckets per user. See [Too Many Buckets](#psync_s2305-too-many-buckets-/-parameter-query-results) above. diff --git a/docs.json b/docs.json index 21866842c..50a4eec91 100644 --- a/docs.json +++ b/docs.json @@ -196,7 +196,7 @@ "sync/streams/ctes", "sync/streams/bucket-count", "sync/streams/client-usage", - "sync/advanced/prioritized-sync", + "sync/streams/prioritized-sync", "sync/streams/examples" ] }, @@ -810,7 +810,11 @@ }, { "source": "/usage/use-case-examples/prioritized-sync", - "destination": "/sync/advanced/prioritized-sync" + "destination": "/sync/streams/prioritized-sync" + }, + { + "source": "/sync/advanced/prioritized-sync", + "destination": "/sync/streams/prioritized-sync" }, { "source": "/usage/sync-rules/client-id", diff --git a/snippets/stream-definition-reference.mdx b/snippets/stream-definition-reference.mdx index 8740faf00..af5e93310 100644 --- a/snippets/stream-definition-reference.mdx +++ b/snippets/stream-definition-reference.mdx @@ -28,5 +28,5 @@ streams: | `queries` | — | Array of queries defining which data to sync. More efficient than defining separate streams: the client manages one subscription and PowerSync merges the data from all queries (see [Multiple Queries per Stream](/sync/streams/queries#multiple-queries-per-stream)). | | `with` | — | [CTEs](/sync/streams/ctes) available to this stream's queries. Define the `with` block inside each stream. | | `auto_subscribe` | `false` | When `true`, clients automatically subscribe on connect. | -| `priority` | — | Sync priority (lower value = higher priority). See [Prioritized Sync](/sync/advanced/prioritized-sync). | +| `priority` | — | Sync priority (lower value = higher priority). See [Prioritized Sync](/sync/streams/prioritized-sync). | | `accept_potentially_dangerous_queries` | `false` | Silences security warnings when queries use client-controlled parameters (i.e. _connection parameters_ and _subscription parameters_), as opposed to _authentication parameters_ that are signed as part of the JWT. Set to `true` only if you've verified the query is safe. See [Using Parameters](/sync/streams/parameters). | diff --git a/sync/rules/migrate-to-sync-streams.mdx b/sync/rules/migrate-to-sync-streams.mdx index 92ba9892a..69a2c4f11 100644 --- a/sync/rules/migrate-to-sync-streams.mdx +++ b/sync/rules/migrate-to-sync-streams.mdx @@ -150,7 +150,7 @@ streams: The tool applies these rules: - **Compatibility edition:** It sets `config: edition: 3`, which Sync Streams require, and keeps any other options in your `config` block. -- **One stream per priority:** Bucket definitions with the same [priority](/sync/advanced/prioritized-sync) are merged into one stream named `migrated_to_streams`. Comments mark which bucket definition each group of queries came from. If your bucket definitions use different priorities, the tool creates one stream per priority, named `migrated_to_streams_prio_`. +- **One stream per priority:** Bucket definitions with the same [priority](/sync/streams/prioritized-sync) are merged into one stream named `migrated_to_streams`. Comments mark which bucket definition each group of queries came from. If your bucket definitions use different priorities, the tool creates one stream per priority, named `migrated_to_streams_prio_`. - **Same sync behavior:** Every stream has `auto_subscribe: true`. Queries are always written as a `queries:` list so that you can add more. - **Parameters:** `request.*` functions become `auth.*` and `connection.*` functions. See [Parameter Syntax Changes](#parameter-syntax-changes) for the full mapping. Parameter queries that only select request values, such as `SELECT request.user_id() as user_id`, are replaced by those values in the data queries: `bucket.user_id` becomes `auth.user_id()`. Parameter queries that read from a table become CTEs in a `with:` block, named `_param`, and the data queries join them under the alias `bucket`. - **Cleanup:** The `bucket_definitions:` section is removed. @@ -182,7 +182,7 @@ When old and new app versions coexist, keep the old stream and add the changed o ### Split the Merged Stream -The tool merges your bucket definitions into one stream. Splitting them into named streams makes each stream's purpose visible and lets you change each one independently later. Global data, which syncs the same rows to every user, and user-scoped data both keep `auto_subscribe: true`. Set [`priority`](/sync/advanced/prioritized-sync) per stream where needed: +The tool merges your bucket definitions into one stream. Splitting them into named streams makes each stream's purpose visible and lets you change each one independently later. Global data, which syncs the same rows to every user, and user-scoped data both keep `auto_subscribe: true`. Set [`priority`](/sync/streams/prioritized-sync) per stream where needed: ```yaml streams: diff --git a/sync/streams/client-usage.mdx b/sync/streams/client-usage.mdx index 1c3ab8fd4..0765ba9c7 100644 --- a/sync/streams/client-usage.mdx +++ b/sync/streams/client-usage.mdx @@ -244,7 +244,7 @@ const { data: todos } = useQuery( } ``` - You can pass `ttl` and `priority` for cache duration and [sync priority](/sync/advanced/prioritized-sync): + You can pass `ttl` and `priority` for cache duration and [sync priority](/sync/streams/prioritized-sync): ```kotlin database.composeSyncStream( @@ -522,7 +522,7 @@ const subB = await db.syncStream('todos', { list_id: 'B' }).subscribe({ ttl: 864 ## Priority Override -Streams can have a default priority set in the YAML sync configuration (see [Prioritized Sync](/sync/advanced/prioritized-sync)). When subscribing, you can override this priority for a specific subscription: +Streams can have a default priority set in the YAML sync configuration (see [Prioritized Sync](/sync/streams/prioritized-sync)). When subscribing, you can override this priority for a specific subscription: ```js // Override the stream's default priority const sub = await db.syncStream('todos', { list_id: 'abc' }).subscribe({ priority: 1 }); diff --git a/sync/streams/prioritized-sync.mdx b/sync/streams/prioritized-sync.mdx new file mode 100644 index 000000000..064c1dd56 --- /dev/null +++ b/sync/streams/prioritized-sync.mdx @@ -0,0 +1,249 @@ +--- +title: "Prioritized Sync" +description: "Prioritize which tables sync first so users can start working immediately while remaining data continues loading in the background." +--- + +## Overview + +PowerSync supports defining sync priorities, which allows you to control the sync order for different data. This is particularly useful when certain data should be available sooner than others. + +In Sync Streams, priorities are assigned to streams and PowerSync manages the underlying buckets internally. (In legacy Sync Rules, priorities were assigned to buckets explicitly.) + +## Why Use Sync Priorities? + +PowerSync's standard sync protocol ensures that: +- The local data view is only updated when a fully consistent checkpoint is available. +- All pending local changes must be uploaded, acknowledged, and synced back before new data is applied. + +While this guarantees consistency, it can lead to delays, especially for large datasets or continuous client-side updates. Sync priorities provide a way to speed up syncing of high-priority data while still maintaining overall integrity. + +## How It Works + +Each bucket is assigned a priority value between 0 and 3, where: + +- 0 is the highest priority and has special behavior (detailed below). +- 3 is the default and lowest priority. +- Lower numbers indicate higher priority. + +Higher-priority data syncs first, and lower-priority data syncs later. If you only use a single priority, there is no difference between priorities 1-3. The difference only comes in when you use multiple different priorities. + + + +In Sync Streams, you assign priorities directly to streams. PowerSync manages buckets internally, so you don't need to think about bucket structure. Each stream with a given priority will have its data synced at that priority level. + +```yaml +streams: + lists: + auto_subscribe: true + query: SELECT * FROM lists WHERE owner_id = auth.user_id() + priority: 1 # Syncs first + + todos: + auto_subscribe: true + query: SELECT * FROM todos WHERE list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) + priority: 2 # Syncs after lists +``` + +Clients can also override the priority when subscribing: + +```js +// Override the stream's default priority for this subscription +const sub = await db.syncStream('todos', { list_id: 'abc' }).subscribe({ priority: 1 }); +``` + +When different components subscribe to the same stream with the same parameters but different priorities, PowerSync uses the highest priority for syncing. That higher priority is kept until the subscription ends (or its TTL expires). Subscriptions with different parameters are independent and do not conflict. + + +In Sync Rules, you assign priorities to bucket definitions. The priority determines when data in that bucket syncs relative to other buckets. + +```yaml +bucket_definitions: + user_lists: + priority: 1 # Syncs first + parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() + data: + - SELECT * FROM lists WHERE id = bucket.list_id + + user_todos: + priority: 2 # Syncs after lists + parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() + data: + - SELECT * FROM todos WHERE list_id = bucket.list_id +``` + + + +## Syntax and Configuration + + + +In Sync Streams, set the `priority` option on the stream definition: + +```yaml +streams: + high_priority_data: + auto_subscribe: true + query: SELECT * FROM important_table WHERE user_id = auth.user_id() + priority: 1 + + low_priority_data: + auto_subscribe: true + query: SELECT * FROM background_table WHERE user_id = auth.user_id() + priority: 2 +``` + + +In Sync Rules, priorities can be defined using the `priority` YAML key on bucket definitions, or with the `_priority` attribute inside parameter queries: + +```yaml +bucket_definitions: + # Using the `priority` YAML key + user_data: + priority: 1 + parameters: SELECT request.user_id() AS id WHERE ... + data: + # ... + + # Using the `_priority` attribute (useful for multiple parameter queries with different priorities) + project_data: + parameters: SELECT id AS project_id, 2 AS _priority FROM projects WHERE ... + data: + # ... +``` + + + + +Priorities must be static and cannot depend on row values within a parameter query. + + +## Example: Syncing Lists Before Todos + +Consider a scenario where you want to display lists immediately while loading todos in the background. This approach allows users to view and interact with lists right away without waiting for todos to sync. + + + +```yaml +config: + edition: 3 + +streams: + lists: + auto_subscribe: true + query: SELECT * FROM lists WHERE owner_id = auth.user_id() + priority: 1 # Syncs first + + todos: + auto_subscribe: true + query: | + SELECT * FROM todos + WHERE list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) + priority: 2 # Syncs after lists +``` + +The `lists` stream syncs first (priority 1), allowing users to see and interact with their lists immediately. The `todos` stream syncs afterward (priority 2), loading in the background. + + +```yaml +bucket_definitions: + user_lists: + priority: 1 # Syncs first + parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() + data: + - SELECT * FROM lists WHERE id = bucket.list_id + + user_todos: + priority: 2 # Syncs after lists + parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() + data: + - SELECT * FROM todos WHERE list_id = bucket.list_id +``` + +The `user_lists` bucket syncs first (priority 1), allowing users to see and interact with their lists immediately. The `user_todos` bucket syncs afterward (priority 2), loading in the background. + + + + +## Behavioral Considerations + +- **Interruption for Higher Priority Data**: Syncing lower-priority data _may_ be interrupted if new data for higher-priority streams/buckets arrives. +- **Local Changes & Consistency**: If local writes fail due to validation or permission issues, they are only reverted after _all_ data has synced. +- **Deleted Data**: Deleted data may only be removed after _all_ priorities have completed syncing. Future updates may improve this behavior. +- **Data Ordering**: Lower-priority data will never appear before higher-priority data. + +## Special Case: Priority 0 + +Priority 0 buckets sync regardless of pending uploads. + +For example, in a collaborative document editing app (e.g., using Yjs), each change is stored as a separate row. Since out-of-order updates don’t affect document integrity, Priority 0 can ensure immediate availability of updates. + +Caution: If misused, Priority 0 may cause flickering or inconsistencies, as updates could arrive out of order. + +## Consistency Considerations + +PowerSync's full consistency guarantees only apply once all priorities have completed syncing. + +When higher-priority data is synced, all inserts and updates at that priority level will be consistent. However, deletes are only applied when the full sync completes, so you may still have some stale data at those priority levels. + +Consider the following example: + +Imagine a task management app where users create lists and todos. Some users have millions of todos. To improve first-load speed: + +- Lists are assigned Priority 1, syncing first to allow UI rendering. +- Todos are assigned Priority 2, loading in the background. + +Now, if another user adds new todos, it’s possible for the list count (synced at Priority 1) to temporarily not match the actual todos (synced at Priority 2). If real-time accuracy is required, both lists and todos should use the same priority. + +## Client-Side Considerations + +PowerSync's client SDKs provide APIs to allow applications to track sync status at different priority levels. Developers can leverage these to ensure critical data is available before proceeding with UI updates or background processing. This includes: + +1. `waitForFirstSync(priority: int)`. When passing the optional `priority` parameter to this method, it will wait for specific priority level to complete syncing. +2. `SyncStatus.priorityStatusEntries()` A list containing sync information for each priority that was seen by the PowerSync Service. +3. `SyncStatus.statusForPriority(priority: int)` This method takes a fixed priority and returns the sync state for that priority by looking it up in `priorityStatusEntries`. + +## Example +Using the above we can render a lists component only once the user's lists (with priority 1) have completed syncing, else display a message indicating that the sync is still in progress: + +```dart + // Define the priority level for lists + static final _listsPriority = BucketPriority(1); + + @override + Widget build(BuildContext context) { + // Use FutureBuilder to wait for the first sync of the specified priority to complete + return FutureBuilder( + future: db.waitForFirstSync(priority: _listsPriority), + builder: (context, snapshot) { + if (snapshot.connectionState == ConnectionState.done) { + // Use StreamBuilder to render the lists once the sync completes + return StreamBuilder( + stream: TodoList.watchListsWithStats(), + builder: (context, snapshot) { + if (snapshot.data case final todoLists?) { + return ListView( + padding: const EdgeInsets.symmetric(vertical: 8.0), + children: todoLists.map((list) { + return ListItemWidget(list: list); + }).toList(), + ); + } else { + return const CircularProgressIndicator(); + } + }, + ); + } else { + return const Text('Busy with sync...'); + } + }, + ); + } + +``` + +Example implementations of prioritized sync are also available in the following apps: +- Flutter: [Supabase To-Do List](https://github.com/powersync-ja/powersync.dart/tree/main/demos/supabase-todolist) +- Kotlin: + - [Supabase To-Do List (KMP)](https://github.com/powersync-ja/powersync-kotlin/blob/main/demos/supabase-todolist/shared/src/commonMain/kotlin/com/powersync/demos/App.kt#L46) + - [Supabase To-Do List (Android)](https://github.com/powersync-ja/powersync-kotlin/blob/main/demos/android-supabase-todolist/src/main/java/com/powersync/androidexample/screens/HomeScreen.kt#L69) +- Swift: [Supabase To-Do List](https://github.com/powersync-ja/powersync-swift/tree/main/Demos/PowerSyncExample) \ No newline at end of file From ae3dabd12cd73d4070cff25afedf3615cc2900d4 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Fri, 18 Sep 2026 18:17:10 +0200 Subject: [PATCH 10/20] Self-contained sync rules and sync streams sections --- .claude/CLAUDE.md | 11 +- .claude/agents/document-reviewer.md | 1 + .claude/skills/doc-author/SKILL.md | 2 +- .claude/skills/pr-to-docs/SKILL.md | 2 +- docs.json | 29 +- package.json | 3 +- scripts/check-links.mjs | 177 ++++++++++++ snippets/sync-shared/case-sensitivity.mdx | 40 +++ snippets/sync-shared/client-id.mdx | 65 +++++ snippets/sync-shared/compatibility.mdx | 217 ++++++++++++++ .../sync-shared/reducing-bucket-count.mdx | 240 ++++++++++++++++ .../sync-shared/schemas-and-connections.mdx | 60 ++++ snippets/sync-shared/sharded-databases.mdx | 44 +++ snippets/sync-shared/storage-version-4.mdx | 170 +++++++++++ snippets/sync-shared/types.mdx | 164 +++++++++++ sync/advanced/case-sensitivity.mdx | 39 +-- sync/advanced/client-id.mdx | 64 +---- sync/advanced/compatibility.mdx | 216 +------------- sync/advanced/multiple-client-versions.mdx | 52 ++-- sync/advanced/partitioned-tables.mdx | 65 ++--- sync/advanced/reducing-bucket-count.mdx | 239 +--------------- sync/advanced/schemas-and-connections.mdx | 59 +--- sync/advanced/sharded-databases.mdx | 43 +-- sync/advanced/storage-version-4.mdx | 169 +---------- sync/advanced/sync-data-by-time.mdx | 267 ++++++------------ sync/grammar/sync-rules/index.mdx | 4 +- sync/rules/case-sensitivity.mdx | 15 + sync/rules/client-id.mdx | 15 + sync/rules/compatibility.mdx | 15 + sync/rules/data-queries.mdx | 2 +- sync/rules/migrate-to-sync-streams.mdx | 4 +- sync/rules/multiple-client-versions.mdx | 39 +++ sync/rules/organize-data-into-buckets.mdx | 2 +- sync/rules/overview.mdx | 25 +- sync/rules/parameter-queries.mdx | 6 +- sync/rules/partitioned-tables.mdx | 37 +++ sync/rules/prioritized-sync.mdx | 126 +++++++++ sync/rules/reducing-bucket-count.mdx | 16 ++ sync/rules/schemas-and-connections.mdx | 15 + sync/rules/sharded-databases.mdx | 15 + sync/rules/storage-version-4.mdx | 15 + sync/rules/supported-sql.mdx | 157 ++++++++++ sync/rules/sync-data-by-time.mdx | 174 ++++++++++++ sync/rules/types.mdx | 16 ++ sync/streams/bucket-count.mdx | 4 - sync/streams/client-usage.mdx | 2 +- sync/streams/examples.mdx | 2 +- sync/streams/overview.mdx | 6 - sync/streams/parameters.mdx | 2 +- sync/streams/prioritized-sync.mdx | 111 +------- sync/streams/quickstart.mdx | 1 - sync/supported-sql.mdx | 122 ++------ sync/types.mdx | 163 +---------- 53 files changed, 2074 insertions(+), 1475 deletions(-) create mode 100644 scripts/check-links.mjs create mode 100644 snippets/sync-shared/case-sensitivity.mdx create mode 100644 snippets/sync-shared/client-id.mdx create mode 100644 snippets/sync-shared/compatibility.mdx create mode 100644 snippets/sync-shared/reducing-bucket-count.mdx create mode 100644 snippets/sync-shared/schemas-and-connections.mdx create mode 100644 snippets/sync-shared/sharded-databases.mdx create mode 100644 snippets/sync-shared/storage-version-4.mdx create mode 100644 snippets/sync-shared/types.mdx create mode 100644 sync/rules/case-sensitivity.mdx create mode 100644 sync/rules/client-id.mdx create mode 100644 sync/rules/compatibility.mdx create mode 100644 sync/rules/multiple-client-versions.mdx create mode 100644 sync/rules/partitioned-tables.mdx create mode 100644 sync/rules/prioritized-sync.mdx create mode 100644 sync/rules/reducing-bucket-count.mdx create mode 100644 sync/rules/schemas-and-connections.mdx create mode 100644 sync/rules/sharded-databases.mdx create mode 100644 sync/rules/storage-version-4.mdx create mode 100644 sync/rules/supported-sql.mdx create mode 100644 sync/rules/sync-data-by-time.mdx create mode 100644 sync/rules/types.mdx diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 17b261d2e..ca29c2dab 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -142,18 +142,19 @@ Update `docs.json` when adding, moving, or removing pages. Add redirects for mov ## Sync Streams and Sync Rules -Sync Streams are the default for new documentation. Keep legacy Sync Rules documentation accurate, but do not add new Sync Rules teaching, examples, or proactive references. +Sync Rules are deprecated. New documentation and updates cover Sync Streams only. Many customers still use Sync Rules, so their documentation stays online but frozen. -When existing content shows both in tabs, preserve equivalent results and filters. Do not add new parallel Sync Rules examples. - -For existing prose that mentions both, use "[Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview))" once per page or major section. Later mentions should omit Sync Rules. +- **Sync Rules docs are frozen.** `sync/rules/` and `sync/grammar/sync-rules/` take error fixes only: no new features, examples, or pages. Each page opens with an `` callout that starts "Sync Rules are deprecated." and links to its Sync Streams version. The sidebar group stays "Sync Rules (Legacy)". +- **Keep the engines apart.** Never place Sync Streams and Sync Rules content side by side: no engine tabs, no "(or legacy Sync Rules)" asides, no pointers to a Sync Rules equivalent. Outside `sync/rules/`, mention Sync Rules only to state a behavior difference that affects Sync Rules users, and remove other mentions when you edit a page. +- **Shared pages.** A page that applies to both engines keeps one body in `snippets/sync-shared/.mdx`, imported by a Sync Streams wrapper at the original path and a Sync Rules wrapper at `sync/rules/.mdx` that adds the callout and `noindex: true`. Edit the snippet, not the wrappers, and keep it valid for both engines. +- **Counterpart comments.** Every split twin, wrapper, and shared snippet starts with an MDX comment naming its counterpart. Read it before editing, apply a fix to both sides where content is shared, and keep the comment when restructuring. ## Verification - Verify technical claims and run code examples before publication. Select other checks appropriate to the change. - Run `vale ` for changed MDX pages. Add new technical terms to `.github/vale/config/vocabularies/PowerSync/accept.txt`; do not add ordinary misspellings. - After link or navigation changes, run `npx mintlify broken-links`. Mintlify requires Node 20.17–24; if needed, use `PATH="/opt/homebrew/opt/node@24/bin:$PATH" npx mintlify broken-links`. -- For anchor and snippet checks, use `pnpm check:links`. Validate repository instruction links as file paths, since the site checker does not cover all of them. +- For anchor and snippet checks, use `pnpm check:links`. It wraps the Mintlify anchor check in `scripts/check-links.mjs` so that anchors defined in imported snippets resolve. Validate repository instruction links as file paths, since the site checker does not cover all of them. - Use [the lint command](commands/lint-docs.md) for the check workflow and [the reviewer](agents/document-reviewer.md) for editorial review. Passing linters does not establish technical accuracy or style compliance. ## Git Workflow diff --git a/.claude/agents/document-reviewer.md b/.claude/agents/document-reviewer.md index 06f904eae..05794244e 100644 --- a/.claude/agents/document-reviewer.md +++ b/.claude/agents/document-reviewer.md @@ -17,6 +17,7 @@ Apply the canonical standards in three passes and report findings from each: 1. **Accuracy:** claims, platform scope, versions, and consistency with the surrounding page. Before reporting a claim as unverified, check the sources the PR or the user cites, such as the source PR, divergence issue, release notes, or code at the merged commit, and follow the links inside them. If nothing is cited, look up the release the text names. Report a claim as unverified only when no source covers it or a source contradicts it. Give evidence, do not invent problems, and do not approve unverified claims as correct. 2. **Necessity:** list every sentence that describes what the product prints, displays, logs, or says in an error. Treat each one as a finding to remove unless it passes the restating rule in [Content Strategy](../CLAUDE.md#content-strategy), and report it even when the sentence is accurate. Describing visible output is not a mechanism, consequence, or signal. Also flag internal mechanics, rare exceptions, and repetition. Flag missing context only when readers need it, and do not require every entry to explain a mechanism, consequence, signal, action, and trade-off. 3. **Clarity and format:** plain technical English, clear actors and actions, and suitable examples and components. +4. **Sync Rules containment:** flag any Sync Rules mention, example, or tab outside `sync/rules/` and `snippets/sync-shared/` that does not state a behavior difference, and any new Sync Rules content anywhere. Check that split twins, wrappers, and shared snippets keep their counterpart comment. ## Default Output diff --git a/.claude/skills/doc-author/SKILL.md b/.claude/skills/doc-author/SKILL.md index 6ac236dc1..84de0db5d 100644 --- a/.claude/skills/doc-author/SKILL.md +++ b/.claude/skills/doc-author/SKILL.md @@ -18,6 +18,6 @@ Use the canonical Working Process for scope changes and unresolved decisions. 1. Identify the reader, desired outcome, and affected feature or concept. 2. Research the implementation and existing coverage. Read the most relevant related pages and `docs.json`; avoid unnecessary duplication. 3. If a plan is needed, state the proposed pages, structure, and unresolved questions before drafting. -4. Write the update under the canonical standards. Keep the existing structure unless the task requires a change. +4. Write the update under the canonical standards. Cover Sync Streams only and keep Sync Rules content out of it; Sync Rules pages take error fixes only. Keep the existing structure unless the task requires a change. 5. Self-review for accuracy, reader understanding, minimum useful detail, and navigation fit. Run the canonical verification checks relevant to the change. 6. Present the result and any unresolved draft TODOs, or complete the delivery workflow already authorized by the user. diff --git a/.claude/skills/pr-to-docs/SKILL.md b/.claude/skills/pr-to-docs/SKILL.md index d5203a4a8..b74ea0ed0 100644 --- a/.claude/skills/pr-to-docs/SKILL.md +++ b/.claude/skills/pr-to-docs/SKILL.md @@ -33,4 +33,4 @@ Ask before expanding scope, documenting a deprecation that needs migration decis ## 4. Draft and Verify -Apply the canonical writing standards, navigation requirements, and verification checks. Preserve existing structure unless restructuring is part of the approved plan. Flag unresolved facts using the canonical draft-TODO convention and report what must be resolved before publication. +Apply the canonical writing standards, navigation requirements, and verification checks. Document new behavior for Sync Streams only; do not add Sync Rules examples or mentions. Preserve existing structure unless restructuring is part of the approved plan. Flag unresolved facts using the canonical draft-TODO convention and report what must be resolved before publication. diff --git a/docs.json b/docs.json index 50a4eec91..7d5d58fa2 100644 --- a/docs.json +++ b/docs.json @@ -235,7 +235,30 @@ "sync/rules/data-queries", "sync/rules/many-to-many-join-tables", "sync/rules/client-parameters", - "sync/grammar/sync-rules/index" + { + "group": "Reference", + "pages": [ + "sync/rules/supported-sql", + "sync/grammar/sync-rules/index", + "sync/rules/types" + ] + }, + { + "group": "Advanced", + "pages": [ + "sync/rules/reducing-bucket-count", + "sync/rules/prioritized-sync", + "sync/rules/client-id", + "sync/rules/case-sensitivity", + "sync/rules/compatibility", + "sync/rules/storage-version-4", + "sync/rules/sync-data-by-time", + "sync/rules/schemas-and-connections", + "sync/rules/multiple-client-versions", + "sync/rules/partitioned-tables", + "sync/rules/sharded-databases" + ] + } ] } ] @@ -800,10 +823,6 @@ "source": "/usage/sync-rules/operators-and-functions", "destination": "/sync/supported-sql" }, - { - "source": "/sync/rules/supported-sql", - "destination": "/sync/supported-sql" - }, { "source": "/usage/sync-rules/advanced-topics", "destination": "/sync/advanced/overview" diff --git a/package.json b/package.json index d83b56d31..490f97ce2 100644 --- a/package.json +++ b/package.json @@ -4,7 +4,8 @@ "packageManager": "pnpm@11.3.0+sha512.2c403d6594527287672b1f7056343a1f7c3634036a67ffabfcc2b3d7595d843768f8787148d1b57cf7956c90606bbd192857c363af19e96d2d0ec9ec5741d215", "scripts": { "dev": "mintlify dev", - "check:links": "mintlify broken-links --check-anchors --check-snippets" + "check:links": "node scripts/check-links.mjs", + "check:links:mintlify": "mintlify broken-links --check-anchors --check-snippets" }, "devDependencies": { "mintlify": "^4.2.520" diff --git a/scripts/check-links.mjs b/scripts/check-links.mjs new file mode 100644 index 000000000..a2fa6c6c8 --- /dev/null +++ b/scripts/check-links.mjs @@ -0,0 +1,177 @@ +#!/usr/bin/env node +/** + * Snippet-aware link check. + * + * Runs `mintlify broken-links --check-anchors --check-snippets`, then re-validates + * every reported `#anchor` against the headings of the target page *including* + * headings that come from snippets the page imports. The Mintlify checker only + * reads page files, so a page whose body lives in a snippet (for example the pages + * shared between the Sync Streams and Sync Rules sections, see snippets/sync-shared/) + * would otherwise fail for every inbound anchor link. + * + * Reported links without a fragment, and anchors that still cannot be found, are + * printed in the Mintlify format and make the script exit with status 1. + */ +import { spawnSync } from 'node:child_process'; +import { existsSync, readFileSync } from 'node:fs'; +import path from 'node:path'; + +const root = process.cwd(); +const localBin = path.join(root, 'node_modules', '.bin', 'mintlify'); +const bin = existsSync(localBin) ? localBin : 'mintlify'; + +const run = spawnSync(bin, ['broken-links', '--check-anchors', '--check-snippets'], { + cwd: root, + encoding: 'utf8', + maxBuffer: 64 * 1024 * 1024, +}); +if (run.error) { + console.error(`Could not run ${bin}: ${run.error.message}`); + process.exit(1); +} + +const output = ((run.stdout ?? '') + (run.stderr ?? '')) + .replace(/\x1B\[[0-9;?]*[A-Za-z]/g, '') + .replace(/\r/g, ''); + +// Parse blocks of "\n ⎿ \n ⎿ ". +const flagged = []; +let currentFile = null; +for (const rawLine of output.split('\n')) { + const line = rawLine.trim(); + if (!line) continue; + if (/checking for broken links/.test(line)) continue; + if (/^found \d+ broken link/.test(line)) continue; + if (/^success/i.test(line)) continue; + const link = line.match(/^⎿\s*(\S.*)$/); + if (link) { + if (currentFile) flagged.push({ file: currentFile, link: link[1].trim() }); + continue; + } + currentFile = line; +} + +if (run.status === 0 && flagged.length === 0) { + console.log('success no broken links found'); + process.exit(0); +} +if (flagged.length === 0) { + // Non-zero exit without a parsable report: show what Mintlify printed. + console.log(output.trim()); + process.exit(run.status ?? 1); +} + +function pageFile(urlPath) { + const p = urlPath.replace(/^\//, '').replace(/\/$/, ''); + for (const candidate of [`${p}.mdx`, `${p}.md`, `${p}/index.mdx`, `${p}/index.md`]) { + if (existsSync(path.join(root, candidate))) return candidate; + } + return null; +} + +function snippetFile(spec) { + const rel = spec.startsWith('/') ? spec.slice(1) : path.posix.join('snippets', spec); + return existsSync(path.join(root, rel)) ? rel : null; +} + +// Page source plus the source of every snippet it imports, recursively. +function collectSource(file, seen = new Set()) { + if (seen.has(file)) return ''; + seen.add(file); + const src = readFileSync(path.join(root, file), 'utf8').replace(/^---\n[\s\S]*?\n---\n/, ''); + let out = src; + for (const m of src.matchAll(/^import\s+\w+\s+from\s+['"]([^'"]+)['"]/gm)) { + const f = snippetFile(m[1]); + if (f) out += `\n${collectSource(f, seen)}`; + } + for (const m of src.matchAll(/]*>/g, ' ') + .replace(/\[([^\]]*)\]\([^)]*\)/g, '$1') + .replace(/[`*]/g, '') + .trim() + .toLowerCase(); + const strict = base.replace(/[^a-z0-9\s_-]/g, '').trim().replace(/\s+/g, '-'); + const keepPunctuation = base.replace(/[^a-z0-9\s_/:-]/g, '').trim().replace(/\s+/g, '-'); + return new Set([strict, keepPunctuation, strict.replace(/-+/g, '-'), keepPunctuation.replace(/-+/g, '-')]); +} + +const anchorCache = new Map(); +function anchorsFor(file) { + if (anchorCache.has(file)) return anchorCache.get(file); + const src = collectSource(file).replace(/```[\s\S]*?```/g, ''); + const anchors = new Set(); + for (const m of src.matchAll(/^#{1,6}\s+(.+?)\s*$/gm)) { + let heading = m[1]; + const explicit = heading.match(/\{#([^}]+)\}\s*$/); + if (explicit) { + anchors.add(explicit[1]); + heading = heading.replace(/\{#[^}]+\}\s*$/, ''); + } + for (const slug of slugCandidates(heading)) anchors.add(slug); + } + for (const m of src.matchAll(/]*\btitle=["']([^"']+)["']/g)) { + for (const slug of slugCandidates(m[1])) anchors.add(slug); + } + for (const m of src.matchAll(/\bid=["']([^"']+)["']/g)) anchors.add(m[1]); + for (const m of src.matchAll(/<(?:ResponseField|ParamField)\b[^>]*\bname=["']([^"']+)["']/g)) { + anchors.add(`param-${m[1].replace(/_/g, '-')}`); + } + anchorCache.set(file, anchors); + return anchors; +} + +const unresolved = []; +for (const item of flagged) { + const hash = item.link.indexOf('#'); + if (hash < 0) { + unresolved.push(item); // a path problem, not an anchor problem + continue; + } + const target = item.link.slice(0, hash).split('?')[0]; + let anchor = item.link.slice(hash + 1); + try { + anchor = decodeURIComponent(anchor); + } catch { + // keep the raw fragment + } + const file = target === '' ? item.file : pageFile(target); + if (!file) { + unresolved.push(item); + continue; + } + const anchors = anchorsFor(file); + const withoutSuffix = anchor.replace(/-\d+$/, ''); + if (anchors.has(anchor) || (withoutSuffix !== anchor && anchors.has(withoutSuffix))) continue; + unresolved.push(item); +} + +const plural = (n, word) => `${n} ${word}${n === 1 ? '' : 's'}`; +if (unresolved.length === 0) { + console.log(`success no broken links found (${plural(flagged.length, 'anchor')} resolved through imported snippets)`); + process.exit(0); +} + +const byFile = new Map(); +for (const item of unresolved) { + if (!byFile.has(item.file)) byFile.set(item.file, []); + byFile.get(item.file).push(item.link); +} +console.log(`found ${plural(unresolved.length, 'broken link')} in ${plural(byFile.size, 'file')}\n`); +for (const [file, links] of byFile) { + console.log(file); + for (const link of links) console.log(` ⎿ ${link}`); + console.log(); +} +process.exit(1); diff --git a/snippets/sync-shared/case-sensitivity.mdx b/snippets/sync-shared/case-sensitivity.mdx new file mode 100644 index 000000000..da99add96 --- /dev/null +++ b/snippets/sync-shared/case-sensitivity.mdx @@ -0,0 +1,40 @@ +{/* Shared body: rendered by sync/advanced/case-sensitivity.mdx (Sync Streams section) and sync/rules/case-sensitivity.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} + +### Case in Sync Rules + +PowerSync converts all table/collection and column/field names to lower-case by default in Sync Rule queries (this is how Postgres also behaves). To preserve the case, surround the names with double quotes, for example: + +```sql +SELECT "ID" as id, "Description", "ListID" FROM "TODOs" WHERE "TODOs"."ListID" = bucket.list_id +``` + +When using `SELECT *`, the original case is preserved for the returned columns/fields. + +### Client-Side Case + +On the client side, the case of table and column names in the [client-side schema](/intro/setup-guide#define-your-client-side-schema) must match the case produced by Sync Rules exactly. For the above example, use the following in Dart: + +```dart + Table('TODOs', [ + Column.text('Description'), + Column.text('ListID') + ]) +``` + +SQLite itself is case-insensitive. When querying and modifying the data on the client, any case may be used. For example, the above table may be queried using `SELECT description FROM todos WHERE listid = ?`. + +Operations (`PUT`/`PATCH`/`DELETE`) are stored in the upload queue using the case as defined in the schema above for table and column names, not the case used in queries. + +As another example, in this Sync Rule query: + +```sql +SELECT ID, todo_description as Description FROM todo_items as TODOs +``` + +Each identifier in the example is unquoted and converted to lower case. That means the client-side schema would be: + +```dart +Table('todos', [ + Column.text('description') +]) +``` diff --git a/snippets/sync-shared/client-id.mdx b/snippets/sync-shared/client-id.mdx new file mode 100644 index 000000000..f6ec13cd5 --- /dev/null +++ b/snippets/sync-shared/client-id.mdx @@ -0,0 +1,65 @@ +{/* Shared body: rendered by sync/advanced/client-id.mdx (Sync Streams section) and sync/rules/client-id.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} + +For tables where the client will create new rows: + +- Postgres, MySQL and SQL Server: use a UUID for `id`. Use the `uuid()` helper to generate a random UUID (v4) on the client. +- MongoDB: use an `ObjectId` for `_id`. Generate an `ObjectId()` in your app code and store it in the client's `id` column as a string; this will map to MongoDB's `_id`. + +To use a different column/field from the server-side database as the record ID on the client, use a column/field alias in your [Sync Streams](/sync/streams/overview) query (or [Sync Rules](/sync/rules/overview) data query): + +```sql +SELECT client_id as id FROM my_data +``` + + + MongoDB uses `_id` as the name of the ID field in collections. You must use `SELECT _id as id` (and include any other columns you need) in [Sync Streams](/sync/streams/overview) queries and [Sync Rules](/sync/rules/overview) data queries when using MongoDB as the backend source database. When inserting new documents from the client, prefer `ObjectId` values for `_id` (stored in the client's `id` column). + + +Custom transformations can also be used for the ID column. This is useful in certain scenarios for example when dealing with join tables, because PowerSync doesn't currently support composite primary keys. For example: + +```sql +-- Concatenate multiple columns into a single id column +SELECT *, item_id || '.' || category_id as id FROM item_categories + +-- the source database schema for the above example is CREATE TABLE item_categories(item_id uuid, category_id uuid, PRIMARY KEY(item_id, category_id)); +``` + + + For multiple columns with the same name (e.g. if there was an `id` column in `*`), the last column wins. Prefer writing the `*` before other columns for this reason. + + If you want to upload data to a table with a custom record ID, ensure that `uploadData()` isn't blindly using a field named `id` when handling CRUD operations. See the [Sequential ID mapping tutorial](/client-sdks/advanced/sequential-id-mapping#update-client-to-use-uuids) for an example where the record ID is aliased to `uuid` on the backend. + + +PowerSync does not perform any validation that IDs are unique. Duplicate IDs on a client could occur in any of these scenarios: + +1. A non-unique column is used for the ID. +2. Multiple table partitions are used (Postgres), with the same ID present in different partitions. +3. Multiple data queries returning the same record. This is typically not an issue if the queries return the same values (same transformations used in each query). + +We recommend using a unique index on the fields in the source database to ensure uniqueness — this will prevent (1) at least. + +If the client does sync multiple records with the same ID, only one will be present in the final database. This would typically be the one modified last, but this is subject to change — do not depend on any specific record being picked. + +### Postgres: Strategies for Auto-Incrementing IDs + +With auto-incrementing / sequential IDs (e.g. `sequence` type in Postgres), the issue is that the ID can only be generated on the server, and not on the client while offline. If this _must_ be used, there are some options, depending on the use case. + +#### Option 1: Generate ID when server receives record + +If the client does not use the ID as a reference (foreign key) elsewhere, insert any unique value on the client in the `id` field, then generate a new ID when the server receives it. + +#### Option 2: Pre-create records on the server + +For some use cases, it could work to have the server pre-create a set of e.g. 100 draft records for each user. While offline, the client can populate these records without needing to generate new IDs. This is similar to providing an employee with a paper book of blank invoices — each with an invoice number pre-printed. + +This does mean that a user has a limit on how many records can be populated while offline. + +Care must be taken if a user can populate the same records from different devices while offline — ideally each device must have a unique set of pre-created records. + +#### Option 3: Use an ID mapping + +Use UUIDs on the client, then map them to sequential IDs when performing an update on the server. This allows using a sequential primary key for each record, with a UUID as a secondary ID. + +This mapping must be performed wherever the UUIDs are referenced, including for every foreign key column. + +For more information, have a look at [Sequential ID Mapping](/client-sdks/advanced/sequential-id-mapping). diff --git a/snippets/sync-shared/compatibility.mdx b/snippets/sync-shared/compatibility.mdx new file mode 100644 index 000000000..6642437f1 --- /dev/null +++ b/snippets/sync-shared/compatibility.mdx @@ -0,0 +1,217 @@ +{/* Shared body: rendered by sync/advanced/compatibility.mdx (Sync Streams section) and sync/rules/compatibility.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} + +To ensure consistency, it is important that the PowerSync Service does not interpret the same source row in different ways after updating to a new version. +At the same time, we want to fix bugs or other inaccuracies that have accumulated during the development of the Service. + +## Overview + +To make this trade‑off explicit, you choose whether to keep the existing behavior or turn on newer fixes that slightly change how data is processed. + +Use the `config` block in your Sync Config YAML to choose the behavior. There are two ways to turn fixes on: + +1. Set an `edition` to enable the full set of fixes for that edition. This is the recommended approach for new projects. +2. Toggle individual options for more fine‑grained control. + +For older projects, the previous behavior remains the default. New projects should enable all current fixes. + +### Configuration + +For new projects, it is recommended to enable all current fixes by setting `edition: `: + +```yaml +config: + edition: 3 # Recommended to set to the latest available edition (see 'Supported fixes' table below) + +streams: + # ... +``` + +Or, specify options individually: + +```yaml +config: + timestamps_iso8601: true + versioned_bucket_ids: true + fixed_json_extract: true + custom_postgres_types: true +``` + +## Sync Streams Requirement + +**New Sync Streams configurations should use `edition: 3`**, which enables the new compiler with an expanded SQL feature set (including `JOIN`, CTEs, multiple queries per stream, `BETWEEN`, `CASE`, and more): + +```yaml +config: + edition: 3 + +streams: + my_stream: + query: SELECT * FROM my_table WHERE user_id = auth.user_id() +``` + + +**Upgrading from alpha**: If you have existing Sync Streams using `edition: 2`, upgrade to `edition: 3` to enable the new compiler with an expanded SQL feature set (including `JOIN`, CTEs, multiple queries per stream, `BETWEEN`, `CASE`, and more). See [Supported SQL](/sync/supported-sql) for the full list of supported features. + + +## Storage Version + +A storage version tells the PowerSync Service how to organize prepared sync data in the [bucket storage database](/architecture/powersync-service#bucket-storage). + +Changing the version does not rewrite the current data in place. When you next deploy the Sync Config, PowerSync prepares a new copy using the selected version. Clients continue using the current copy until the new one is ready. This avoids taking the instance offline for a bucket storage migration. + +### Optional `config.storage_version` + +You can choose the bucket storage version in the `config` block: + +```yaml +config: + edition: 3 + storage_version: 4 + +streams: + todos: + query: SELECT * FROM todos WHERE owner_id = auth.user_id() +``` + +When you omit `storage_version`, the PowerSync Service uses its default, which is version 2 in v1.26.0. On PowerSync Cloud, PowerSync manages the default. For self-hosted deployments, set `config.storage_version` explicitly to select a different version. + +Set `storage_version` when you need to: + +- Use [storage version 4](/sync/advanced/storage-version-4), which is in Beta and enables incremental reprocessing and S3 object storage. +- Delay a storage upgrade. When the default moves to a newer version, pin `storage_version` to the version your data already uses. This keeps later Sync Config deployments on that format. Remove the pin when you are ready for the new format. +- Prepare for a Service downgrade. Select a version supported by the older Service, deploy the Sync Config, and wait for the new copy to finish before downgrading. + +### Available Versions + +All PowerSync Cloud instances use MongoDB bucket storage, so they are compatible with all available storage versions. Self-hosted instances with Postgres bucket storage can use versions 1 and 2 only. + +| Version | Bucket storage | Status | +| --- | --- | --- | +| `1` | MongoDB or Postgres | Legacy format, retained for existing deployments. | +| `2` | MongoDB or Postgres | Stable. The default in v1.26.0. | +| `3` | MongoDB | Experimental. The unstable predecessor of version 4, with the same format. Do not use it in production. Deploy with version 4 instead. | +| `4` | MongoDB | Stable. Enables [incremental reprocessing and S3 object storage](/sync/advanced/storage-version-4) (Beta). | + +Version numbers follow a pattern. Even numbers are stable formats: they stay backwards compatible and later Service versions continue to support them. Stable makes no guarantee that a format is bug-free. Odd numbers are experimental formats: their layout can change without notice and support can be removed in a later release, so use them only for testing, never in production. + +## Supported Fixes + +This table lists all fixes currently supported: + +| Name | Explanation | Added in Service version | Fixed in edition | +|----------------------------|------------------------------------|--------------|------------------| +| `timestamps_iso8601` | [Link](#timestamps_iso8601) | 1.15.0 | 2 | +| `versioned_bucket_ids` | [Link](#versioned_bucket_ids) | 1.15.0 | 2 | +| `fixed_json_extract` | [Link](#fixed_json_extract) | 1.15.0 | 2 | +| `custom_postgres_types` | [Link](#custom_postgres_types) | 1.15.3 | 2 | +| `unstable_sqlite_expression_engine` | [Link](#unstable_sqlite_expression_engine). | 1.22.0 | None (unstable) | + +### `timestamps_iso8601` + +PowerSync is supposed to encode timestamps according to the ISO-8601 standard. +Without this fix, the service encoded timestamps from MongoDB and Postgres source databases incorrectly. +To ensure time values from Postgres compare lexicographically, they're also padded to six digits of accuracy when encoded. +Since MongoDB only stores values with an accuracy of milliseconds, only three digits of accuracy are used. + +For instance, the value `2025-09-22T14:29:30` would be encoded as follows: + +- For Postgres: `2025-09-22 14:29:30` without the fix, `2025-09-22T14:29:30.000000` with the fix applied. +- For MongoDB: `2025-09-22 14:29:30.000` without the fix, `2025-09-22T14:29:30.000` with the fix applied. + +Note that MySQL has never been affected by this issue, and thus behaves the same regardless of the option used. + +#### Configurable Sub-Second Datetime Precision + +When the `timestamps_iso8601` option is enabled, PowerSync will sync date and time values with a higher +precision depending on the source database. +You can use the `timestamp_max_precision` option to configure the actual precision to use. +For instance, a Postgres timestamp value would sync as `2025-09-22T14:29:30.000000` by default. +If you don't want that level of precision, you can use the following options to make it sync as `2025-09-22T14:29:30.000`: + +```yaml sync-config.yaml +config: + edition: 3 + timestamp_max_precision: milliseconds +``` + +Valid options for `timestamp_max_precision` are `seconds`, `milliseconds`, `microseconds` and `nanoseconds`. When an explicit +value is given, all synced time values will use that precision. +If a source value has a higher precision, it will be truncated (it is not rounded). +If a source value has a lower precision, it will be padded (so setting the option to `microseconds` with a MongoDB source database +will sync values as `2025-09-22T14:29:30.123000`, with the last three sub-second digits always being set to zero). + +If no option is given, the default precision depends on the source database: + +| Source database | Default precision | Max precision | Notes | +|-----------------|-------------------|---------------|---------------------------------------------------------------------------------------------------------| +| MongoDB | Milliseconds | Milliseconds | | +| Postgres | Microseconds | Microseconds | | +| MySQL | Milliseconds | Microseconds | Defaults to milliseconds, but can be expanded with the option. | +| SQL Server | Nanoseconds | Nanoseconds | SQL Server supports 7 digits of accuracy, the sync service pads values to always use 9 for nanoseconds. | + +### `versioned_bucket_ids` + +Sync Rules define buckets, which rows to sync are then assigned to. When you run a full defragmentation or +redeploy Sync Rules, the same bucket identifiers are re-used when processing data again. + +Because the second iteration uses different checksums for the same bucket ids, clients may sync data +twice before realizing that something is off and starting from scratch. + +Applying this fix improves client-side progress estimation and is more efficient, since data would not get +downloaded twice. + +For how bucket identifiers are represented in bucket storage at the persistence layer (including automatic use of versioned bucket names with newer storage formats), see [Storage version](#storage-version). + +### `fixed_json_extract` + +This fixes the `json_extract` functions as well as the `->` and `->>` operators in Sync Rules to behave similar +to recent SQLite versions: We only split on `.` if the path starts with `$.`. + +For instance, `'json_extract({"foo.bar": "baz"}', 'foo.bar')` would evaluate to: + +1. `baz` with the option enabled. +2. `null` with the option disabled. + +### `custom_postgres_types` + +If you have custom Postgres types in your backend source database schema, older versions of the PowerSync Service +would not recognize these values and sync them with the textual wire representation used by Postgres. +This is especially noticeable when defining `DOMAIN` types with e.g. a `REAL` inner type: The wrapped +`DOMAIN` type should get synced as a real value as well, but it would actually get synced as a string. + +With this fix applied: + +- `DOMAIN TYPE`s are synced as their inner type. +- Array types of custom types get parsed correctly, and sync as a JSON array. +- Custom types get parsed and synced as a JSON object containing their members. +- Ranges sync as a JSON object corresponding to the following TypeScript definition: + ```TypeScript + export type Range = + | { + lower: T | null; + upper: T | null; + lower_exclusive: boolean; + upper_exclusive: boolean; + } + | 'empty'; + ``` +- Multi-ranges sync as an array of ranges. + +### `unstable_sqlite_expression_engine` + + +This option is experimental: When enabled, updates to the PowerSync Service might change how rows are processed +and this option may be removed in a future version of the Service. + + +Sync Streams support scalar SQL operators (like `+`, `-` and `||`) and [functions](/sync/supported-sql#functions). +SQL in Sync Streams should behave exactly as it would in SQLite, but the Service uses a custom implementation which differs +from SQLite for some edge cases. + +To perfectly align the behavior of the Service and SQLite, enabling this option makes the Service use an actual +SQLite database to evaluate Sync Streams. +Some known issues with the JavaScript evaluator that are fixed by this option are: + +- Exact null handling: `NOT NULL` evaluates to `TRUE` without this option, enabling it yields `NULL`. +- Without this option, `substr()` and `length()` operate on UTF-16 code units. Enabling it makes them operate on + Unicode code points. diff --git a/snippets/sync-shared/reducing-bucket-count.mdx b/snippets/sync-shared/reducing-bucket-count.mdx new file mode 100644 index 000000000..84568d37c --- /dev/null +++ b/snippets/sync-shared/reducing-bucket-count.mdx @@ -0,0 +1,240 @@ +{/* Shared body: rendered by sync/advanced/reducing-bucket-count.mdx (Sync Streams section) and sync/rules/reducing-bucket-count.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} + +import BucketCountExampleApp from '/snippets/bucket-count-example-app.mdx'; + +If a user syncs too many buckets, or you hit a `PSYNC_S2305` error, this page shows how to find the cause and bring the count down. For how buckets are counted in the first place, see [Bucket Count](/sync/streams/bucket-count). + +PowerSync enforces two limits per user, both with a default of 1,000. One is the number of unique buckets. The other is the number of parameter query results, counted before duplicates are removed. Exceeding either fails the sync with a `PSYNC_S2305` error. The fix is different for each, so start by finding out which one you hit from the error message. See [Limits](/sync/streams/bucket-count#limits) for the full difference. + +## Diagnosing High Bucket Count + +### Reading the Error Message First + +The `PSYNC_S2305` message tells you which limit you reached. The fix is different for each, so read it first. + +- `Too many buckets` means you reached the bucket limit. Reduce the number of unique buckets. Any strategy below helps. +- `Too many parameter query results` means you reached the parameter limit. Reduce the rows your parameter lookups return. Only some strategies help here: [Denormalizing the Scope Key](#denormalizing-the-scope-key) and [Querying the Membership Table Directly](#querying-the-membership-table-directly) cut the lookups themselves, so they lower both counts. + +```mermaid +flowchart TD + E["PSYNC_S2305 error"] --> M{"Which message?"} + M -->|"Too many buckets"| Bk["Reduce unique buckets"] + M -->|"Too many parameter query results"| Pr["Reduce parameter rows"] + Bk --> D["Denormalize the scope key,
or merge streams"] + Pr --> D +``` + +### The Contributor Breakdown + +The `PSYNC_S2305` log includes a breakdown of the streams that contribute the most. + +- For a bucket-limit error, it lists streams by bucket count, highest first. +- For a parameter-limit error, it lists the streams that returned the most rows, and then the stream that exceeded the limit. Each listed stream shows how many rows it returned. The failing stream instead shows how much budget was left when it failed. + + +For a parameter-limit error, the last stream in the breakdown is the one that ran when the limit was reached. This stream is not always the cause. PowerSync adds up parameter results across streams in order. The last stream is only the one that exceeded the limit. Check every stream in the breakdown, not just the last one. + + +### Checkpoint Logs + +Checkpoint logs record the counts for each connection. Find them in your [instance logs](/maintenance-ops/monitoring-and-alerting). For example: + +```text +New checkpoint: 800178 | write: null | buckets: 7 | param_results: 6 ["5#org_data|0[\"ef718ff3...\"]","5#org_data|1[\"1ddeddba...\"]", ...] +``` + +- `buckets` is the number of unique buckets for this connection. +- `param_results` is the total number of parameter rows for this connection. +- The array lists the bucket names. Each name already includes its parameter value. The list stops after 20 names. + +### Sync Diagnostics Client + +The [Sync Diagnostics Client](/tools/diagnostics-client) shows the buckets for one user. It does not load for a user who is over the limit, because that user's sync fails before the data loads. Use the instance logs and the error breakdown for those users. The client shows the bucket count, which may not be the limit you reached. Confirm the limit from the error message. + + + +## Reducing Bucket Count + +Start with the strategy that matches your query pattern. Most high counts come from hierarchical or many-to-many data, where denormalizing the scope key gives the biggest reduction. + +### Multiple Queries per Stream + +**Reduces:** bucket count. + +Use `queries` instead of separate streams to group related tables. All queries in a stream that filter the same way share one bucket per value. See [multiple queries per stream](/sync/streams/queries#multiple-queries-per-stream). + +**Before**: 5 separate streams, each with a direct `auth.user_id()` filter, create 5 buckets per user. + +**After**: 1 stream with 5 queries creates 1 bucket per user. + +```yaml +streams: + user_settings: # [!code --] + query: SELECT * FROM settings WHERE user_id = auth.user_id() # [!code --] + user_prefs: # [!code --] + query: SELECT * FROM preferences WHERE user_id = auth.user_id() # [!code --] + user_org_list: # [!code --] + query: SELECT * FROM org_membership WHERE user_id = auth.user_id() # [!code --] + user_region: # [!code --] + query: SELECT * FROM region_members WHERE user_id = auth.user_id() # [!code --] + user_profile: # [!code --] + query: SELECT * FROM profiles WHERE user_id = auth.user_id() # [!code --] + user_data: # [!code ++] + queries: # [!code ++] + - SELECT * FROM settings WHERE user_id = auth.user_id() # [!code ++] + - SELECT * FROM preferences WHERE user_id = auth.user_id() # [!code ++] + - SELECT * FROM org_membership WHERE user_id = auth.user_id() # [!code ++] + - SELECT * FROM region_members WHERE user_id = auth.user_id() # [!code ++] + - SELECT * FROM profiles WHERE user_id = auth.user_id() # [!code ++] +``` + +### Denormalizing the Scope Key + +**Reduces:** bucket count and parameter query results. + +This is the most effective fix for parent-child data. When chained queries through org → project → task create too many buckets, filter every table with the same top-level parameter, such as `org_id`. A bucket's key must be a column on the table you sync (see [The Partition Key Must Exist on the Row](#the-partition-key-must-exist-on-the-row) below). So this works only if the child tables have that column. If tasks only have `project_id`, add `org_id` to the tasks table. + +**Before**: chained queries create 10 + 500 = 510 buckets for 10 orgs with 50 projects each. Projects and tasks share buckets because they use the same filter. Orgs use a different filter, so they add their own buckets. + +**After**: add `org_id` to the tasks table, drop the `user_projects` CTE, and filter every table by org. This creates 10 buckets. + +```yaml +streams: + org_projects_tasks: + with: + user_orgs: SELECT org_id FROM org_membership WHERE user_id = auth.user_id() + user_projects: SELECT id FROM projects WHERE org_id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) # [!code --] + queries: + - SELECT * FROM orgs WHERE id IN user_orgs + - SELECT * FROM projects WHERE id IN user_projects # [!code --] + - SELECT * FROM projects WHERE org_id IN user_orgs # [!code ++] + - SELECT * FROM tasks WHERE project_id IN user_projects # [!code --] + - SELECT * FROM tasks WHERE org_id IN user_orgs # [!code ++] +``` + +### Querying the Membership Table Directly + +**Reduces:** bucket count and parameter query results. + +When a subquery or JOIN through a membership table creates N buckets, query the membership table directly with a direct auth filter. Use no subquery and no JOIN. You often need fields from the related table, such as the org name, alongside each membership row. Denormalize those fields onto the membership table so they are available without a JOIN. + +**Before**: N org memberships create N buckets. + +**After**: 1 bucket per user, with org fields denormalized onto `org_membership`. + +```yaml +streams: + org_data: # [!code --] + query: SELECT * FROM orgs WHERE id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) # [!code --] + my_org_memberships: # [!code ++] + query: SELECT * FROM org_membership WHERE user_id = auth.user_id() # [!code ++] +``` + +### Many-to-Many via a JSON Array Column + +**Reduces:** bucket count. + +A join through a link table creates one bucket per row of the table you select from. For assets linked to projects through `project_assets`, you get one bucket per asset. + +Add a denormalized `project_ids` JSON array column to `assets`, maintained with database triggers. Then use `json_each()` to traverse it. This lets PowerSync key the bucket by project ID instead of asset ID. + +**Before**: one bucket per asset. 2,000 assets create 2,000 buckets. + +**After**: key by project. 50 projects create 50 buckets. + +```yaml +streams: + assets_in_projects: + with: + user_projects: SELECT id FROM projects WHERE org_id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) + query: SELECT assets.* FROM assets JOIN project_assets ON project_assets.asset_id = assets.id WHERE project_assets.project_id IN user_projects # [!code --] + query: SELECT assets.* FROM assets INNER JOIN json_each(assets.project_ids) AS p INNER JOIN user_projects ON p.value = user_projects.id # [!code ++] +``` + +The `INNER JOIN user_projects` syncs only assets that belong to at least one of the user's projects. The bucket key is the project ID, so the count matches the number of projects, not assets. + +### Subscription Parameters for On-Demand Sync + +**Reduces:** bucket count. + +Buckets are created per active subscription, not from every possible value. Use `subscription.parameter('project_id')` so the count is bounded by how many subscriptions the client has active. + +**Before**: a subquery returns all of the user's projects. 50 projects create 50 buckets. + +**After**: the client subscribes per project on demand. 3 open projects create 3 buckets. + +```yaml +streams: + project_tasks: + with: + user_projects: SELECT id FROM projects WHERE org_id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) + query: SELECT * FROM tasks WHERE project_id IN user_projects # [!code --] + query: SELECT * FROM tasks WHERE project_id = subscription.parameter('project_id') AND project_id IN user_projects # [!code ++] +``` + +The client subscribes when the user opens a project and unsubscribes when they leave. This works only when the user does not need every record available offline at the same time. + +## Edge Cases and Gotchas + +### The Partition Key Must Exist on the Row + +A bucket's key must be a value that physically exists on a row of the table you sync. You cannot split a table into buckets by a column it does not have. This is why denormalizing the scope key onto child tables is the standard fix. If tasks only have `project_id`, you cannot key their buckets by `org_id` until you add `org_id` to the tasks table. + +### Subscription Parameters Choose Buckets, Not Re-Partition Them + +A subscription parameter lets the client choose which existing buckets to sync. It does not change how those buckets are defined. + +For a parameter to select a bucket, its value must match a value on the row being synced. For example, each task has a `project_id`, so you can use that column to group tasks into project buckets: + +```yaml +streams: + project_tasks: + query: SELECT * FROM tasks WHERE project_id = subscription.parameter('project_id') +``` + +Assets are different. An asset can belong to multiple projects, so the asset row does not have a single `project_id`. Passing a `project_id` as a subscription parameter therefore cannot make PowerSync group those assets by project. The asset row has no project ID to match against. + +If you want to sync assets by project, the asset row needs to contain a project reference first. For example, you could add a `project_ids` array column as described in [Reducing Bucket Count](#reducing-bucket-count). + +### Correlated Joins Behave Like Subqueries + +A correlated JOIN and an `IN (subquery)` compile to the same internal form. They create the same number of buckets. Rewriting one as the other does not reduce the count. + +### CTEs Cannot Reference Each Other + +Each CTE must be self-contained. A CTE cannot reference another CTE by name. If it does, the deploy fails. Inline the nested subquery instead. See [CTE limitations](/sync/streams/ctes#limitations). + +### Global Buckets Multiply Storage and Cost + +A stream with no filter creates one global bucket that every user syncs. Under `auto_subscribe: true`, every write to that table fans out to every user. This drives up synced data volume and cost. Scope global buckets carefully, and only mark truly shared reference data as global. + +### Bucket Storage Does Not Shrink When You Archive + +Buckets are append-only. Marking a row as archived does not remove it from bucket storage on its own. A row leaves storage only when it stops matching the data query, through a hard delete or a filter on the table's own column. Storage reclaims space during [compaction](/maintenance-ops/compacting-buckets). Filtering through a parent table does not shrink a child table's stored data. + +## Increasing the Limit + +Raise the limit only after you exhaust the reduction strategies above. + +Before you raise it, weigh the cost. Sync overhead scales roughly linearly with the number of buckets per user. Doubling the bucket count roughly doubles sync latency for a single operation. It also roughly doubles CPU and memory use on the server and the client. Many operations inside a single bucket scale much more efficiently than many buckets. The 1,000 default exists to encourage fewer, larger buckets and to protect the service from excessive counts. + +On PowerSync Cloud, you can request a higher limit on [Team and Enterprise](https://www.powersync.com/pricing) plans, up to 10,000. The limit applies per user, so your instance can still track far more buckets in total. + +For self-hosted deployments, set the limits under `api.parameters`: + +```yaml service.yaml +api: + parameters: + max_buckets_per_connection: 5000 + max_parameter_query_results: 5000 +``` + +Set both. Raising one without the other still leaves you capped by the limit you did not change. + +## Related Pages + +- [Bucket Count](/sync/streams/bucket-count) explains how buckets are counted and the two limits. +- [Writing Queries](/sync/streams/queries) covers the query syntax that determines your bucket count. +- [Common Table Expressions (CTEs)](/sync/streams/ctes) covers shared filtering logic. +- [Troubleshooting](/debugging/troubleshooting#psync_s2305-too-many-buckets-/-parameter-query-results) covers the `PSYNC_S2305` error. +- [Performance and Limits](/resources/performance-and-limits) lists the Service limits. diff --git a/snippets/sync-shared/schemas-and-connections.mdx b/snippets/sync-shared/schemas-and-connections.mdx new file mode 100644 index 000000000..f922962c8 --- /dev/null +++ b/snippets/sync-shared/schemas-and-connections.mdx @@ -0,0 +1,60 @@ +{/* Shared body: rendered by sync/advanced/schemas-and-connections.mdx (Sync Streams section) and sync/rules/schemas-and-connections.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} + +## Schemas (Postgres) + +When no schema is specified, the Postgres `public` schema is used for every query. A different schema can be specified as a prefix: + +```sql +-- Note: the schema must be in double quotes +SELECT * FROM "other"."assets" +``` + +## Wildcard Schemas (Postgres) + + + Wildcard schemas require [Sync Streams](/sync/streams/overview) and PowerSync Service v1.24.0 or later. They are currently only supported for Postgres connections. + + +Use `%` as a wildcard in the schema name to match tables with the same name across multiple schemas. `"%"` matches every schema, and a prefix such as `"tenant_%"` matches every schema whose name starts with `tenant_`. The wildcard can only be the last character of the schema name. Postgres system schemas (`pg_*` and `information_schema`) are never matched. + +Combine a wildcard schema with the [`schema()` function](/sync/supported-sql#functions), which returns the schema each row was replicated from, to filter rows by schema. This supports schema-per-tenant databases (a single database with one identical schema per tenant): one stream covers every tenant schema, and each client syncs only its own tenant's data, resolved from a JWT claim. + +```yaml +config: + edition: 3 + +streams: + work_orders: + query: SELECT * FROM "%".work_orders WHERE work_orders.schema() = auth.parameter('tenant_schema') +``` + +In this example, rows are grouped into a bucket per schema, and each client syncs only the bucket matching the `tenant_schema` claim in its JWT. Rows from all matched schemas sync into a single client-side table, named after the table in the query (`work_orders` here). + + + Each matched table must be part of the [PowerSync publication](/configuration/source-db/setup#3-create-powersync-publication). Tables that are not in the publication are skipped. + + +## High Availability / Replicated Databases (Postgres) + +When the source Postgres database is replicated, for example with Amazon RDS Multi-AZ deployments, specify a single connection with multiple host endpoints. Each host endpoint will be tried in sequence, with the first available primary connection being used. + +For this, each endpoint must point to the same physical database, with the same replication slots. This is the case when block-level replication is used between the databases, but not when streaming physical or logical replication is used. In those cases, replication slots are unique on each host, and all data would be re-synced in a fail-over event. + +## Multiple Separate Database Connections (Planned) + + + This feature will be available in a future release. See this [item on our roadmap](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections). + + +In the future, it will be possible to configure PowerSync with multiple separate source database connections, where each connection is concurrently replicated. + +You should not add multiple connections to multiple replicas of the same database — this would cause data duplication. Only use this when the data on each connection does not overlap. + +It will be possible for each connection to be configured with a "tag", to distinguish these connections in Sync Rules. The same tag may be used for multiple connections (if the schema is the same in each). + +By default, queries will reference the "default" tag. To use a different connection or connections, assign a different tag, and specify it in the query as a schema prefix. In this case, the schema itself must also be specified. + +```sql +-- Note the usage of quotes here +SELECT * FROM "secondconnection.public"."assets" +``` diff --git a/snippets/sync-shared/sharded-databases.mdx b/snippets/sync-shared/sharded-databases.mdx new file mode 100644 index 000000000..9ea54cd92 --- /dev/null +++ b/snippets/sync-shared/sharded-databases.mdx @@ -0,0 +1,44 @@ +{/* Shared body: rendered by sync/advanced/sharded-databases.mdx (Sync Streams section) and sync/rules/sharded-databases.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} + +In the case of Postgres, PowerSync cannot replicate Postgres [foreign tables](https://www.postgresql.org/docs/current/ddl-foreign-data.html). + +However, PowerSync does have options available to support sharded databases in general. + + + When using MongoDB, MySQL, or SQL Server as the backend source database, PowerSync does not currently support connecting to sharded clusters. + + +The primary options are: + +1. Use a separate PowerSync Service instance per database. +2. Add a connection for each database in the same PowerSync Service instance ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release). + +Where feasible, using separate PowerSync Service instances would give better performance and give more control over how changes are rolled out, especially around Sync Rule reprocessing. + +Some specific scenarios: + +#### 1\. Different Tables on Different Databases + +This is common when separate "services" use separate databases, but multiple tables across those databases need to be synced to the same users. + +Use a single PowerSync Service instance, with a separate connection for each source database ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release). Use a unique [connection tag](/sync/advanced/schemas-and-connections) for each source database, allowing them to be distinguished in your [Sync Streams](/sync/streams/overview) or [Sync Rules](/sync/rules/overview). + +#### 2a. All Data for a Single Customer Is Contained in a Single Shard + +This is common when sharding per customer account / organization. + +In this case, use a separate PowerSync Service instance for each database. + +#### 2b. Most Customer Data Is in a Single Shard, but Some Data Is in a Shared Database + +If the amount of shared data is small, still use a separate PowerSync Service instance for each database, but also add the shared database connection to each PowerSync Service instance using a separate connection tag ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release). + +#### 2c. Data for a Single Customer Is Split Across Multiple Shards + +This is more complicated than the other cases listed above. Please [reach out to us](/resources/contact-us) if this is your architecture. + +#### 3\. Only Some Tables Are Sharded + +In some cases, most tables would be on a shared server, with only a few large tables being sharded. + +For this case, use a single PowerSync Service instance. Add each shard as a new connection on this instance ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release) — all with the same connection tag, so that the same [Sync Streams](/sync/streams/overview) or [Sync Rules](/sync/rules/overview) applies to each. diff --git a/snippets/sync-shared/storage-version-4.mdx b/snippets/sync-shared/storage-version-4.mdx new file mode 100644 index 000000000..17a4df47d --- /dev/null +++ b/snippets/sync-shared/storage-version-4.mdx @@ -0,0 +1,170 @@ +{/* Shared body: rendered by sync/advanced/storage-version-4.mdx (Sync Streams section) and sync/rules/storage-version-4.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} + +Storage version 4 is a new version of the format the PowerSync Service uses to store the data it syncs to clients. It is in [Beta](/resources/feature-status) as of PowerSync Service v1.26.0. + +Compared to version 2, it provides: + +- Faster sync and faster reprocessing after a deployment. +- [Incremental reprocessing](#incremental-reprocessing): a Sync Streams deployment reprocesses only the streams you added or changed. Clients no longer download all their data again after every deployment. +- [S3 object storage](#s3-object-storage): larger blocks of synced data move from the storage database to S3. This reduces load on the storage database when many clients sync at once or sync large amounts of data. + +## Availability + +Storage version 4 is compatible with all PowerSync Cloud instances, which already use MongoDB [bucket storage](/architecture/powersync-service#bucket-storage). Self-hosted instances must also use MongoDB bucket storage. Postgres bucket storage is not currently supported. + +The PowerSync Cloud and self-hosted columns below apply during the Beta only. Once storage version 4 is generally available, it will become the default for all supported instances. S3 object storage is then also enabled on all PowerSync Cloud instances. For self-hosted deployments, follow the [S3 setup instructions](#self-hosted-s3-setup). + +| | Source database | Sync Config | PowerSync Cloud (Beta) | Self-hosted (Beta) | +| --- | --- | --- | --- | --- | +| Storage version 4 | Any | Sync Streams or Sync Rules | Free plan: automatic. Other plans: [opt in](#opt-in). | [Opt in](#opt-in) | +| Incremental reprocessing | MongoDB | Sync Streams | Included with version 4 | Included with version 4 | +| S3 object storage | Any | Sync Streams or Sync Rules | Enabled per instance by PowerSync on request | [Set up S3 object storage](#self-hosted-s3-setup) | + +Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/rules/migrate-to-sync-streams). + +## Opt In + +Version 4 is not the default in PowerSync Service v1.26.0. Moving a Sync Config to version 4 runs like any other deployment: + +1. PowerSync reprocesses all data selected by your Sync Config in the background. The current version keeps serving clients, so there is no downtime. +2. When the new copy is ready, PowerSync switches to it. On PowerSync Cloud, this appears as a new deploy event in the PowerSync Dashboard. +3. Clients download their data again once, as after any deployment before version 4. On self-hosted deployments with many clients, scale out the API before the switch to absorb the re-sync. + +After this first deployment, later Sync Streams deployments use incremental reprocessing automatically when your instance meets its requirements. There is no separate setting. + +### PowerSync Cloud + +Free plan instances are upgraded automatically during the Beta. No action is needed. + +On other plans, add `storage_version: 4` to the `config` block of your Sync Config and deploy it: + +```yaml +config: + edition: 3 + storage_version: 4 + +streams: + todos: + query: SELECT * FROM todos WHERE owner_id = auth.user_id() +``` + +### Self-Hosted + + + Postgres bucket storage is not supported with version 4. + + +Add `storage_version: 4` to the `config` block of each Sync Config as shown above, then deploy or redeploy it to use version 4. + +To move a Sync Config back to version 2, set `storage_version: 2` and deploy again. This is another full reprocess. + +To also enable S3 object storage, follow the [self-hosted S3 setup instructions](#self-hosted-s3-setup) to prepare a bucket and configure the Service. + +## Incremental Reprocessing + +Incremental reprocessing is active when you use a MongoDB source database, Sync Streams, and storage version 4. + + + Self-hosted instances with Postgres bucket storage are not supported. + + +Without it, every deployment reads all data selected by the Sync Config from your source database and prepares a complete new copy. Clients then download all their data again, even if only one stream changed. + +With incremental reprocessing, PowerSync compares the new Sync Config with the current one and reprocesses only the streams you added or changed. Unchanged streams keep their data on the PowerSync Service and on clients. Deployments finish faster, your source database does less work, and clients download only the data for affected streams they subscribe to. + +- Adding a stream reads only the data that stream selects. +- Removing a stream requires no new source reads. PowerSync cleans up stored definitions when no active Sync Config still uses them. +- Renaming a stream counts as removing it and adding a new one, so its data is rebuilt. +- Changing a stream's queries may reprocess affected definitions. Changes that only affect how request parameters select existing buckets do not require reprocessing. + +For example, changing `SELECT * FROM projects WHERE user_id = auth.user_id()` to `SELECT * FROM projects WHERE user_id = auth.jwt() ->> 'owner'` reuses the existing bucket data. The data is still grouped by `user_id`; only the JWT field used to select buckets changes. + +The time saved depends on how your data is split across streams. If one stream selects most of your data, changing that stream still takes about as long as a full reprocess. + +PowerSync favors correctness over reuse. When it cannot confirm that a change leaves a stream's data unchanged, it rebuilds that stream. A deployment that reprocesses more than you expect is not an error. + +Event definitions for [Custom Write Checkpoints](/handling-writes/custom-write-checkpoints) follow the same rules. Unchanged events keep their data, and new or changed events are read again. + +### When PowerSync Reprocesses Everything + +Some changes start a full reprocess, after which clients download all their data again: + +- The first deployment on storage version 4. +- Changes to the `config` block of the Sync Config, such as `edition`, compatibility fixes, or `storage_version`. +- The **Defragment** action in the PowerSync Dashboard, which exists to rebuild all data. See [Defragmenting](/maintenance-ops/compacting-buckets#defragmenting). +- Replication failures, for example when PowerSync loses its position in the MongoDB change stream and has to start over. + +### Sync Config Versions and Replication Streams + +Each deployment has a Sync Config version. With incremental reprocessing, multiple versions can share a replication stream, the replication process and stored state. A full reprocess creates a new replication stream. + +See the [Log Reference](/debugging/log-reference#message-prefixes) for how to identify these versions and streams in your logs. + +For implementation details, see the [storage design](https://github.com/powersync-ja/powersync-service/blob/main/docs/storage/storage-v3.md). The document describes the design introduced in version 3 and carried into version 4. + +### Checking What a Deployment Reprocessed + +If a deployment takes longer or reprocesses more than you expect, see [Checking What a Deployment Reprocessed](/debugging/log-reference#checking-what-a-deployment-reprocessed) in the Log Reference for what to look for in your logs. + +## S3 Object Storage + +Your instance keeps the data it syncs to clients in its bucket storage database, alongside everything else it needs to run. With S3 object storage, larger blocks of that data move to Amazon S3 or an S3-compatible object store, and the PowerSync Service syncs them to clients directly from there. Smaller blocks, and the metadata that locates each block, stay in MongoDB. + +Reading larger blocks from S3 reduces the data MongoDB must read and transfer during sync. When those reads limit performance, offloading them can speed up initial sync and let an instance serve more concurrent clients. The benefit is most noticeable when clients sync large amounts of data or many clients connect at once. The PowerSync Service still handles every client connection, so its CPU and memory capacity also limit concurrency. + +For self-hosted instances, offloading bucket data to S3 can reduce storage and data transfer costs. Compare the reduction in database costs with the object store's storage, request, and data transfer charges for your workload. + +Clients connect only to the PowerSync Service and never to the object store, so no client changes are needed. If the object store becomes unreachable, sync is interrupted until it recovers. Clients reconnect and resume automatically. + +S3 object storage requires storage version 4 and works with Sync Streams and legacy Sync Rules. It is compatible with all PowerSync Cloud instances. + + + S3 object storage holds PowerSync's internal sync data. To store files uploaded by your app, use [Attachments](/client-sdks/advanced/attachments). + + +### PowerSync Cloud + +During the Beta, PowerSync enables S3 object storage per instance. [Contact us](/resources/contact-us) if you want it on your instance before we enable it for all instances. + +### Self-Hosted S3 Setup + + + Self-hosted instances with Postgres bucket storage are not supported. + + + + + Create a bucket. Use the same region as the PowerSync Service where possible, to keep latency low and avoid cross-region data transfer charges. Use a dedicated bucket, or a unique `prefix` per PowerSync instance, so that instances never read or delete each other's files. Give the PowerSync Service permission to list the bucket and to read, write, and delete objects under the prefix. + + Leave object versioning off, or suspend it if the bucket already has it, and leave Object Lock off. PowerSync deletes files itself once they are no longer needed, so versioning keeps charging for old versions and locked objects cannot be cleaned up. Do not add an expiration lifecycle rule: an expired object may still be referenced by MongoDB, which breaks sync for that data. + + + Add `object_storage` to the `storage` section of `service.yaml`: + + ```yaml service.yaml + storage: + type: mongodb + uri: !env PS_MONGO_STORAGE_URI + object_storage: + type: s3 + bucket: powersync-bucket-data + region: us-east-1 + prefix: production + ``` + + Without `access_key_id` and `secret_access_key`, PowerSync uses the AWS credentials available to the process, such as an IAM role. For S3-compatible providers such as MinIO or Cloudflare R2, also set `endpoint`, and set `force_path_style: true` if the provider requires path-style requests. + + Restart or redeploy the PowerSync Service to load the updated `service.yaml`. If you run replication, API, and compacting in separate containers or jobs, apply the same object storage configuration to each. + + + Deploy your Sync Configs on storage version 4 as described in [Opt In](#opt-in). Sync Configs on version 2 keep all data in MongoDB, even when `object_storage` is configured. + + Once replication reaches a healthy checkpoint, confirm that objects appear under the prefix, run a test initial sync, and run `compact` once to surface permission errors early. + + + +After enabling S3 object storage, you can raise [`max_concurrent_connections`](/configuration/powersync-service/self-hosted-instances#param-max-concurrent-connections) from its default of 200 per API process. With storage version 4 and S3 object storage, each API process can handle up to 1,000 concurrent client connections. Performance degrades if a large share of those clients run an initial sync at the same time, so scale out the API before a deployment that makes all clients download their data again. More concurrent connections also increase CPU and memory usage. + +The [S3 object storage configuration reference](/configuration/powersync-service/self-hosted-instances#param-object-storage) lists all supported settings, including timeouts, request concurrency, and the size threshold below which blocks stay in MongoDB. + +Keep the scheduled [compact](/maintenance-ops/compacting-buckets) job running. It removes files that are no longer needed. The `teardown` command deletes PowerSync's files under the prefix before it drops the storage database. The `powersync_object_storage_size_bytes` [metric](/maintenance-ops/self-hosting/monitoring) reports how much object storage PowerSync uses. diff --git a/snippets/sync-shared/types.mdx b/snippets/sync-shared/types.mdx new file mode 100644 index 000000000..e3ea89acc --- /dev/null +++ b/snippets/sync-shared/types.mdx @@ -0,0 +1,164 @@ +{/* Shared body: rendered by sync/types.mdx (Sync Streams section) and sync/rules/types.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} + +import BinaryType from '/snippets/binary-type.mdx'; + +The supported client-side SQLite types are: + +1. `null` +2. `integer`: a 64-bit signed integer +3. `real`: a 64-bit floating point number +4. `text`: A UTF-8 text string +5. `blob`: Binary data + + +## Postgres Type Mapping + +Postgres types are mapped to SQLite types as follows: + +| Postgres Data Type | PowerSync / SQLite Column Type | Notes | +|--------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `text`, `varchar` | `text` | | +| `int2`, `int4`, `int8` | `integer` | | +| `numeric` / `decimal` | `text` | These types have arbitrary precision in Postgres, so can only be represented accurately as text in SQLite | +| `bool` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | +| `float4`, `float8` | `real` | | +| `enum` | `text` | | +| `uuid` | `text` | | +| `timestamptz` | `text` | Format: `YYYY-MM-DD hh:mm:ss.sssZ`. This is compatible with ISO8601 and SQLite's functions. Precision matches the precision used in Postgres. `-infinity` becomes `0000-01-01 00:00:00Z` and `infinity` becomes `9999-12-31 23:59:59Z`. | +| `timestamp` | `text` | Format: `YYYY-MM-DD hh:mm:ss.sss`. In most cases, timestamptz should be used instead. `-infinity` becomes `0000-01-01 00:00:00` and `infinity` becomes `9999-12-31 23:59:59`. | +| `date`, `time` | `text` | | +| `json`, `jsonb` | `text` | `json` and `jsonb` values are treated as `text` values in their serialized representation. [JSON functions and operators](/sync/supported-sql#operators) operate directly on these `text` values. | +| `interval` | `text` | | +| `macaddr` | `text` | | +| `inet` | `text` | | +| `bytea` | `blob` | Cannot sync directly to client — convert to hex or base64 first. See [Operators & Functions](/sync/supported-sql). | +| `geometry` (PostGIS) | `text` | Hex string of the binary data. Use the [ST functions](/sync/supported-sql#functions) to convert to other formats | +| Arrays | `text` | JSON array. | +| `DOMAIN` types | `text` / depends | Depending on [compatibility options](/sync/advanced/compatibility#custom_postgres_types), inner type or raw wire representation (legacy). | +| Custom types | `text` | Depending on [compatibility options](/sync/advanced/compatibility#custom_postgres_types), JSON object or raw wire representation (legacy). | +| (Multi-)ranges | `text` | Depending on [compatibility options](/sync/advanced/compatibility#custom_postgres_types), JSON object (array for multi-ranges) or raw wire representation (legacy). | + + + + +## Convex Type Mapping + + + The Convex replicator is currently released as an [experimental feature](/resources/feature-status). APIs and + behavior may change, and we can't yet guarantee continued support or long-term stability. + + +Convex values are mapped to SQLite types as follows: + +| Convex Type | TS/JS Type | PowerSync / SQLite Column Type | Notes | +| ----------- | ---------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `Id` | `string` | `text` | Convex document IDs are exposed as `_id` and can be synced as `text`. For synced client tables, use client-side ID mapping with a stable UUID column as `id` instead of relying on Convex-generated `_id` values. | +| `Null` | `null` | `null` | | +| `Int64` | `base-10 string` | `text` | Cast to `INTEGER` in Sync Streams when you want to sync the value as a SQLite integer. | +| `Float64` | `number` | `real` | | +| `Boolean` | `boolean` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | +| `String` | `string` | `text` | | +| `Bytes` | `base64 string` | `text` | Decode from base64 in your app if you need binary data on the client. | +| `Array` | `Array` | `text` | Converted to a JSON string. | +| `Object` | `Object` | `text` | Converted to a JSON string. | +| `Record` | `Record` | `text` | Converted to a JSON string. | + +- Convex documents are converted to a flat list of columns, one column per top-level field. +- Nested objects and arrays are converted to JSON, and [JSON functions and operators](/sync/supported-sql#operators) can be used to query them in Sync Streams or on the client-side SQLite database. +- Cast Convex `Int64` fields to `INTEGER` in Sync Streams when you want SQLite integer values on the client, for example `CAST(an_int64_column AS INTEGER) AS an_int64_column`. + + +## MongoDB Type Mapping + +MongoDB types are mapped to SQLite types as follows: + +| BSON Type | PowerSync / SQLite Column Type | Notes | +|--------------------|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------| +| `String` | `text` | | +| `Int`, `Long` | `integer` | | +| `Double` | `real` | | +| `Decimal128` | `text` | | +| `Object` | `text` | Converted to a JSON string | +| `Array` | `text` | Converted to a JSON string | +| `ObjectId` | `text` | Lower-case hex string | +| `UUID` | `text` | Lower-case hex string | +| `Boolean` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | +| `Date` | `text` | Format: `YYYY-MM-DD hh:mm:ss.sssZ` | +| `Null` | `null` | | +| `Binary` | `blob` | Cannot sync directly to client — convert to hex or base64 first. See [Operators & Functions](/sync/supported-sql). | +| Regular Expression | `text` | JSON text in the format `{"pattern":"...","options":"..."}` | +| `Timestamp` | `integer` | Converted to a 64-bit integer | +| `Undefined` | `null` | | +| `DBPointer` | `text` | JSON text in the format `{"collection":"...","oid":"...","db":"...","fields":...}` | +| `JavaScript` | `text` | JSON text in the format `{"code": "...", "scope": ...}` | +| `Symbol` | `text` | | +| `MinKey`, `MaxKey` | `null` | | + +* Data is converted to a flat list of columns, one column per top-level field in the MongoDB document. +* Special BSON types are converted to plain SQLite alternatives. For example, `ObjectId`, `Date`, `UUID` are all converted to a plain `TEXT` column. +* Nested objects and arrays are converted to JSON, and [JSON functions and operators](/sync/supported-sql#operators) can be used to query them (in the Sync Streams / Sync Rules and/or on the client-side SQLite statements). +* Binary data nested in objects or arrays is not supported. + + + + +## MySQL Type Mapping + +MySQL support is currently in a [Beta release](/resources/feature-status). + +MySQL types are mapped to SQLite types as follows: + +| MySQL Data Type | PowerSync / SQLite Column Type | Notes | +|----------------------------------------------------|--------------------------------|-----------------------------------------------------------------------------------| +| `tinyint`, `smallint`, `mediumint`, `bigint`, `integer`, `int` | `integer` | | +| `numeric`, `decimal` | `text` | | +| `bool`, `boolean` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | +| `float`, `double`, `real` | `real` | | +| `enum` | `text` | | +| `set` | `text` | Converted to JSON array | +| `char`, `varchar` | `text` | | +| `tinytext`, `text`, `mediumtext`, `longtext` | `text` | | +| `timestamp` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | +| `date` | `text` | Format: `YYYY-MM-DD` | +| `time`, `datetime` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | +| `year` | `text` | | +| `json` | `text` | There is no dedicated JSON type in SQLite — JSON functions operate directly on text values. | +| `bit` | `blob` | * See note below regarding syncing binary types | +| `binary`, `varbinary` | `blob` | | +| `image` | `blob` | | +| `geometry`, `geometrycollection` | `blob` | | +| `point`, `multipoint` | `blob` | | +| `linestring`, `multilinestring` | `blob` | | +| `polygon`, `multipolygon` | `blob` | | + + + + +## SQL Server Type Mapping + +SQL Server support is currently in a [Beta release](/resources/feature-status). + +SQL Server types are mapped to SQLite types as follows: + +| SQL Server Data Type | PowerSync / SQLite Column Type | Notes | +|----------------------------------------------------|--------------------------------|--------------------------------------------------------| +| `tinyint`, `smallint`, `int`, `bigint` | `integer` | | +| `numeric`, `decimal` | `text` | Numeric string | +| `float`, `real` | `real` | | +| `bit` | `integer` | | +| `money`, `smallmoney` | `text` | Numeric string | +| `xml` | `text` | | +| `char`, `nchar`, `ntext` | `text` | | +| `varchar`, `nvarchar`, `text` | `text` | | +| `uniqueidentifier` | `text` | | +| `timestamp` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | +| `date` | `text` | Format: `YYYY-MM-DD` | +| `time` | `text` | Format: `HH:mm:ss.sss` | +| `datetime`, `datetime2`, `smalldatetime`, `datetimeoffset` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | +| `json` | `text` | Only exists for Azure SQL Database and SQL Server 2025 | +| `geometry`, `geography` | `text` | `text` of JSON object describing the spatial data type | +| `binary`, `varbinary`, `image` | `blob` | * See note below regarding binary types | +| `rowversion`, `timestamp` | `blob` | * See note below regarding binary types | +| User Defined Types: `hiearchyid` | `blob` | * See note below regarding binary types | + + diff --git a/sync/advanced/case-sensitivity.mdx b/sync/advanced/case-sensitivity.mdx index 3591e8f74..67cf813e7 100644 --- a/sync/advanced/case-sensitivity.mdx +++ b/sync/advanced/case-sensitivity.mdx @@ -3,41 +3,8 @@ title: "Case Sensitivity" description: "Handle case-sensitive table and column names in PowerSync Sync Streams/Rules, with best practices for lowercase identifiers and quoting strategies." --- -### Case in Sync Rules +{/* Wrapper page: the content is snippets/sync-shared/case-sensitivity.mdx, which also renders at sync/rules/case-sensitivity.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} -PowerSync converts all table/collection and column/field names to lower-case by default in Sync Rule queries (this is how Postgres also behaves). To preserve the case, surround the names with double quotes, for example: +import CaseSensitivity from '/snippets/sync-shared/case-sensitivity.mdx'; -```sql -SELECT "ID" as id, "Description", "ListID" FROM "TODOs" WHERE "TODOs"."ListID" = bucket.list_id -``` - -When using `SELECT *`, the original case is preserved for the returned columns/fields. - -### Client-Side Case - -On the client side, the case of table and column names in the [client-side schema](/intro/setup-guide#define-your-client-side-schema) must match the case produced by Sync Rules exactly. For the above example, use the following in Dart: - -```dart - Table('TODOs', [ - Column.text('Description'), - Column.text('ListID') - ]) -``` - -SQLite itself is case-insensitive. When querying and modifying the data on the client, any case may be used. For example, the above table may be queried using `SELECT description FROM todos WHERE listid = ?`. - -Operations (`PUT`/`PATCH`/`DELETE`) are stored in the upload queue using the case as defined in the schema above for table and column names, not the case used in queries. - -As another example, in this Sync Rule query: - -```sql -SELECT ID, todo_description as Description FROM todo_items as TODOs -``` - -Each identifier in the example is unquoted and converted to lower case. That means the client-side schema would be: - -```dart -Table('todos', [ - Column.text('description') -]) -``` + diff --git a/sync/advanced/client-id.mdx b/sync/advanced/client-id.mdx index c60869117..5a1683bca 100644 --- a/sync/advanced/client-id.mdx +++ b/sync/advanced/client-id.mdx @@ -3,66 +3,8 @@ title: "Client ID" description: "Understand PowerSync's requirement for a single text-type primary key column called id." --- -For tables where the client will create new rows: +{/* Wrapper page: the content is snippets/sync-shared/client-id.mdx, which also renders at sync/rules/client-id.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} -- Postgres, MySQL and SQL Server: use a UUID for `id`. Use the `uuid()` helper to generate a random UUID (v4) on the client. -- MongoDB: use an `ObjectId` for `_id`. Generate an `ObjectId()` in your app code and store it in the client's `id` column as a string; this will map to MongoDB's `_id`. +import ClientId from '/snippets/sync-shared/client-id.mdx'; -To use a different column/field from the server-side database as the record ID on the client, use a column/field alias in your [Sync Streams](/sync/streams/overview) query (or [Sync Rules](/sync/rules/overview) data query): - -```sql -SELECT client_id as id FROM my_data -``` - - - MongoDB uses `_id` as the name of the ID field in collections. You must use `SELECT _id as id` (and include any other columns you need) in [Sync Streams](/sync/streams/overview) queries and [Sync Rules](/sync/rules/overview) data queries when using MongoDB as the backend source database. When inserting new documents from the client, prefer `ObjectId` values for `_id` (stored in the client's `id` column). - - -Custom transformations can also be used for the ID column. This is useful in certain scenarios for example when dealing with join tables, because PowerSync doesn't currently support composite primary keys. For example: - -```sql --- Concatenate multiple columns into a single id column -SELECT *, item_id || '.' || category_id as id FROM item_categories - --- the source database schema for the above example is CREATE TABLE item_categories(item_id uuid, category_id uuid, PRIMARY KEY(item_id, category_id)); -``` - - - For multiple columns with the same name (e.g. if there was an `id` column in `*`), the last column wins. Prefer writing the `*` before other columns for this reason. - - If you want to upload data to a table with a custom record ID, ensure that `uploadData()` isn't blindly using a field named `id` when handling CRUD operations. See the [Sequential ID mapping tutorial](/client-sdks/advanced/sequential-id-mapping#update-client-to-use-uuids) for an example where the record ID is aliased to `uuid` on the backend. - - -PowerSync does not perform any validation that IDs are unique. Duplicate IDs on a client could occur in any of these scenarios: - -1. A non-unique column is used for the ID. -2. Multiple table partitions are used (Postgres), with the same ID present in different partitions. -3. Multiple data queries returning the same record. This is typically not an issue if the queries return the same values (same transformations used in each query). - -We recommend using a unique index on the fields in the source database to ensure uniqueness — this will prevent (1) at least. - -If the client does sync multiple records with the same ID, only one will be present in the final database. This would typically be the one modified last, but this is subject to change — do not depend on any specific record being picked. - -### Postgres: Strategies for Auto-Incrementing IDs - -With auto-incrementing / sequential IDs (e.g. `sequence` type in Postgres), the issue is that the ID can only be generated on the server, and not on the client while offline. If this _must_ be used, there are some options, depending on the use case. - -#### Option 1: Generate ID when server receives record - -If the client does not use the ID as a reference (foreign key) elsewhere, insert any unique value on the client in the `id` field, then generate a new ID when the server receives it. - -#### Option 2: Pre-create records on the server - -For some use cases, it could work to have the server pre-create a set of e.g. 100 draft records for each user. While offline, the client can populate these records without needing to generate new IDs. This is similar to providing an employee with a paper book of blank invoices — each with an invoice number pre-printed. - -This does mean that a user has a limit on how many records can be populated while offline. - -Care must be taken if a user can populate the same records from different devices while offline — ideally each device must have a unique set of pre-created records. - -#### Option 3: Use an ID mapping - -Use UUIDs on the client, then map them to sequential IDs when performing an update on the server. This allows using a sequential primary key for each record, with a UUID as a secondary ID. - -This mapping must be performed wherever the UUIDs are referenced, including for every foreign key column. - -For more information, have a look at [Sequential ID Mapping](/client-sdks/advanced/sequential-id-mapping). \ No newline at end of file + diff --git a/sync/advanced/compatibility.mdx b/sync/advanced/compatibility.mdx index 444e6e9a0..286263be8 100644 --- a/sync/advanced/compatibility.mdx +++ b/sync/advanced/compatibility.mdx @@ -3,218 +3,8 @@ title: "Compatibility" description: "Configure compatibility editions and bucket storage format version in PowerSync's Sync Config." --- -To ensure consistency, it is important that the PowerSync Service does not interpret the same source row in different ways after updating to a new version. -At the same time, we want to fix bugs or other inaccuracies that have accumulated during the development of the Service. +{/* Wrapper page: the content is snippets/sync-shared/compatibility.mdx, which also renders at sync/rules/compatibility.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} -## Overview +import Compatibility from '/snippets/sync-shared/compatibility.mdx'; -To make this trade‑off explicit, you choose whether to keep the existing behavior or turn on newer fixes that slightly change how data is processed. - -Use the `config` block in your Sync Config YAML to choose the behavior. There are two ways to turn fixes on: - -1. Set an `edition` to enable the full set of fixes for that edition. This is the recommended approach for new projects. -2. Toggle individual options for more fine‑grained control. - -For older projects, the previous behavior remains the default. New projects should enable all current fixes. - -### Configuration - -For new projects, it is recommended to enable all current fixes by setting `edition: `: - -```yaml -config: - edition: 3 # Recommended to set to the latest available edition (see 'Supported fixes' table below) - -streams: - # ... -``` - -Or, specify options individually: - -```yaml -config: - timestamps_iso8601: true - versioned_bucket_ids: true - fixed_json_extract: true - custom_postgres_types: true -``` - -## Sync Streams Requirement - -**New Sync Streams configurations should use `edition: 3`**, which enables the new compiler with an expanded SQL feature set (including `JOIN`, CTEs, multiple queries per stream, `BETWEEN`, `CASE`, and more): - -```yaml -config: - edition: 3 - -streams: - my_stream: - query: SELECT * FROM my_table WHERE user_id = auth.user_id() -``` - - -**Upgrading from alpha**: If you have existing Sync Streams using `edition: 2`, upgrade to `edition: 3` to enable the new compiler with an expanded SQL feature set (including `JOIN`, CTEs, multiple queries per stream, `BETWEEN`, `CASE`, and more). See [Supported SQL](/sync/supported-sql) for the full list of supported features. - - -## Storage Version - -A storage version tells the PowerSync Service how to organize prepared sync data in the [bucket storage database](/architecture/powersync-service#bucket-storage). - -Changing the version does not rewrite the current data in place. When you next deploy the Sync Config, PowerSync prepares a new copy using the selected version. Clients continue using the current copy until the new one is ready. This avoids taking the instance offline for a bucket storage migration. - -### Optional `config.storage_version` - -You can choose the bucket storage version in the `config` block: - -```yaml -config: - edition: 3 - storage_version: 4 - -streams: - todos: - query: SELECT * FROM todos WHERE owner_id = auth.user_id() -``` - -When you omit `storage_version`, the PowerSync Service uses its default, which is version 2 in v1.26.0. On PowerSync Cloud, PowerSync manages the default. For self-hosted deployments, set `config.storage_version` explicitly to select a different version. - -Set `storage_version` when you need to: - -- Use [storage version 4](/sync/advanced/storage-version-4), which is in Beta and enables incremental reprocessing and S3 object storage. -- Delay a storage upgrade. When the default moves to a newer version, pin `storage_version` to the version your data already uses. This keeps later Sync Config deployments on that format. Remove the pin when you are ready for the new format. -- Prepare for a Service downgrade. Select a version supported by the older Service, deploy the Sync Config, and wait for the new copy to finish before downgrading. - -### Available Versions - -All PowerSync Cloud instances use MongoDB bucket storage, so they are compatible with all available storage versions. Self-hosted instances with Postgres bucket storage can use versions 1 and 2 only. - -| Version | Bucket storage | Status | -| --- | --- | --- | -| `1` | MongoDB or Postgres | Legacy format, retained for existing deployments. | -| `2` | MongoDB or Postgres | Stable. The default in v1.26.0. | -| `3` | MongoDB | Experimental. The unstable predecessor of version 4, with the same format. Do not use it in production. Deploy with version 4 instead. | -| `4` | MongoDB | Stable. Enables [incremental reprocessing and S3 object storage](/sync/advanced/storage-version-4) (Beta). | - -Version numbers follow a pattern. Even numbers are stable formats: they stay backwards compatible and later Service versions continue to support them. Stable makes no guarantee that a format is bug-free. Odd numbers are experimental formats: their layout can change without notice and support can be removed in a later release, so use them only for testing, never in production. - -## Supported Fixes - -This table lists all fixes currently supported: - -| Name | Explanation | Added in Service version | Fixed in edition | -|----------------------------|------------------------------------|--------------|------------------| -| `timestamps_iso8601` | [Link](#timestamps_iso8601) | 1.15.0 | 2 | -| `versioned_bucket_ids` | [Link](#versioned_bucket_ids) | 1.15.0 | 2 | -| `fixed_json_extract` | [Link](#fixed_json_extract) | 1.15.0 | 2 | -| `custom_postgres_types` | [Link](#custom_postgres_types) | 1.15.3 | 2 | -| `unstable_sqlite_expression_engine` | [Link](#unstable_sqlite_expression_engine). | 1.22.0 | None (unstable) | - -### `timestamps_iso8601` - -PowerSync is supposed to encode timestamps according to the ISO-8601 standard. -Without this fix, the service encoded timestamps from MongoDB and Postgres source databases incorrectly. -To ensure time values from Postgres compare lexicographically, they're also padded to six digits of accuracy when encoded. -Since MongoDB only stores values with an accuracy of milliseconds, only three digits of accuracy are used. - -For instance, the value `2025-09-22T14:29:30` would be encoded as follows: - -- For Postgres: `2025-09-22 14:29:30` without the fix, `2025-09-22T14:29:30.000000` with the fix applied. -- For MongoDB: `2025-09-22 14:29:30.000` without the fix, `2025-09-22T14:29:30.000` with the fix applied. - -Note that MySQL has never been affected by this issue, and thus behaves the same regardless of the option used. - -#### Configurable Sub-Second Datetime Precision - -When the `timestamps_iso8601` option is enabled, PowerSync will sync date and time values with a higher -precision depending on the source database. -You can use the `timestamp_max_precision` option to configure the actual precision to use. -For instance, a Postgres timestamp value would sync as `2025-09-22T14:29:30.000000` by default. -If you don't want that level of precision, you can use the following options to make it sync as `2025-09-22T14:29:30.000`: - -```yaml sync-config.yaml -config: - edition: 3 - timestamp_max_precision: milliseconds -``` - -Valid options for `timestamp_max_precision` are `seconds`, `milliseconds`, `microseconds` and `nanoseconds`. When an explicit -value is given, all synced time values will use that precision. -If a source value has a higher precision, it will be truncated (it is not rounded). -If a source value has a lower precision, it will be padded (so setting the option to `microseconds` with a MongoDB source database -will sync values as `2025-09-22T14:29:30.123000`, with the last three sub-second digits always being set to zero). - -If no option is given, the default precision depends on the source database: - -| Source database | Default precision | Max precision | Notes | -|-----------------|-------------------|---------------|---------------------------------------------------------------------------------------------------------| -| MongoDB | Milliseconds | Milliseconds | | -| Postgres | Microseconds | Microseconds | | -| MySQL | Milliseconds | Microseconds | Defaults to milliseconds, but can be expanded with the option. | -| SQL Server | Nanoseconds | Nanoseconds | SQL Server supports 7 digits of accuracy, the sync service pads values to always use 9 for nanoseconds. | - -### `versioned_bucket_ids` - -Sync Rules define buckets, which rows to sync are then assigned to. When you run a full defragmentation or -redeploy Sync Rules, the same bucket identifiers are re-used when processing data again. - -Because the second iteration uses different checksums for the same bucket ids, clients may sync data -twice before realizing that something is off and starting from scratch. - -Applying this fix improves client-side progress estimation and is more efficient, since data would not get -downloaded twice. - -For how bucket identifiers are represented in bucket storage at the persistence layer (including automatic use of versioned bucket names with newer storage formats), see [Storage version](#storage-version). - -### `fixed_json_extract` - -This fixes the `json_extract` functions as well as the `->` and `->>` operators in Sync Rules to behave similar -to recent SQLite versions: We only split on `.` if the path starts with `$.`. - -For instance, `'json_extract({"foo.bar": "baz"}', 'foo.bar')` would evaluate to: - -1. `baz` with the option enabled. -2. `null` with the option disabled. - -### `custom_postgres_types` - -If you have custom Postgres types in your backend source database schema, older versions of the PowerSync Service -would not recognize these values and sync them with the textual wire representation used by Postgres. -This is especially noticeable when defining `DOMAIN` types with e.g. a `REAL` inner type: The wrapped -`DOMAIN` type should get synced as a real value as well, but it would actually get synced as a string. - -With this fix applied: - -- `DOMAIN TYPE`s are synced as their inner type. -- Array types of custom types get parsed correctly, and sync as a JSON array. -- Custom types get parsed and synced as a JSON object containing their members. -- Ranges sync as a JSON object corresponding to the following TypeScript definition: - ```TypeScript - export type Range = - | { - lower: T | null; - upper: T | null; - lower_exclusive: boolean; - upper_exclusive: boolean; - } - | 'empty'; - ``` -- Multi-ranges sync as an array of ranges. - -### `unstable_sqlite_expression_engine` - - -This option is experimental: When enabled, updates to the PowerSync Service might change how rows are processed -and this option may be removed in a future version of the Service. - - -Sync Streams support scalar SQL operators (like `+`, `-` and `||`) and [functions](/sync/supported-sql#functions). -SQL in Sync Streams should behave exactly as it would in SQLite, but the Service uses a custom implementation which differs -from SQLite for some edge cases. - -To perfectly align the behavior of the Service and SQLite, enabling this option makes the Service use an actual -SQLite database to evaluate Sync Streams. -Some known issues with the JavaScript evaluator that are fixed by this option are: - -- Exact null handling: `NOT NULL` evaluates to `TRUE` without this option, enabling it yields `NULL`. -- Without this option, `substr()` and `length()` operate on UTF-16 code units. Enabling it makes them operate on - Unicode code points. + diff --git a/sync/advanced/multiple-client-versions.mdx b/sync/advanced/multiple-client-versions.mdx index 1c1cb9f35..4775b1b9e 100644 --- a/sync/advanced/multiple-client-versions.mdx +++ b/sync/advanced/multiple-client-versions.mdx @@ -3,6 +3,8 @@ title: "Multiple Client Versions" description: "Handle multiple client app versions that require different output schemas from Sync Streams." --- +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/multiple-client-versions.mdx. That page is deprecated and frozen, but when you fix an error here, check whether it needs the same fix. Do not mention Sync Rules on this page. */} + When schema changes are additive, old clients ignore the new tables and columns, and no special handling is required. More drastic changes, such as renaming tables or changing a table's structure, can break older app versions that are still in use. In these cases, define separate versions of the affected [Sync Streams](/sync/streams/overview) so that each client version receives the tables and columns it expects. ## Versioning by Stream Name @@ -33,45 +35,25 @@ Once the older app versions are no longer in use, remove the old stream from you ## Versioning with Connection Parameters -Alternatively, clients can pass their version to the PowerSync Service as a [connection parameter](/sync/streams/parameters#connection-parameters), and stream queries filter on it so each client only receives data for its version. This approach is useful when your streams are auto-subscribed: auto-subscribed streams sync to every client on connect, so clients cannot select a stream version by name. In legacy [Sync Rules](/sync/rules/overview), connection parameters are called [client parameters](/sync/rules/client-parameters). +Alternatively, clients can pass their version to the PowerSync Service as a [connection parameter](/sync/streams/parameters#connection-parameters), and stream queries filter on it so each client only receives data for its version. This approach is useful when your streams are auto-subscribed: auto-subscribed streams sync to every client on connect, so clients cannot select a stream version by name. The example below implements the same `assets` use case, with both stream versions auto-subscribed and filtered by a `schema_version` connection parameter: - - - ```yaml - # Client passes connection params: {"schema_version": } - streams: - user_assets: - auto_subscribe: true - query: SELECT * FROM assets - WHERE user_id = auth.user_id() - AND connection.parameter('schema_version') = '1' - - user_assets_v2: - auto_subscribe: true - query: SELECT * FROM assets AS assets_v2 - WHERE user_id = auth.user_id() - AND connection.parameter('schema_version') = '2' - ``` - - - ```yaml - # Client passes in: "params": {"schema_version": } - user_assets: - parameters: SELECT request.user_id() AS user_id - WHERE request.parameters() ->> 'schema_version' = '1' - data: - - SELECT * FROM assets WHERE user_id = bucket.user_id +```yaml +# Client passes connection params: {"schema_version": } +streams: + user_assets: + auto_subscribe: true + query: SELECT * FROM assets + WHERE user_id = auth.user_id() + AND connection.parameter('schema_version') = '1' - user_assets_v2: - parameters: SELECT request.user_id() AS user_id - WHERE request.parameters() ->> 'schema_version' = '2' - data: - - SELECT * FROM assets AS assets_v2 WHERE user_id = bucket.user_id - ``` - - + user_assets_v2: + auto_subscribe: true + query: SELECT * FROM assets AS assets_v2 + WHERE user_id = auth.user_id() + AND connection.parameter('schema_version') = '2' +``` Handle queries based on parameters set by the client with care. The client can send any value for these parameters, so it's not a good place to do authorization. If the parameter must be authenticated, use parameters from the JWT instead. diff --git a/sync/advanced/partitioned-tables.mdx b/sync/advanced/partitioned-tables.mdx index 251d7b2bf..0c90f8bb8 100644 --- a/sync/advanced/partitioned-tables.mdx +++ b/sync/advanced/partitioned-tables.mdx @@ -3,55 +3,34 @@ title: "Partitioned Tables (Postgres)" description: "Sync data from Postgres partitioned tables using wildcard table name matching." --- -For partitioned tables in Postgres, each individual partition is replicated and processed using [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)). +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/partitioned-tables.mdx. That page is deprecated and frozen, but when you fix an error here, check whether it needs the same fix. Do not mention Sync Rules on this page. */} + +For partitioned tables in Postgres, each individual partition is replicated and processed using [Sync Streams](/sync/streams/overview). To use the same queries and same output table name for each partition, use `%` for wildcard suffix matching of the table name: - - - ```yaml - streams: - user_todos: - queries: - # Wildcard matches all user partition tables (e.g. users_2024, users_2025) - - SELECT * FROM "users_%" WHERE id = auth.user_id() - # Wildcard matches all todo partition tables (e.g. todos_2024, todos_2025) - - SELECT * FROM "todos_%" AS todos WHERE user_id = auth.user_id() - ``` - - - ```yaml - by_user: - # Use wildcard in a parameter query - parameters: SELECT id AS user_id FROM "users_%" - data: - # Use wildcard in a data query - - SELECT * FROM "todos_%" AS todos WHERE user_id = bucket.user_id - ``` - - +```yaml +streams: + user_todos: + queries: + # Wildcard matches all user partition tables (e.g. users_2024, users_2025) + - SELECT * FROM "users_%" WHERE id = auth.user_id() + # Wildcard matches all todo partition tables (e.g. todos_2024, todos_2025) + - SELECT * FROM "todos_%" AS todos WHERE user_id = auth.user_id() +``` The wildcard character can only be used as the last character in the table name. To match tables across multiple schemas instead, see [Wildcard Schemas](/sync/advanced/schemas-and-connections#wildcard-schemas-postgres). -When using wildcard table names, you can filter on the original table suffix. In Sync Streams, use the `table_suffix()` function, prefixed with the table name or alias from the `FROM` clause (requires PowerSync Service v1.24.0 or later). In legacy Sync Rules, the suffix is available as the special `_table_suffix` column instead: - - - - ```yaml - config: - edition: 3 - - streams: - active_todos: - query: SELECT * FROM "todos_%" AS todos WHERE todos.table_suffix() != 'archived' - ``` - - - ```sql - SELECT * FROM "todos_%" AS todos WHERE _table_suffix != 'archived' - ``` - - +When using wildcard table names, you can filter on the original table suffix with the `table_suffix()` function, prefixed with the table name or alias from the `FROM` clause. This requires PowerSync Service v1.24.0 or later: + +```yaml +config: + edition: 3 + +streams: + active_todos: + query: SELECT * FROM "todos_%" AS todos WHERE todos.table_suffix() != 'archived' +``` When no table alias is provided, the original table name is preserved. diff --git a/sync/advanced/reducing-bucket-count.mdx b/sync/advanced/reducing-bucket-count.mdx index 16c42f2f2..37162620d 100644 --- a/sync/advanced/reducing-bucket-count.mdx +++ b/sync/advanced/reducing-bucket-count.mdx @@ -4,241 +4,8 @@ description: "Diagnose a high bucket count, reduce the number of buckets a user sidebarTitle: "Reducing Buckets" --- -import BucketCountExampleApp from '/snippets/bucket-count-example-app.mdx'; +{/* Wrapper page: the content is snippets/sync-shared/reducing-bucket-count.mdx, which also renders at sync/rules/reducing-bucket-count.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} -If a user syncs too many buckets, or you hit a `PSYNC_S2305` error, this page shows how to find the cause and bring the count down. For how buckets are counted in the first place, see [Bucket Count](/sync/streams/bucket-count). +import ReducingBucketCount from '/snippets/sync-shared/reducing-bucket-count.mdx'; -PowerSync enforces two limits per user, both with a default of 1,000. One is the number of unique buckets. The other is the number of parameter query results, counted before duplicates are removed. Exceeding either fails the sync with a `PSYNC_S2305` error. The fix is different for each, so start by finding out which one you hit from the error message. See [Limits](/sync/streams/bucket-count#limits) for the full difference. - -## Diagnosing High Bucket Count - -### Reading the Error Message First - -The `PSYNC_S2305` message tells you which limit you reached. The fix is different for each, so read it first. - -- `Too many buckets` means you reached the bucket limit. Reduce the number of unique buckets. Any strategy below helps. -- `Too many parameter query results` means you reached the parameter limit. Reduce the rows your parameter lookups return. Only some strategies help here: [Denormalizing the Scope Key](#denormalizing-the-scope-key) and [Querying the Membership Table Directly](#querying-the-membership-table-directly) cut the lookups themselves, so they lower both counts. - -```mermaid -flowchart TD - E["PSYNC_S2305 error"] --> M{"Which message?"} - M -->|"Too many buckets"| Bk["Reduce unique buckets"] - M -->|"Too many parameter query results"| Pr["Reduce parameter rows"] - Bk --> D["Denormalize the scope key,
or merge streams"] - Pr --> D -``` - -### The Contributor Breakdown - -The `PSYNC_S2305` log includes a breakdown of the streams that contribute the most. - -- For a bucket-limit error, it lists streams by bucket count, highest first. -- For a parameter-limit error, it lists the streams that returned the most rows, and then the stream that exceeded the limit. Each listed stream shows how many rows it returned. The failing stream instead shows how much budget was left when it failed. - - -For a parameter-limit error, the last stream in the breakdown is the one that ran when the limit was reached. This stream is not always the cause. PowerSync adds up parameter results across streams in order. The last stream is only the one that exceeded the limit. Check every stream in the breakdown, not just the last one. - - -### Checkpoint Logs - -Checkpoint logs record the counts for each connection. Find them in your [instance logs](/maintenance-ops/monitoring-and-alerting). For example: - -```text -New checkpoint: 800178 | write: null | buckets: 7 | param_results: 6 ["5#org_data|0[\"ef718ff3...\"]","5#org_data|1[\"1ddeddba...\"]", ...] -``` - -- `buckets` is the number of unique buckets for this connection. -- `param_results` is the total number of parameter rows for this connection. -- The array lists the bucket names. Each name already includes its parameter value. The list stops after 20 names. - -### Sync Diagnostics Client - -The [Sync Diagnostics Client](/tools/diagnostics-client) shows the buckets for one user. It does not load for a user who is over the limit, because that user's sync fails before the data loads. Use the instance logs and the error breakdown for those users. The client shows the bucket count, which may not be the limit you reached. Confirm the limit from the error message. - - - -## Reducing Bucket Count - -Start with the strategy that matches your query pattern. Most high counts come from hierarchical or many-to-many data, where denormalizing the scope key gives the biggest reduction. - -### Multiple Queries per Stream - -**Reduces:** bucket count. - -Use `queries` instead of separate streams to group related tables. All queries in a stream that filter the same way share one bucket per value. See [multiple queries per stream](/sync/streams/queries#multiple-queries-per-stream). - -**Before**: 5 separate streams, each with a direct `auth.user_id()` filter, create 5 buckets per user. - -**After**: 1 stream with 5 queries creates 1 bucket per user. - -```yaml -streams: - user_settings: # [!code --] - query: SELECT * FROM settings WHERE user_id = auth.user_id() # [!code --] - user_prefs: # [!code --] - query: SELECT * FROM preferences WHERE user_id = auth.user_id() # [!code --] - user_org_list: # [!code --] - query: SELECT * FROM org_membership WHERE user_id = auth.user_id() # [!code --] - user_region: # [!code --] - query: SELECT * FROM region_members WHERE user_id = auth.user_id() # [!code --] - user_profile: # [!code --] - query: SELECT * FROM profiles WHERE user_id = auth.user_id() # [!code --] - user_data: # [!code ++] - queries: # [!code ++] - - SELECT * FROM settings WHERE user_id = auth.user_id() # [!code ++] - - SELECT * FROM preferences WHERE user_id = auth.user_id() # [!code ++] - - SELECT * FROM org_membership WHERE user_id = auth.user_id() # [!code ++] - - SELECT * FROM region_members WHERE user_id = auth.user_id() # [!code ++] - - SELECT * FROM profiles WHERE user_id = auth.user_id() # [!code ++] -``` - -### Denormalizing the Scope Key - -**Reduces:** bucket count and parameter query results. - -This is the most effective fix for parent-child data. When chained queries through org → project → task create too many buckets, filter every table with the same top-level parameter, such as `org_id`. A bucket's key must be a column on the table you sync (see [The Partition Key Must Exist on the Row](#the-partition-key-must-exist-on-the-row) below). So this works only if the child tables have that column. If tasks only have `project_id`, add `org_id` to the tasks table. - -**Before**: chained queries create 10 + 500 = 510 buckets for 10 orgs with 50 projects each. Projects and tasks share buckets because they use the same filter. Orgs use a different filter, so they add their own buckets. - -**After**: add `org_id` to the tasks table, drop the `user_projects` CTE, and filter every table by org. This creates 10 buckets. - -```yaml -streams: - org_projects_tasks: - with: - user_orgs: SELECT org_id FROM org_membership WHERE user_id = auth.user_id() - user_projects: SELECT id FROM projects WHERE org_id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) # [!code --] - queries: - - SELECT * FROM orgs WHERE id IN user_orgs - - SELECT * FROM projects WHERE id IN user_projects # [!code --] - - SELECT * FROM projects WHERE org_id IN user_orgs # [!code ++] - - SELECT * FROM tasks WHERE project_id IN user_projects # [!code --] - - SELECT * FROM tasks WHERE org_id IN user_orgs # [!code ++] -``` - -### Querying the Membership Table Directly - -**Reduces:** bucket count and parameter query results. - -When a subquery or JOIN through a membership table creates N buckets, query the membership table directly with a direct auth filter. Use no subquery and no JOIN. You often need fields from the related table, such as the org name, alongside each membership row. Denormalize those fields onto the membership table so they are available without a JOIN. - -**Before**: N org memberships create N buckets. - -**After**: 1 bucket per user, with org fields denormalized onto `org_membership`. - -```yaml -streams: - org_data: # [!code --] - query: SELECT * FROM orgs WHERE id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) # [!code --] - my_org_memberships: # [!code ++] - query: SELECT * FROM org_membership WHERE user_id = auth.user_id() # [!code ++] -``` - -### Many-to-Many via a JSON Array Column - -**Reduces:** bucket count. - -A join through a link table creates one bucket per row of the table you select from. For assets linked to projects through `project_assets`, you get one bucket per asset. - -Add a denormalized `project_ids` JSON array column to `assets`, maintained with database triggers. Then use `json_each()` to traverse it. This lets PowerSync key the bucket by project ID instead of asset ID. - -**Before**: one bucket per asset. 2,000 assets create 2,000 buckets. - -**After**: key by project. 50 projects create 50 buckets. - -```yaml -streams: - assets_in_projects: - with: - user_projects: SELECT id FROM projects WHERE org_id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) - query: SELECT assets.* FROM assets JOIN project_assets ON project_assets.asset_id = assets.id WHERE project_assets.project_id IN user_projects # [!code --] - query: SELECT assets.* FROM assets INNER JOIN json_each(assets.project_ids) AS p INNER JOIN user_projects ON p.value = user_projects.id # [!code ++] -``` - -The `INNER JOIN user_projects` syncs only assets that belong to at least one of the user's projects. The bucket key is the project ID, so the count matches the number of projects, not assets. - -### Subscription Parameters for On-Demand Sync - -**Reduces:** bucket count. - -Buckets are created per active subscription, not from every possible value. Use `subscription.parameter('project_id')` so the count is bounded by how many subscriptions the client has active. - -**Before**: a subquery returns all of the user's projects. 50 projects create 50 buckets. - -**After**: the client subscribes per project on demand. 3 open projects create 3 buckets. - -```yaml -streams: - project_tasks: - with: - user_projects: SELECT id FROM projects WHERE org_id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) - query: SELECT * FROM tasks WHERE project_id IN user_projects # [!code --] - query: SELECT * FROM tasks WHERE project_id = subscription.parameter('project_id') AND project_id IN user_projects # [!code ++] -``` - -The client subscribes when the user opens a project and unsubscribes when they leave. This works only when the user does not need every record available offline at the same time. - -## Edge Cases and Gotchas - -### The Partition Key Must Exist on the Row - -A bucket's key must be a value that physically exists on a row of the table you sync. You cannot split a table into buckets by a column it does not have. This is why denormalizing the scope key onto child tables is the standard fix. If tasks only have `project_id`, you cannot key their buckets by `org_id` until you add `org_id` to the tasks table. - -### Subscription Parameters Choose Buckets, Not Re-Partition Them - -A subscription parameter lets the client choose which existing buckets to sync. It does not change how those buckets are defined. - -For a parameter to select a bucket, its value must match a value on the row being synced. For example, each task has a `project_id`, so you can use that column to group tasks into project buckets: - -```yaml -streams: - project_tasks: - query: SELECT * FROM tasks WHERE project_id = subscription.parameter('project_id') -``` - -Assets are different. An asset can belong to multiple projects, so the asset row does not have a single `project_id`. Passing a `project_id` as a subscription parameter therefore cannot make PowerSync group those assets by project. The asset row has no project ID to match against. - -If you want to sync assets by project, the asset row needs to contain a project reference first. For example, you could add a `project_ids` array column as described in [Reducing Bucket Count](#reducing-bucket-count). - -### Correlated Joins Behave Like Subqueries - -A correlated JOIN and an `IN (subquery)` compile to the same internal form. They create the same number of buckets. Rewriting one as the other does not reduce the count. - -### CTEs Cannot Reference Each Other - -Each CTE must be self-contained. A CTE cannot reference another CTE by name. If it does, the deploy fails. Inline the nested subquery instead. See [CTE limitations](/sync/streams/ctes#limitations). - -### Global Buckets Multiply Storage and Cost - -A stream with no filter creates one global bucket that every user syncs. Under `auto_subscribe: true`, every write to that table fans out to every user. This drives up synced data volume and cost. Scope global buckets carefully, and only mark truly shared reference data as global. - -### Bucket Storage Does Not Shrink When You Archive - -Buckets are append-only. Marking a row as archived does not remove it from bucket storage on its own. A row leaves storage only when it stops matching the data query, through a hard delete or a filter on the table's own column. Storage reclaims space during [compaction](/maintenance-ops/compacting-buckets). Filtering through a parent table does not shrink a child table's stored data. - -## Increasing the Limit - -Raise the limit only after you exhaust the reduction strategies above. - -Before you raise it, weigh the cost. Sync overhead scales roughly linearly with the number of buckets per user. Doubling the bucket count roughly doubles sync latency for a single operation. It also roughly doubles CPU and memory use on the server and the client. Many operations inside a single bucket scale much more efficiently than many buckets. The 1,000 default exists to encourage fewer, larger buckets and to protect the service from excessive counts. - -On PowerSync Cloud, you can request a higher limit on [Team and Enterprise](https://www.powersync.com/pricing) plans, up to 10,000. The limit applies per user, so your instance can still track far more buckets in total. - -For self-hosted deployments, set the limits under `api.parameters`: - -```yaml service.yaml -api: - parameters: - max_buckets_per_connection: 5000 - max_parameter_query_results: 5000 -``` - -Set both. Raising one without the other still leaves you capped by the limit you did not change. - -## Related Pages - -- [Bucket Count](/sync/streams/bucket-count) explains how buckets are counted and the two limits. -- [Writing Queries](/sync/streams/queries) covers the query syntax that determines your bucket count. -- [Common Table Expressions (CTEs)](/sync/streams/ctes) covers shared filtering logic. -- [Troubleshooting](/debugging/troubleshooting#psync_s2305-too-many-buckets-/-parameter-query-results) covers the `PSYNC_S2305` error. -- [Performance and Limits](/resources/performance-and-limits) lists the Service limits. + diff --git a/sync/advanced/schemas-and-connections.mdx b/sync/advanced/schemas-and-connections.mdx index 7c6633b3a..4a666c980 100644 --- a/sync/advanced/schemas-and-connections.mdx +++ b/sync/advanced/schemas-and-connections.mdx @@ -3,61 +3,8 @@ title: "Schemas and Connections" description: "Configure Postgres schema usage in Sync Streams/Rules queries, including wildcard schemas for schema-per-tenant setups, and connect to high-availability replicas." --- -## Schemas (Postgres) +{/* Wrapper page: the content is snippets/sync-shared/schemas-and-connections.mdx, which also renders at sync/rules/schemas-and-connections.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} -When no schema is specified, the Postgres `public` schema is used for every query. A different schema can be specified as a prefix: +import SchemasAndConnections from '/snippets/sync-shared/schemas-and-connections.mdx'; -```sql --- Note: the schema must be in double quotes -SELECT * FROM "other"."assets" -``` - -## Wildcard Schemas (Postgres) - - - Wildcard schemas require [Sync Streams](/sync/streams/overview) and PowerSync Service v1.24.0 or later. They are currently only supported for Postgres connections. - - -Use `%` as a wildcard in the schema name to match tables with the same name across multiple schemas. `"%"` matches every schema, and a prefix such as `"tenant_%"` matches every schema whose name starts with `tenant_`. The wildcard can only be the last character of the schema name. Postgres system schemas (`pg_*` and `information_schema`) are never matched. - -Combine a wildcard schema with the [`schema()` function](/sync/supported-sql#functions), which returns the schema each row was replicated from, to filter rows by schema. This supports schema-per-tenant databases (a single database with one identical schema per tenant): one stream covers every tenant schema, and each client syncs only its own tenant's data, resolved from a JWT claim. - -```yaml -config: - edition: 3 - -streams: - work_orders: - query: SELECT * FROM "%".work_orders WHERE work_orders.schema() = auth.parameter('tenant_schema') -``` - -In this example, rows are grouped into a bucket per schema, and each client syncs only the bucket matching the `tenant_schema` claim in its JWT. Rows from all matched schemas sync into a single client-side table, named after the table in the query (`work_orders` here). - - - Each matched table must be part of the [PowerSync publication](/configuration/source-db/setup#3-create-powersync-publication). Tables that are not in the publication are skipped. - - -## High Availability / Replicated Databases (Postgres) - -When the source Postgres database is replicated, for example with Amazon RDS Multi-AZ deployments, specify a single connection with multiple host endpoints. Each host endpoint will be tried in sequence, with the first available primary connection being used. - -For this, each endpoint must point to the same physical database, with the same replication slots. This is the case when block-level replication is used between the databases, but not when streaming physical or logical replication is used. In those cases, replication slots are unique on each host, and all data would be re-synced in a fail-over event. - -## Multiple Separate Database Connections (Planned) - - - This feature will be available in a future release. See this [item on our roadmap](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections). - - -In the future, it will be possible to configure PowerSync with multiple separate source database connections, where each connection is concurrently replicated. - -You should not add multiple connections to multiple replicas of the same database — this would cause data duplication. Only use this when the data on each connection does not overlap. - -It will be possible for each connection to be configured with a "tag", to distinguish these connections in Sync Rules. The same tag may be used for multiple connections (if the schema is the same in each). - -By default, queries will reference the "default" tag. To use a different connection or connections, assign a different tag, and specify it in the query as a schema prefix. In this case, the schema itself must also be specified. - -```sql --- Note the usage of quotes here -SELECT * FROM "secondconnection.public"."assets" -``` + diff --git a/sync/advanced/sharded-databases.mdx b/sync/advanced/sharded-databases.mdx index afde1abe3..6c5e37c45 100644 --- a/sync/advanced/sharded-databases.mdx +++ b/sync/advanced/sharded-databases.mdx @@ -3,45 +3,8 @@ title: "Sharded Databases" description: "Sync data from sharded Postgres databases with per-shard PowerSync connection configuration." --- -In the case of Postgres, PowerSync cannot replicate Postgres [foreign tables](https://www.postgresql.org/docs/current/ddl-foreign-data.html). +{/* Wrapper page: the content is snippets/sync-shared/sharded-databases.mdx, which also renders at sync/rules/sharded-databases.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} -However, PowerSync does have options available to support sharded databases in general. +import ShardedDatabases from '/snippets/sync-shared/sharded-databases.mdx'; - - When using MongoDB, MySQL, or SQL Server as the backend source database, PowerSync does not currently support connecting to sharded clusters. - - -The primary options are: - -1. Use a separate PowerSync Service instance per database. -2. Add a connection for each database in the same PowerSync Service instance ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release). - -Where feasible, using separate PowerSync Service instances would give better performance and give more control over how changes are rolled out, especially around Sync Rule reprocessing. - -Some specific scenarios: - -#### 1\. Different Tables on Different Databases - -This is common when separate "services" use separate databases, but multiple tables across those databases need to be synced to the same users. - -Use a single PowerSync Service instance, with a separate connection for each source database ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release). Use a unique [connection tag](/sync/advanced/schemas-and-connections) for each source database, allowing them to be distinguished in your [Sync Streams](/sync/streams/overview) or [Sync Rules](/sync/rules/overview). - -#### 2a. All Data for a Single Customer Is Contained in a Single Shard - -This is common when sharding per customer account / organization. - -In this case, use a separate PowerSync Service instance for each database. - -#### 2b. Most Customer Data Is in a Single Shard, but Some Data Is in a Shared Database - -If the amount of shared data is small, still use a separate PowerSync Service instance for each database, but also add the shared database connection to each PowerSync Service instance using a separate connection tag ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release). - -#### 2c. Data for a Single Customer Is Split Across Multiple Shards - -This is more complicated than the other cases listed above. Please [reach out to us](/resources/contact-us) if this is your architecture. - -#### 3\. Only Some Tables Are Sharded - -In some cases, most tables would be on a shared server, with only a few large tables being sharded. - -For this case, use a single PowerSync Service instance. Add each shard as a new connection on this instance ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release) — all with the same connection tag, so that the same [Sync Streams](/sync/streams/overview) or [Sync Rules](/sync/rules/overview) applies to each. + diff --git a/sync/advanced/storage-version-4.mdx b/sync/advanced/storage-version-4.mdx index 2b6b4b9ed..e2b71e31a 100644 --- a/sync/advanced/storage-version-4.mdx +++ b/sync/advanced/storage-version-4.mdx @@ -3,171 +3,8 @@ title: "Storage Version 4" description: "Opt in to storage version 4 for faster sync, incremental reprocessing of Sync Streams changes, and S3 object storage." --- -Storage version 4 is a new version of the format the PowerSync Service uses to store the data it syncs to clients. It is in [Beta](/resources/feature-status) as of PowerSync Service v1.26.0. +{/* Wrapper page: the content is snippets/sync-shared/storage-version-4.mdx, which also renders at sync/rules/storage-version-4.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} -Compared to version 2, it provides: +import StorageVersion4 from '/snippets/sync-shared/storage-version-4.mdx'; -- Faster sync and faster reprocessing after a deployment. -- [Incremental reprocessing](#incremental-reprocessing): a Sync Streams deployment reprocesses only the streams you added or changed. Clients no longer download all their data again after every deployment. -- [S3 object storage](#s3-object-storage): larger blocks of synced data move from the storage database to S3. This reduces load on the storage database when many clients sync at once or sync large amounts of data. - -## Availability - -Storage version 4 is compatible with all PowerSync Cloud instances, which already use MongoDB [bucket storage](/architecture/powersync-service#bucket-storage). Self-hosted instances must also use MongoDB bucket storage. Postgres bucket storage is not currently supported. - -The PowerSync Cloud and self-hosted columns below apply during the Beta only. Once storage version 4 is generally available, it will become the default for all supported instances. S3 object storage is then also enabled on all PowerSync Cloud instances. For self-hosted deployments, follow the [S3 setup instructions](#self-hosted-s3-setup). - -| | Source database | Sync Config | PowerSync Cloud (Beta) | Self-hosted (Beta) | -| --- | --- | --- | --- | --- | -| Storage version 4 | Any | Sync Streams or Sync Rules | Free plan: automatic. Other plans: [opt in](#opt-in). | [Opt in](#opt-in) | -| Incremental reprocessing | MongoDB | Sync Streams | Included with version 4 | Included with version 4 | -| S3 object storage | Any | Sync Streams or Sync Rules | Enabled per instance by PowerSync on request | [Set up S3 object storage](#self-hosted-s3-setup) | - -Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/rules/migrate-to-sync-streams). - -## Opt In - -Version 4 is not the default in PowerSync Service v1.26.0. Moving a Sync Config to version 4 runs like any other deployment: - -1. PowerSync reprocesses all data selected by your Sync Config in the background. The current version keeps serving clients, so there is no downtime. -2. When the new copy is ready, PowerSync switches to it. On PowerSync Cloud, this appears as a new deploy event in the PowerSync Dashboard. -3. Clients download their data again once, as after any deployment before version 4. On self-hosted deployments with many clients, scale out the API before the switch to absorb the re-sync. - -After this first deployment, later Sync Streams deployments use incremental reprocessing automatically when your instance meets its requirements. There is no separate setting. - -### PowerSync Cloud - -Free plan instances are upgraded automatically during the Beta. No action is needed. - -On other plans, add `storage_version: 4` to the `config` block of your Sync Config and deploy it: - -```yaml -config: - edition: 3 - storage_version: 4 - -streams: - todos: - query: SELECT * FROM todos WHERE owner_id = auth.user_id() -``` - -### Self-Hosted - - - Postgres bucket storage is not supported with version 4. - - -Add `storage_version: 4` to the `config` block of each Sync Config as shown above, then deploy or redeploy it to use version 4. - -To move a Sync Config back to version 2, set `storage_version: 2` and deploy again. This is another full reprocess. - -To also enable S3 object storage, follow the [self-hosted S3 setup instructions](#self-hosted-s3-setup) to prepare a bucket and configure the Service. - -## Incremental Reprocessing - -Incremental reprocessing is active when you use a MongoDB source database, Sync Streams, and storage version 4. - - - Self-hosted instances with Postgres bucket storage are not supported. - - -Without it, every deployment reads all data selected by the Sync Config from your source database and prepares a complete new copy. Clients then download all their data again, even if only one stream changed. - -With incremental reprocessing, PowerSync compares the new Sync Config with the current one and reprocesses only the streams you added or changed. Unchanged streams keep their data on the PowerSync Service and on clients. Deployments finish faster, your source database does less work, and clients download only the data for affected streams they subscribe to. - -- Adding a stream reads only the data that stream selects. -- Removing a stream requires no new source reads. PowerSync cleans up stored definitions when no active Sync Config still uses them. -- Renaming a stream counts as removing it and adding a new one, so its data is rebuilt. -- Changing a stream's queries may reprocess affected definitions. Changes that only affect how request parameters select existing buckets do not require reprocessing. - -For example, changing `SELECT * FROM projects WHERE user_id = auth.user_id()` to `SELECT * FROM projects WHERE user_id = auth.jwt() ->> 'owner'` reuses the existing bucket data. The data is still grouped by `user_id`; only the JWT field used to select buckets changes. - -The time saved depends on how your data is split across streams. If one stream selects most of your data, changing that stream still takes about as long as a full reprocess. - -PowerSync favors correctness over reuse. When it cannot confirm that a change leaves a stream's data unchanged, it rebuilds that stream. A deployment that reprocesses more than you expect is not an error. - -Event definitions for [Custom Write Checkpoints](/handling-writes/custom-write-checkpoints) follow the same rules. Unchanged events keep their data, and new or changed events are read again. - -### When PowerSync Reprocesses Everything - -Some changes start a full reprocess, after which clients download all their data again: - -- The first deployment on storage version 4. -- Changes to the `config` block of the Sync Config, such as `edition`, compatibility fixes, or `storage_version`. -- The **Defragment** action in the PowerSync Dashboard, which exists to rebuild all data. See [Defragmenting](/maintenance-ops/compacting-buckets#defragmenting). -- Replication failures, for example when PowerSync loses its position in the MongoDB change stream and has to start over. - -### Sync Config Versions and Replication Streams - -Each deployment has a Sync Config version. With incremental reprocessing, multiple versions can share a replication stream, the replication process and stored state. A full reprocess creates a new replication stream. - -See the [Log Reference](/debugging/log-reference#message-prefixes) for how to identify these versions and streams in your logs. - -For implementation details, see the [storage design](https://github.com/powersync-ja/powersync-service/blob/main/docs/storage/storage-v3.md). The document describes the design introduced in version 3 and carried into version 4. - -### Checking What a Deployment Reprocessed - -If a deployment takes longer or reprocesses more than you expect, see [Checking What a Deployment Reprocessed](/debugging/log-reference#checking-what-a-deployment-reprocessed) in the Log Reference for what to look for in your logs. - -## S3 Object Storage - -Your instance keeps the data it syncs to clients in its bucket storage database, alongside everything else it needs to run. With S3 object storage, larger blocks of that data move to Amazon S3 or an S3-compatible object store, and the PowerSync Service syncs them to clients directly from there. Smaller blocks, and the metadata that locates each block, stay in MongoDB. - -Reading larger blocks from S3 reduces the data MongoDB must read and transfer during sync. When those reads limit performance, offloading them can speed up initial sync and let an instance serve more concurrent clients. The benefit is most noticeable when clients sync large amounts of data or many clients connect at once. The PowerSync Service still handles every client connection, so its CPU and memory capacity also limit concurrency. - -For self-hosted instances, offloading bucket data to S3 can reduce storage and data transfer costs. Compare the reduction in database costs with the object store's storage, request, and data transfer charges for your workload. - -Clients connect only to the PowerSync Service and never to the object store, so no client changes are needed. If the object store becomes unreachable, sync is interrupted until it recovers. Clients reconnect and resume automatically. - -S3 object storage requires storage version 4 and works with Sync Streams and legacy Sync Rules. It is compatible with all PowerSync Cloud instances. - - - S3 object storage holds PowerSync's internal sync data. To store files uploaded by your app, use [Attachments](/client-sdks/advanced/attachments). - - -### PowerSync Cloud - -During the Beta, PowerSync enables S3 object storage per instance. [Contact us](/resources/contact-us) if you want it on your instance before we enable it for all instances. - -### Self-Hosted S3 Setup - - - Self-hosted instances with Postgres bucket storage are not supported. - - - - - Create a bucket. Use the same region as the PowerSync Service where possible, to keep latency low and avoid cross-region data transfer charges. Use a dedicated bucket, or a unique `prefix` per PowerSync instance, so that instances never read or delete each other's files. Give the PowerSync Service permission to list the bucket and to read, write, and delete objects under the prefix. - - Leave object versioning off, or suspend it if the bucket already has it, and leave Object Lock off. PowerSync deletes files itself once they are no longer needed, so versioning keeps charging for old versions and locked objects cannot be cleaned up. Do not add an expiration lifecycle rule: an expired object may still be referenced by MongoDB, which breaks sync for that data. - - - Add `object_storage` to the `storage` section of `service.yaml`: - - ```yaml service.yaml - storage: - type: mongodb - uri: !env PS_MONGO_STORAGE_URI - object_storage: - type: s3 - bucket: powersync-bucket-data - region: us-east-1 - prefix: production - ``` - - Without `access_key_id` and `secret_access_key`, PowerSync uses the AWS credentials available to the process, such as an IAM role. For S3-compatible providers such as MinIO or Cloudflare R2, also set `endpoint`, and set `force_path_style: true` if the provider requires path-style requests. - - Restart or redeploy the PowerSync Service to load the updated `service.yaml`. If you run replication, API, and compacting in separate containers or jobs, apply the same object storage configuration to each. - - - Deploy your Sync Configs on storage version 4 as described in [Opt In](#opt-in). Sync Configs on version 2 keep all data in MongoDB, even when `object_storage` is configured. - - Once replication reaches a healthy checkpoint, confirm that objects appear under the prefix, run a test initial sync, and run `compact` once to surface permission errors early. - - - -After enabling S3 object storage, you can raise [`max_concurrent_connections`](/configuration/powersync-service/self-hosted-instances#param-max-concurrent-connections) from its default of 200 per API process. With storage version 4 and S3 object storage, each API process can handle up to 1,000 concurrent client connections. Performance degrades if a large share of those clients run an initial sync at the same time, so scale out the API before a deployment that makes all clients download their data again. More concurrent connections also increase CPU and memory usage. - -The [S3 object storage configuration reference](/configuration/powersync-service/self-hosted-instances#param-object-storage) lists all supported settings, including timeouts, request concurrency, and the size threshold below which blocks stay in MongoDB. - -Keep the scheduled [compact](/maintenance-ops/compacting-buckets) job running. It removes files that are no longer needed. The `teardown` command deletes PowerSync's files under the prefix before it drops the storage database. The `powersync_object_storage_size_bytes` [metric](/maintenance-ops/self-hosting/monitoring) reports how much object storage PowerSync uses. + diff --git a/sync/advanced/sync-data-by-time.mdx b/sync/advanced/sync-data-by-time.mdx index ea503e502..afd0b875c 100644 --- a/sync/advanced/sync-data-by-time.mdx +++ b/sync/advanced/sync-data-by-time.mdx @@ -1,23 +1,18 @@ --- title: "Sync Data by Time with Sync Streams" -description: "Filter and sync data based on time ranges using Sync Streams/Sync Rules, with patterns for recent-only and sliding-window queries." +description: "Filter and sync data based on time ranges using Sync Streams, with patterns for recent-only and sliding-window queries." sidebarTitle: "Sync Data by Time" --- +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/sync-data-by-time.mdx. That page is deprecated and frozen, but when you fix an error here, check whether it needs the same fix. Do not mention Sync Rules on this page. */} + A common need in offline-first apps is syncing data based on time, for example, only syncing issues updated in the last 7 days instead of the entire dataset. You might expect to write something like: ```yaml -# Sync Streams streams: issues_after_start_date: - query: SELECT * FROM issues WHERE updated_at > subscription.parameter('start_at') - -# Sync Rules -bucket_definitions: - issues_after_start_date: - parameters: SELECT request.parameters() ->> 'start_at' as start_at - data: SELECT * FROM issues WHERE updated_at > bucket.start_date + query: SELECT * FROM issues WHERE updated_at > subscription.parameter('start_at') ``` However, this won't work. Here's why. @@ -28,8 +23,6 @@ PowerSync pre-computes and caches which rows belong to which parameters to enabl Additionally, time-based functions like `now()` aren't allowed in parameter expressions because the result changes depending on when the query runs, making pre-computation impossible. -These constraints apply to both Sync Streams and legacy Sync Rules. - This guide covers a few practical workarounds. We are working on a more elegant solution for this problem. When ready, this guide will be updated accordingly. @@ -51,73 +44,38 @@ Update it periodically using a cron job (e.g., with `pg_cron`): UPDATE issues SET updated_this_week = (updated_at > now() - interval '7 days'); ``` - - - ```yaml - config: - edition: 3 - streams: - recent_issues: - auto_subscribe: true - query: SELECT * FROM issues WHERE updated_this_week = true - ``` - - For multiple time ranges, define a stream per range and let the client subscribe to the one it needs: - - ```yaml - config: - edition: 3 - streams: - issues_1week: - query: SELECT * FROM issues WHERE updated_this_week = true - - issues_1month: - query: SELECT * FROM issues WHERE updated_this_month = true - ``` - - The client subscribes to the desired range: - - ```javascript - // Subscribe to one-week range - await db.syncStream('issues_1week').subscribe(); - // Or subscribe to one-month range - await db.syncStream('issues_1month').subscribe(); - ``` - - - ```yaml - bucket_definitions: - recent_issues: - data: - - SELECT * FROM issues WHERE updated_this_week = true - ``` - - For multiple time ranges, add multiple bucket definitions and let the client choose which bucket to sync: - - ```yaml - bucket_definitions: - issues_1week: - parameters: SELECT WHERE request.parameters() ->> 'range' = '1week' - data: - - SELECT * FROM issues WHERE updated_this_week = true - - issues_1month: - parameters: SELECT WHERE request.parameters() ->> 'range' = '1month' - data: - - SELECT * FROM issues WHERE updated_this_month = true - ``` - - The client passes the desired range as a client parameter: - - ```javascript - await db.connect(connector, { - params: { - range: '1week', - }, - }) - ``` - - +Then filter on the column in a stream query: + +```yaml +config: + edition: 3 +streams: + recent_issues: + auto_subscribe: true + query: SELECT * FROM issues WHERE updated_this_week = true +``` + +For multiple time ranges, define a stream per range and let the client subscribe to the one it needs: + +```yaml +config: + edition: 3 +streams: + issues_1week: + query: SELECT * FROM issues WHERE updated_this_week = true + + issues_1month: + query: SELECT * FROM issues WHERE updated_this_month = true +``` + +The client subscribes to the desired range: + +```javascript +// Subscribe to one-week range +await db.syncStream('issues_1week').subscribe(); +// Or subscribe to one-month range +await db.syncStream('issues_1month').subscribe(); +``` This approach works well when you have a small, fixed set of time ranges. However, it requires schema changes and a scheduled job to keep the columns updated. @@ -133,50 +91,25 @@ Instead of pre-defined ranges, create a bucket for each date and let the client Use `substring` to extract the date portion from a timestamp and match it with `=`: -For a complete working example, see the [PowerSync + Supabase: Time-Based Sync demo](https://github.com/powersync-ja/powersync-js/tree/main/demos/react-supabase-time-based-sync). - - - - ```yaml - config: - edition: 3 - streams: - issues_by_date: - query: SELECT * FROM issues WHERE substring(updated_at, 1, 10) = subscription.parameter('date') - ``` - - The client subscribes once per date it wants to sync: - - ```javascript - await db.syncStream('issues_by_date', { date: '2026-01-07' }).subscribe(); - await db.syncStream('issues_by_date', { date: '2026-01-08' }).subscribe(); - await db.syncStream('issues_by_date', { date: '2026-01-09' }).subscribe(); - ``` - - Each subscription can be managed independently — you can subscribe and unsubscribe to individual dates without affecting others. - - - ```yaml - bucket_definitions: - issues_by_update_at: - parameters: SELECT value as date FROM json_each(request.parameters() ->> 'dates') - data: - - SELECT * FROM issues WHERE substring(updated_at, 1, 10) = bucket.date - ``` - - The client passes the dates it wants as client parameters: - - ```javascript - await db.connect(connector, { - params: { - dates: ["2026-01-07", "2026-01-08", "2026-01-09"], - }, - }) - ``` - - - -This gives users full control over which dates to sync, with no schema changes or scheduled jobs required. +```yaml +config: + edition: 3 +streams: + issues_by_date: + query: SELECT * FROM issues WHERE substring(updated_at, 1, 10) = subscription.parameter('date') +``` + +The client subscribes once per date it wants to sync: + +```javascript +await db.syncStream('issues_by_date', { date: '2026-01-07' }).subscribe(); +await db.syncStream('issues_by_date', { date: '2026-01-08' }).subscribe(); +await db.syncStream('issues_by_date', { date: '2026-01-09' }).subscribe(); +``` + +Each subscription can be managed independently — you can subscribe and unsubscribe to individual dates without affecting others. + +This gives users full control over which dates to sync, with no schema changes or scheduled jobs required. For a complete working example, see the [PowerSync + Supabase: Time-Based Sync demo](https://github.com/powersync-ja/powersync-js/tree/main/demos/react-supabase-time-based-sync). The trade-off is granularity. In this example we're using daily buckets. If you need finer precision (hourly), syncing a large range means many buckets, which can degrade sync performance and approach [PowerSync's limit of 1,000 buckets per user](/resources/performance-and-limits#limits). If you use larger buckets (monthly), you lose the ability to filter accurately. @@ -190,69 +123,33 @@ You have to pick a granularity and stick with it. If that's a problem—say, you Combine multiple granularities in a single definition. This lets you use larger buckets (days) for older data and smaller buckets (hours, minutes) for recent data. - - - ```yaml - config: - edition: 3 - streams: - issues_by_partition: - queries: - # By day (e.g., "2026-01-07") - - SELECT * FROM issues WHERE substring(updated_at, 1, 10) = subscription.parameter('partition') - # By hour (e.g., "2026-01-07T14") - - SELECT * FROM issues WHERE substring(updated_at, 1, 13) = subscription.parameter('partition') - # By 10 minutes (e.g., "2026-01-07T14:3") - - SELECT * FROM issues WHERE substring(updated_at, 1, 15) = subscription.parameter('partition') - ``` - - The client subscribes once per partition, mixing granularities as needed: - - ```javascript - await db.syncStream('issues_by_partition', { partition: '2026-01-05' }).subscribe(); - await db.syncStream('issues_by_partition', { partition: '2026-01-06' }).subscribe(); - await db.syncStream('issues_by_partition', { partition: '2026-01-07T10' }).subscribe(); - await db.syncStream('issues_by_partition', { partition: '2026-01-07T11' }).subscribe(); - await db.syncStream('issues_by_partition', { partition: '2026-01-07T12:0' }).subscribe(); - await db.syncStream('issues_by_partition', { partition: '2026-01-07T12:1' }).subscribe(); - await db.syncStream('issues_by_partition', { partition: '2026-01-07T12:2' }).subscribe(); - ``` - - Each query naturally acts as a filter based on the length of the partition value — a day-format partition only matches the day query, an hour-format partition only matches the hour query, and so on. - - - ```yaml - bucket_definitions: - issues_by_time: - parameters: SELECT value as partition FROM json_each(request.parameters() ->> 'partitions') - data: - # By day (e.g., "2026-01-07") - - SELECT * FROM issues WHERE substring(updated_at, 1, 10) = bucket.partition - # By hour (e.g., "2026-01-07T14") - - SELECT * FROM issues WHERE substring(updated_at, 1, 13) = bucket.partition - # By 10 minutes (e.g., "2026-01-07T14:3") - - SELECT * FROM issues WHERE substring(updated_at, 1, 15) = bucket.partition - ``` - - The client then mixes granularities as needed: - - ```javascript - await db.connect(connector, { - params: { - partitions: [ - "2026-01-05", - "2026-01-06", - "2026-01-07T10", - "2026-01-07T11", - "2026-01-07T12:0", - "2026-01-07T12:1", - "2026-01-07T12:2" - ] - }, - }) - ``` - - +```yaml +config: + edition: 3 +streams: + issues_by_partition: + queries: + # By day (e.g., "2026-01-07") + - SELECT * FROM issues WHERE substring(updated_at, 1, 10) = subscription.parameter('partition') + # By hour (e.g., "2026-01-07T14") + - SELECT * FROM issues WHERE substring(updated_at, 1, 13) = subscription.parameter('partition') + # By 10 minutes (e.g., "2026-01-07T14:3") + - SELECT * FROM issues WHERE substring(updated_at, 1, 15) = subscription.parameter('partition') +``` + +The client subscribes once per partition, mixing granularities as needed: + +```javascript +await db.syncStream('issues_by_partition', { partition: '2026-01-05' }).subscribe(); +await db.syncStream('issues_by_partition', { partition: '2026-01-06' }).subscribe(); +await db.syncStream('issues_by_partition', { partition: '2026-01-07T10' }).subscribe(); +await db.syncStream('issues_by_partition', { partition: '2026-01-07T11' }).subscribe(); +await db.syncStream('issues_by_partition', { partition: '2026-01-07T12:0' }).subscribe(); +await db.syncStream('issues_by_partition', { partition: '2026-01-07T12:1' }).subscribe(); +await db.syncStream('issues_by_partition', { partition: '2026-01-07T12:2' }).subscribe(); +``` + +Each query naturally acts as a filter based on the length of the partition value — a day-format partition only matches the day query, an hour-format partition only matches the hour query, and so on. This syncs January 5–6 by day, the morning of January 7 by hour, and the last 30 minutes in 10-minute chunks, without creating hundreds of buckets. diff --git a/sync/grammar/sync-rules/index.mdx b/sync/grammar/sync-rules/index.mdx index 4f7c46d28..c96e0e1bc 100644 --- a/sync/grammar/sync-rules/index.mdx +++ b/sync/grammar/sync-rules/index.mdx @@ -3,9 +3,9 @@ title: "Grammar Reference (Sync Rules)" description: "Railroad diagram reference for the SQL grammar supported in legacy Sync Rules queries." --- -This page is a formal grammar reference for Sync Rules: it shows the syntax accepted for parameter queries and data queries using railroad diagrams. This page complements the [Supported SQL](/sync/supported-sql) guide, which explains in prose what you can write, with examples and restrictions. +This page is a formal grammar reference for Sync Rules: it shows the syntax accepted for parameter queries and data queries using railroad diagrams. This page complements the [Supported SQL](/sync/rules/supported-sql) guide, which explains in prose what you can write, with examples and restrictions. -**When to use this page:** If you need to check whether a construct is valid, see how parameter vs data query syntax differs, or you're used to grammar specs, use the diagrams and the "Used by" / "References" links to navigate. For most users just getting started, see [Supported SQL](/sync/supported-sql) and the [Sync Rules](/sync/rules/overview) docs. +**When to use this page:** If you need to check whether a construct is valid, see how parameter vs data query syntax differs, or you're used to grammar specs, use the diagrams and the "Used by" / "References" links to navigate. For most users just getting started, see [Supported SQL](/sync/rules/supported-sql) and the [Sync Rules](/sync/rules/overview) docs. ## ParameterQuery diff --git a/sync/rules/case-sensitivity.mdx b/sync/rules/case-sensitivity.mdx new file mode 100644 index 000000000..c4b61d188 --- /dev/null +++ b/sync/rules/case-sensitivity.mdx @@ -0,0 +1,15 @@ +--- +title: "Case Sensitivity" +description: "Handle case-sensitive table and column names in PowerSync Sync Streams/Rules, with best practices for lowercase identifiers and quoting strategies." +noindex: true +--- + +{/* Wrapper page: the content is snippets/sync-shared/case-sensitivity.mdx, which also renders at sync/advanced/case-sensitivity.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} + +import CaseSensitivity from '/snippets/sync-shared/case-sensitivity.mdx'; + + +Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. + + + diff --git a/sync/rules/client-id.mdx b/sync/rules/client-id.mdx new file mode 100644 index 000000000..07d440ebb --- /dev/null +++ b/sync/rules/client-id.mdx @@ -0,0 +1,15 @@ +--- +title: "Client ID" +description: "Understand PowerSync's requirement for a single text-type primary key column called id." +noindex: true +--- + +{/* Wrapper page: the content is snippets/sync-shared/client-id.mdx, which also renders at sync/advanced/client-id.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} + +import ClientId from '/snippets/sync-shared/client-id.mdx'; + + +Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. + + + diff --git a/sync/rules/compatibility.mdx b/sync/rules/compatibility.mdx new file mode 100644 index 000000000..b9fa6991e --- /dev/null +++ b/sync/rules/compatibility.mdx @@ -0,0 +1,15 @@ +--- +title: "Compatibility" +description: "Configure compatibility editions and bucket storage format version in PowerSync's Sync Config." +noindex: true +--- + +{/* Wrapper page: the content is snippets/sync-shared/compatibility.mdx, which also renders at sync/advanced/compatibility.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} + +import Compatibility from '/snippets/sync-shared/compatibility.mdx'; + + +Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. Configuration examples use Sync Streams syntax. + + + diff --git a/sync/rules/data-queries.mdx b/sync/rules/data-queries.mdx index a9892bbe9..b654435fb 100644 --- a/sync/rules/data-queries.mdx +++ b/sync/rules/data-queries.mdx @@ -19,7 +19,7 @@ Data Queries are used to group data into buckets, so each Data Query must use ev ## Supported SQL -The supported SQL in Data Queries is based on a small subset of the SQL standard syntax. Not all SQL constructs are supported. See [Supported SQL](/sync/supported-sql) for full details. +The supported SQL in Data Queries is based on a small subset of the SQL standard syntax. Not all SQL constructs are supported. See [Supported SQL](/sync/rules/supported-sql) for full details. ## Examples diff --git a/sync/rules/migrate-to-sync-streams.mdx b/sync/rules/migrate-to-sync-streams.mdx index 69a2c4f11..e4a563424 100644 --- a/sync/rules/migrate-to-sync-streams.mdx +++ b/sync/rules/migrate-to-sync-streams.mdx @@ -18,7 +18,9 @@ If your Sync Config has a `bucket_definitions:` section, you use Sync Rules and ## Why Migrate? -Beyond matching Sync Rules, Sync Streams add: +{/* TODO: Link to the Sync Rules deprecation announcement once it is published. */} + +Sync Rules are deprecated, and PowerSync is phasing them out in favor of Sync Streams. Beyond matching Sync Rules, Sync Streams add: - **More expressive queries:** Stream queries support JOINs, [CTEs](/sync/streams/ctes), subqueries, and [multiple queries per stream](/sync/streams/queries#multiple-queries-per-stream), with syntax closer to plain SQL. You write one query instead of separate `parameters:` and `data:` blocks. - **On-demand syncing:** Define a stream once, then subscribe from your app one or more times with different parameters. Each subscription has its own lifecycle, so two screens or browser tabs can subscribe to the same stream independently. With Sync Rules, Client Parameters approximate this. You have to aggregate the parameter values yourself across screens and tabs, and remove them when they are no longer needed. diff --git a/sync/rules/multiple-client-versions.mdx b/sync/rules/multiple-client-versions.mdx new file mode 100644 index 000000000..0db532955 --- /dev/null +++ b/sync/rules/multiple-client-versions.mdx @@ -0,0 +1,39 @@ +--- +title: "Multiple Client Versions with Sync Rules" +sidebarTitle: "Multiple Client Versions" +description: "Handle multiple client app versions that require different output schemas from legacy Sync Rules." +--- + +{/* Split page: the Sync Streams version of this page is sync/advanced/multiple-client-versions.mdx. This page is deprecated and frozen: no new features or examples. When you fix an error here, check whether the Sync Streams page needs the same fix. */} + + +Sync Rules are deprecated. This page is not updated with new features. For the Sync Streams version, see [Multiple Client Versions](/sync/advanced/multiple-client-versions). + + +When schema changes are additive, old clients ignore the new tables and columns, and no special handling is required. More drastic changes, such as renaming tables or changing a table's structure, can break older app versions that are still in use. In these cases, define separate versions of the affected bucket definitions so that each client version receives the tables and columns it expects. + +## Versioning with Client Parameters + +Clients pass their version to the PowerSync Service as a [client parameter](/sync/rules/client-parameters), and parameter queries filter on it so that each client only receives the buckets for its version. + +For example, suppose a new app version changes the structure of the `assets` table in its [client-side schema](/intro/setup-guide#define-your-client-side-schema), defining it as `assets_v2`, while older app versions still define `assets`. Define a second bucket definition alongside the existing one, using an alias to map the source `assets` table to the new client-side name, and filter each on a `schema_version` client parameter: + +```yaml +# Client passes in: "params": {"schema_version": } +bucket_definitions: + user_assets: + parameters: SELECT request.user_id() AS user_id + WHERE request.parameters() ->> 'schema_version' = '1' + data: + - SELECT * FROM assets WHERE user_id = bucket.user_id + + user_assets_v2: + parameters: SELECT request.user_id() AS user_id + WHERE request.parameters() ->> 'schema_version' = '2' + data: + - SELECT * FROM assets AS assets_v2 WHERE user_id = bucket.user_id +``` + + + Handle queries based on parameters set by the client with care. The client can send any value for these parameters, so it's not a good place to do authorization. If the parameter must be authenticated, use parameters from the JWT instead. + diff --git a/sync/rules/organize-data-into-buckets.mdx b/sync/rules/organize-data-into-buckets.mdx index 130b0b560..8888b9757 100644 --- a/sync/rules/organize-data-into-buckets.mdx +++ b/sync/rules/organize-data-into-buckets.mdx @@ -44,7 +44,7 @@ bucket_definitions: - The supported SQL in _Parameter Queries_ and _Data Queries_ is based on a small subset of the SQL standard syntax. Not all SQL constructs are supported. See [Supported SQL](/sync/supported-sql). + The supported SQL in _Parameter Queries_ and _Data Queries_ is based on a small subset of the SQL standard syntax. Not all SQL constructs are supported. See [Supported SQL](/sync/rules/supported-sql). diff --git a/sync/rules/overview.mdx b/sync/rules/overview.mdx index 17921d0e2..78ae6a770 100644 --- a/sync/rules/overview.mdx +++ b/sync/rules/overview.mdx @@ -4,15 +4,17 @@ sidebarTitle: "Overview & Key Concepts" description: "Understand legacy Sync Rules for controlling which data syncs to each client." --- -Sync Rules are PowerSync's original system for partial sync, using YAML bucket definitions. They remain supported for existing projects but are considered legacy. +Sync Rules are PowerSync's original system for partial sync, using YAML bucket definitions. They are deprecated. Existing instances keep working and stay supported while you migrate to Sync Streams. - -**Sync Streams Recommended** +{/* TODO: Link to the Sync Rules deprecation announcement on releases.powersync.com once it is published. */} -[Sync Streams](/sync/streams/overview) are the recommended approach to partial sync for both new and existing projects. They support everything Sync Rules do, plus more expressive queries (including JOIN support), on-demand syncing, and a simpler developer experience (e.g. React hooks that manage subscriptions automatically). + +**Sync Rules are deprecated** -You can migrate in a few clicks. Click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to generate a draft from your current config. See [Migrate to Sync Streams](/sync/rules/migrate-to-sync-streams) for details. - +PowerSync is phasing out Sync Rules in favor of [Sync Streams](/sync/streams/overview), which support everything Sync Rules do and add on-demand syncing, JOINs, CTEs, and subqueries. Nothing changes for your instance today: Sync Rules keep working and stay supported while you migrate. New sync config features are added to Sync Streams only. + +To migrate, click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to convert your current config. Migrating does not change what your app syncs. See [Migrate to Sync Streams](/sync/rules/migrate-to-sync-streams). + Sync Rules are defined in a YAML file. For PowerSync Cloud, they are edited and deployed to a specific PowerSync instance in the [PowerSync Dashboard](/tools/powersync-dashboard#project-&-instance-level). For self-hosting setups, they are defined as part of your [instance configuration](/configuration/powersync-service/self-hosted-instances). @@ -52,7 +54,7 @@ The following values can be selected in Parameter Queries: - **Client Parameters** (see below) - **Values From a Table/Collection** (see below) -See [Parameter Queries](/sync/rules/parameter-queries) for more details and examples. Also see [Supported SQL](/sync/supported-sql) for limitations. +See [Parameter Queries](/sync/rules/parameter-queries) for more details and examples. Also see [Supported SQL](/sync/rules/supported-sql) for limitations. ### Authentication Parameters @@ -72,11 +74,11 @@ Clients can specify **Client Parameters** when connecting to PowerSync (i.e. whe ```yaml Example of selecting a Client Parameter in a Parameter Query parameters: SELECT (request.parameters() ->> 'current_project') as current_project ``` -The `->>` operator in the above example extracts a value from a string containing JSON (which is the format provided by ``request.parameters()``). See [Operators and Functions](/sync/supported-sql#operators) +The `->>` operator in the above example extracts a value from a string containing JSON (which is the format provided by ``request.parameters()``). See [Operators and Functions](/sync/rules/supported-sql#operators) A client can pass any value for a Client Parameter. Hence, Client Parameters should always be treated with care, and should [not be used](/sync/rules/client-parameters#security-consideration) for access control purposes. -That being said, Client Parameters can be useful for use cases such as syncing different buckets based on state in the client app, for example only syncing data for the project currently selected, or syncing different buckets based on the client version ([see here](/sync/advanced/multiple-client-versions)). +That being said, Client Parameters can be useful for use cases such as syncing different buckets based on state in the client app, for example only syncing data for the project currently selected, or syncing different buckets based on the client version ([see here](/sync/rules/multiple-client-versions)). See [Client Parameters](/sync/rules/client-parameters) and [Parameter Queries](/sync/rules/parameter-queries) for more details and examples. @@ -103,7 +105,7 @@ data: - SELECT * FROM lists WHERE owner_id = bucket.user_id ``` -See [Data Queries](/sync/rules/data-queries) for more details and examples. Also see [Supported SQL](/sync/supported-sql) for limitations. +See [Data Queries](/sync/rules/data-queries) for more details and examples. Also see [Supported SQL](/sync/rules/supported-sql) for limitations. ### Global Buckets @@ -127,6 +129,3 @@ This architecture is key to the scalability and performance of PowerSync. See th Designing your Sync Rules is basically about _organizing data into buckets_, and creating the bucket definitions accordingly. See [Organize Data Into Buckets](/sync/rules/organize-data-into-buckets). - - - diff --git a/sync/rules/parameter-queries.mdx b/sync/rules/parameter-queries.mdx index 47372f0e6..c8c3972a9 100644 --- a/sync/rules/parameter-queries.mdx +++ b/sync/rules/parameter-queries.mdx @@ -27,7 +27,7 @@ The following functions allow you to select Authentication Parameters in your Pa | `request.user_id()` | Returns the JWT subject (`sub`). Same as `request.jwt() ->> 'sub'` (see below) | | `request.jwt()` | Returns the entire (signed) JWT payload as a JSON string. If there are other _claims_ in your JWT (in addition to the user ID), you can select them from this JSON string. | -Since `request.jwt()` is a string containing JSON, use the `->>` [operator](/sync/supported-sql#operators) to select values from it: +Since `request.jwt()` is a string containing JSON, use the `->>` [operator](/sync/rules/supported-sql#operators) to select values from it: ```sql request.jwt() ->> 'sub' -- the 'subject' of the JWT - same as `request.user_id() @@ -119,7 +119,7 @@ bucket_definitions: ## Supported SQL -The supported SQL in Parameter Queries is based on a small subset of the SQL standard syntax. Not all SQL constructs are supported. See [Supported SQL](/sync/supported-sql) for full details. +The supported SQL in Parameter Queries is based on a small subset of the SQL standard syntax. Not all SQL constructs are supported. See [Supported SQL](/sync/rules/supported-sql) for full details. ## Usage Examples @@ -203,7 +203,7 @@ For more advanced details on many-to-many relationships and join tables, see [th ### Expanding JSON Array Into Multiple Parameters -Using the `json_each()` [function](/sync/supported-sql#functions) and `->` [operator](/sync/supported-sql#operators), we can expand a parameter that is a JSON array into multiple rows, thereby filtering by multiple parameter values: +Using the `json_each()` [function](/sync/rules/supported-sql#functions) and `->` [operator](/sync/rules/supported-sql#operators), we can expand a parameter that is a JSON array into multiple rows, thereby filtering by multiple parameter values: ```yaml bucket_definitions: diff --git a/sync/rules/partitioned-tables.mdx b/sync/rules/partitioned-tables.mdx new file mode 100644 index 000000000..729f31b4d --- /dev/null +++ b/sync/rules/partitioned-tables.mdx @@ -0,0 +1,37 @@ +--- +title: "Partitioned Tables (Postgres) with Sync Rules" +sidebarTitle: "Partitioned Tables (Postgres)" +description: "Sync data from Postgres partitioned tables in legacy Sync Rules using wildcard table name matching." +--- + +{/* Split page: the Sync Streams version of this page is sync/advanced/partitioned-tables.mdx. This page is deprecated and frozen: no new features or examples. When you fix an error here, check whether the Sync Streams page needs the same fix. */} + + +Sync Rules are deprecated. This page is not updated with new features. For the Sync Streams version, see [Partitioned Tables (Postgres)](/sync/advanced/partitioned-tables). + + +For partitioned tables in Postgres, each individual partition is replicated and processed using [Sync Rules](/sync/rules/overview). + +To use the same queries and same output table name for each partition, use `%` for wildcard suffix matching of the table name: + +```yaml +bucket_definitions: + by_user: + # Use wildcard in a parameter query + parameters: SELECT id AS user_id FROM "users_%" + data: + # Use wildcard in a data query + - SELECT * FROM "todos_%" AS todos WHERE user_id = bucket.user_id +``` + +The wildcard character can only be used as the last character in the table name. To match tables across multiple schemas instead, see [Wildcard Schemas](/sync/advanced/schemas-and-connections#wildcard-schemas-postgres). + +When using wildcard table names, the original table suffix is available as the special `_table_suffix` column, which you can use to filter rows: + +```sql +SELECT * FROM "todos_%" AS todos WHERE _table_suffix != 'archived' +``` + +When no table alias is provided, the original table name is preserved. + +`publish_via_partition_root` on the publication is not supported. The individual partitions must be published. diff --git a/sync/rules/prioritized-sync.mdx b/sync/rules/prioritized-sync.mdx new file mode 100644 index 000000000..e0f397f2e --- /dev/null +++ b/sync/rules/prioritized-sync.mdx @@ -0,0 +1,126 @@ +--- +title: "Prioritized Sync with Sync Rules" +sidebarTitle: "Prioritized Sync" +description: "Assign sync priorities to bucket definitions in legacy Sync Rules so that important data syncs before the rest." +--- + +{/* Split page: the Sync Streams version of this page is sync/streams/prioritized-sync.mdx. This page is deprecated and frozen: no new features or examples. When you fix an error here, check whether the Sync Streams page needs the same fix. */} + + +Sync Rules are deprecated. This page is not updated with new features. For the Sync Streams version, see [Prioritized Sync](/sync/streams/prioritized-sync). + + +## Overview + +PowerSync supports defining sync priorities, which allows you to control the sync order for different data. This is useful when certain data should be available sooner than others. + +In Sync Rules, you assign priorities to bucket definitions. The priority determines when data in that bucket syncs relative to other buckets. + + +**Availability** + +This feature was introduced in version **1.7.1** of the PowerSync Service, and in the following SDK versions: +- [Flutter v1.12.0](/client-sdks/reference/flutter) +- [React Native v1.18.1](/client-sdks/reference/react-native-and-expo) +- [JavaScript Web v1.14.2](/client-sdks/reference/javascript-web) +- [Kotlin v1.0.0-BETA26](/client-sdks/reference/kotlin) +- [Swift v1.0.0-Beta.8](/client-sdks/reference/swift) +- [.NET v0.0.6-alpha.1](/client-sdks/reference/dotnet) + + +## Why Use Sync Priorities? + +PowerSync's standard sync protocol ensures that: +- The local data view is only updated when a fully consistent checkpoint is available. +- All pending local changes must be uploaded, acknowledged, and synced back before new data is applied. + +While this guarantees consistency, it can lead to delays, especially for large datasets or continuous client-side updates. Sync priorities provide a way to speed up syncing of high-priority data while still maintaining overall integrity. + +## How It Works + +Each bucket is assigned a priority value between 0 and 3, where: + +- 0 is the highest priority and has special behavior (detailed below). +- 3 is the default and lowest priority. +- Lower numbers indicate higher priority. + +Higher-priority data syncs first, and lower-priority data syncs later. If you only use a single priority, there is no difference between priorities 1-3. The difference only comes in when you use multiple different priorities. + +## Syntax and Configuration + +Define priorities using the `priority` YAML key on a bucket definition, or with the `_priority` attribute inside a parameter query: + +```yaml +bucket_definitions: + # Using the `priority` YAML key + user_data: + priority: 1 + parameters: SELECT request.user_id() AS id WHERE ... + data: + # ... + + # Using the `_priority` attribute (useful for multiple parameter queries with different priorities) + project_data: + parameters: SELECT id AS project_id, 2 AS _priority FROM projects WHERE ... + data: + # ... +``` + + +Priorities must be static and cannot depend on row values within a parameter query. + + +## Example: Syncing Lists Before Todos + +Consider a scenario where you want to display lists immediately while loading todos in the background. This approach allows users to view and interact with lists right away without waiting for todos to sync. + +```yaml +bucket_definitions: + user_lists: + priority: 1 # Syncs first + parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() + data: + - SELECT * FROM lists WHERE id = bucket.list_id + + user_todos: + priority: 2 # Syncs after lists + parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() + data: + - SELECT * FROM todos WHERE list_id = bucket.list_id +``` + +The `user_lists` bucket syncs first (priority 1), allowing users to see and interact with their lists immediately. The `user_todos` bucket syncs afterward (priority 2), loading in the background. + +## Behavioral Considerations + +- **Interruption for Higher Priority Data:** Syncing lower-priority data _may_ be interrupted if new data for higher-priority buckets arrives. +- **Local Changes & Consistency:** If local writes fail due to validation or permission issues, they are only reverted after _all_ data has synced. +- **Deleted Data:** Deleted data may only be removed after _all_ priorities have completed syncing. +- **Data Ordering:** Lower-priority data will never appear before higher-priority data. + +## Special Case: Priority 0 + +Priority 0 buckets sync regardless of pending uploads. + +For example, in a collaborative document editing app (e.g., using Yjs), each change is stored as a separate row. Since out-of-order updates don't affect document integrity, Priority 0 can ensure immediate availability of updates. + +Caution: If misused, Priority 0 may cause flickering or inconsistencies, as updates could arrive out of order. + +## Consistency Considerations + +PowerSync's full consistency guarantees only apply once all priorities have completed syncing. + +When higher-priority data is synced, all inserts and updates at that priority level will be consistent. However, deletes are only applied when the full sync completes, so you may still have some stale data at those priority levels. + +Consider the following example: + +Imagine a task management app where users create lists and todos. Some users have millions of todos. To improve first-load speed: + +- Lists are assigned Priority 1, syncing first to allow UI rendering. +- Todos are assigned Priority 2, loading in the background. + +Now, if another user adds new todos, it's possible for the list count (synced at Priority 1) to temporarily not match the actual todos (synced at Priority 2). If real-time accuracy is required, both lists and todos should use the same priority. + +## Client-Side Considerations + +The client SDK APIs for tracking sync status per priority are the same for Sync Rules and Sync Streams: `waitForFirstSync(priority)`, `SyncStatus.priorityStatusEntries()`, and `SyncStatus.statusForPriority(priority)`. See [Client-Side Considerations](/sync/streams/prioritized-sync#client-side-considerations) on the Sync Streams page for details and a Flutter example. diff --git a/sync/rules/reducing-bucket-count.mdx b/sync/rules/reducing-bucket-count.mdx new file mode 100644 index 000000000..c20a2ec30 --- /dev/null +++ b/sync/rules/reducing-bucket-count.mdx @@ -0,0 +1,16 @@ +--- +title: "Reducing Bucket Count" +description: "Diagnose a high bucket count, reduce the number of buckets a user syncs, and raise the per-user limits when needed." +sidebarTitle: "Reducing Buckets" +noindex: true +--- + +{/* Wrapper page: the content is snippets/sync-shared/reducing-bucket-count.mdx, which also renders at sync/advanced/reducing-bucket-count.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} + +import ReducingBucketCount from '/snippets/sync-shared/reducing-bucket-count.mdx'; + + +Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. Configuration examples use Sync Streams syntax. + + + diff --git a/sync/rules/schemas-and-connections.mdx b/sync/rules/schemas-and-connections.mdx new file mode 100644 index 000000000..f6a70c675 --- /dev/null +++ b/sync/rules/schemas-and-connections.mdx @@ -0,0 +1,15 @@ +--- +title: "Schemas and Connections" +description: "Configure Postgres schema usage in Sync Streams/Rules queries, including wildcard schemas for schema-per-tenant setups, and connect to high-availability replicas." +noindex: true +--- + +{/* Wrapper page: the content is snippets/sync-shared/schemas-and-connections.mdx, which also renders at sync/advanced/schemas-and-connections.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} + +import SchemasAndConnections from '/snippets/sync-shared/schemas-and-connections.mdx'; + + +Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. Configuration examples use Sync Streams syntax. + + + diff --git a/sync/rules/sharded-databases.mdx b/sync/rules/sharded-databases.mdx new file mode 100644 index 000000000..8ec1b0a78 --- /dev/null +++ b/sync/rules/sharded-databases.mdx @@ -0,0 +1,15 @@ +--- +title: "Sharded Databases" +description: "Sync data from sharded Postgres databases with per-shard PowerSync connection configuration." +noindex: true +--- + +{/* Wrapper page: the content is snippets/sync-shared/sharded-databases.mdx, which also renders at sync/advanced/sharded-databases.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} + +import ShardedDatabases from '/snippets/sync-shared/sharded-databases.mdx'; + + +Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. + + + diff --git a/sync/rules/storage-version-4.mdx b/sync/rules/storage-version-4.mdx new file mode 100644 index 000000000..e5e974de9 --- /dev/null +++ b/sync/rules/storage-version-4.mdx @@ -0,0 +1,15 @@ +--- +title: "Storage Version 4" +description: "Opt in to storage version 4 for faster sync, incremental reprocessing of Sync Streams changes, and S3 object storage." +noindex: true +--- + +{/* Wrapper page: the content is snippets/sync-shared/storage-version-4.mdx, which also renders at sync/advanced/storage-version-4.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} + +import StorageVersion4 from '/snippets/sync-shared/storage-version-4.mdx'; + + +Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. Configuration examples use Sync Streams syntax. + + + diff --git a/sync/rules/supported-sql.mdx b/sync/rules/supported-sql.mdx new file mode 100644 index 000000000..3f98c7f1f --- /dev/null +++ b/sync/rules/supported-sql.mdx @@ -0,0 +1,157 @@ +--- +title: "Supported SQL in Sync Rules" +sidebarTitle: "Supported SQL" +description: "Reference for SQL syntax, operators, built-in functions, and type casting supported in legacy Sync Rules queries." +--- + +{/* Split page: the Sync Streams version of this page is sync/supported-sql.mdx. This page is deprecated and frozen: no new features or examples. When you fix an error here, check whether the Sync Streams page needs the same fix. */} + + +Sync Rules are deprecated. This page is not updated with new features. For the Sync Streams version, see [Supported SQL](/sync/supported-sql). + + +This guide explains the SQL supported in [Sync Rules](/sync/rules/overview) parameter queries and data queries: what you can write, with examples and restrictions. + +For the exact syntax the compiler accepts, with railroad diagrams and grammar-rule references, see the [Sync Rules grammar reference](/sync/grammar/sync-rules/index). + + + Some fundamental restrictions on the usage of SQL expressions are: + + 1. They must be deterministic: no random or time-based functions. + 2. No external state can be used. + 3. They must operate on data available within a single row/document. For example, no aggregation functions are allowed. + + For parameter-specific WHERE restrictions, see [Filtering: WHERE Clause](#filtering-where-clause). + + +## Query Syntax + +The supported SQL is based on a small subset of the standard SQL syntax: + +- Simple `SELECT` with column selection +- `WHERE` filtering on parameters (see [Filtering: WHERE Clause](#filtering-where-clause)) +- A limited set of [operators](#operators) and [functions](#functions) + +**Not supported:** subqueries, JOINs, CTEs, aggregation, sorting, or set operations (`GROUP BY`, `ORDER BY`, `LIMIT`, `UNION`, etc.). + +## Filtering: WHERE Clause + +Sync Rules queries support a subset of SQL `WHERE` syntax. Allowed operators and combinations are more restrictive than standard SQL. + +**`=` and `IS NULL`:** Compare a row column to a static value or a bucket parameter: + +```sql +-- Static value +WHERE status = 'active' +WHERE deleted_at IS NULL + +-- Bucket parameter +WHERE owner_id = bucket.user_id +``` + +**`AND`:** Supported in both Parameter Queries and Data Queries. In Parameter Queries, each condition may match a different parameter. However, you cannot combine two `IN` expressions on parameters in the same `AND`; split them into separate Parameter Queries instead. + +```sql +-- Supported: parameter condition + row-value condition +WHERE users.id = request.user_id() + AND users.is_admin = true + +-- Not supported: two IN expressions on parameters in the same AND +-- WHERE bucket.list_id IN lists.allowed_ids +-- AND bucket.org_id IN lists.allowed_org_ids +``` + +**`OR`:** Supported when both sides of the `OR` reference the exact same set of parameters. If the two sides use different parameters, use separate parameter queries instead. + +```sql +-- Supported: both sides reference the same parameter +WHERE lists.owner_id = request.user_id() + OR lists.shared_with = request.user_id() + +-- Not supported: sides reference different parameters +-- WHERE lists.owner_id = request.user_id() +-- OR lists.org_id = bucket.org_id +``` + +**`NOT`:** Supported for simple row-value conditions. Not supported on parameter-matching expressions. + +```sql +-- Supported +WHERE status != 'archived' +WHERE deleted_at IS NOT NULL +WHERE NOT users.is_admin = true + +-- Not supported in parameter queries +-- WHERE NOT users.id = request.user_id() +``` + +## Operators + +Operators can be used in `WHERE` clauses and in `SELECT` expressions. When filtering on parameters (e.g. `request.user_id()`, `bucket.user_id`), some combinations are restricted. See [Filtering: WHERE Clause](#filtering-where-clause). + + + + - **Comparison:** `=`, `!=`, `<`, `>`, `<=`, `>=` — If either side is `null`, the result is `null`. + - **Null:** `IS NULL`, `IS NOT NULL` + + + - **Logical:** `AND`, `OR`, `NOT` — See [Filtering: WHERE Clause](#filtering-where-clause) for restrictions when filtering on parameters. + - **Mathematical:** `+`, `-`, `*`, `/` + + + - `||` — Joins two text values together. + + + - `json -> 'path'` — Returns the value as a JSON string. + - `json ->> 'path'` — Returns the extracted value. + + + - `left IN right` — Returns true if `left` is in the `right` JSON array. In Data Queries, `left` must be a row column and `right` cannot be a bucket parameter. In Parameter Queries, either side may be a parameter. + + + +## Functions + +Functions can be used to transform columns/fields before being synced to a client. They operate on row data or parameters. Type names below (`text`, `integer`, `real`, `blob`, `null`) refer to [SQLite storage classes](https://www.sqlite.org/datatype3.html). + +Most functions are from [SQLite built-in functions](https://www.sqlite.org/lang_corefunc.html) and [SQLite JSON functions](https://www.sqlite.org/json1.html). + + + + - **[upper(text)](https://www.sqlite.org/lang_corefunc.html#upper)** — Convert text to upper case. + - **[lower(text)](https://www.sqlite.org/lang_corefunc.html#lower)** — Convert text to lower case. + - **[substring(text, start, length)](https://www.sqlite.org/lang_corefunc.html#substr)** — Extracts a portion of a string based on specified start index and length. Start index is 1-based. Example: `substring(created_at, 1, 10)` returns the date portion of the timestamp. + - **[instr(string, substring)](https://www.sqlite.org/lang_corefunc.html#instr)** — Finds the first occurrence of the substring within the string and returns the number of prior characters plus 1, or 0 if the substring is not found. Useful for locating a delimiter in compound strings. For example, `substring(value, 1, instr(value, '|') - 1)` extracts the portion before a `|` character. + - **[hex(data)](https://www.sqlite.org/lang_corefunc.html#hex)** — Convert blob or text data to hexadecimal text. + - **base64(data)** — Convert blob or text data to base64 text. + - **[length(data)](https://www.sqlite.org/lang_corefunc.html#length)** — For text, return the number of characters. For blob, return the number of bytes. For null, return null. For integer and real, convert to text and return the number of characters. + + + - `CAST(x AS type)` or `x :: type` — Cast to `text`, `numeric`, `integer`, `real`, or `blob`. See [Type mapping](/sync/types) and [SQLite types](https://www.sqlite.org/datatype3.html). + - **[typeof(data)](https://www.sqlite.org/lang_corefunc.html#typeof)** — Returns `text`, `integer`, `real`, `blob`, or `null`. + + + - **[json_each(data)](https://www.sqlite.org/json1.html#jeach)** — Expands a JSON array or object from a request or token parameter into a set of parameter rows. Example: `SELECT value AS project_id FROM json_each(request.jwt() -> 'project_ids')`. See [Expanding JSON Array Into Multiple Parameters](/sync/rules/parameter-queries#expanding-json-array-into-multiple-parameters). + - **[json_extract(data, path)](https://www.sqlite.org/json1.html#jex)** — Same as `->>` operator, but the path must start with `$.` + - **[json_array_length(data)](https://www.sqlite.org/json1.html#jarraylen)** — Given a JSON array (as text), returns the length of the array. If data is null, returns null. If the value is not a JSON array, returns 0. + - **[json_valid(data)](https://www.sqlite.org/json1.html#jvalid)** — Returns 1 if the data can be parsed as JSON, 0 otherwise. + - **json_keys(data)** — Returns the set of keys of a JSON object as a JSON array. Example: `SELECT * FROM items WHERE bucket.user_id IN json_keys(permissions_json)`. + + + - **[ifnull(x, y)](https://www.sqlite.org/lang_corefunc.html#ifnull)** — Returns x if non-null, otherwise returns y. + + + - **[iif(x, y, z)](https://www.sqlite.org/lang_corefunc.html#iif)** — Returns y if x is true, otherwise returns z. + + + - **[unixepoch(time-value, [modifier])](https://www.sqlite.org/lang_datefunc.html)** — Returns a time-value as Unix timestamp. If modifier is "subsec", the result is a floating point number, with milliseconds included in the fraction. The time-value argument is required. This function cannot be used to get the current time. + - **[datetime(time-value, [modifier])](https://www.sqlite.org/lang_datefunc.html)** — Returns a time-value as a date and time string, in the format YYYY-MM-DD HH:MM:SS. If the specifier is "subsec", milliseconds are also included. If the modifier is "unixepoch", the argument is interpreted as a Unix timestamp. Both modifiers can be included: `datetime(timestamp, 'unixepoch', 'subsec')`. The time-value argument is required. This function cannot be used to get the current time. + - **[uuid_blob(id)](https://sqlite.org/src/file/ext/misc/uuid.c)** — Convert a UUID string to bytes. + + + - **[ST_AsGeoJSON(geometry)](/client-sdks/advanced/gis-data-postgis)** — Convert [PostGIS](/client-sdks/advanced/gis-data-postgis) (in Postgres) geometry from WKB to GeoJSON. Combine with JSON operators to extract specific fields. + - **[ST_AsText(geometry)](/client-sdks/advanced/gis-data-postgis)** — Convert [PostGIS](/client-sdks/advanced/gis-data-postgis) (in Postgres) geometry from WKB to Well-Known Text (WKT). + - **[ST_X(point)](/client-sdks/advanced/gis-data-postgis)** — Get the X coordinate of a [PostGIS](/client-sdks/advanced/gis-data-postgis) point (in Postgres). + - **[ST_Y(point)](/client-sdks/advanced/gis-data-postgis)** — Get the Y coordinate of a [PostGIS](/client-sdks/advanced/gis-data-postgis) point (in Postgres). + + diff --git a/sync/rules/sync-data-by-time.mdx b/sync/rules/sync-data-by-time.mdx new file mode 100644 index 000000000..3860f06ab --- /dev/null +++ b/sync/rules/sync-data-by-time.mdx @@ -0,0 +1,174 @@ +--- +title: "Sync Data by Time with Sync Rules" +sidebarTitle: "Sync Data by Time" +description: "Filter and sync data based on time ranges using legacy Sync Rules, with patterns for recent-only and sliding-window queries." +--- + +{/* Split page: the Sync Streams version of this page is sync/advanced/sync-data-by-time.mdx. This page is deprecated and frozen: no new features or examples. When you fix an error here, check whether the Sync Streams page needs the same fix. */} + + +Sync Rules are deprecated. This page is not updated with new features. For the Sync Streams version, see [Sync Data by Time](/sync/advanced/sync-data-by-time). + + +A common need is syncing data based on time, for example, only syncing issues updated in the last 7 days instead of the entire dataset. You might expect to write something like: + +```yaml +bucket_definitions: + issues_after_start_date: + parameters: SELECT request.parameters() ->> 'start_at' AS start_at + data: + - SELECT * FROM issues WHERE updated_at > bucket.start_at +``` + +However, this does not work. Here's why. + +## The Problem + +PowerSync pre-computes and caches which rows belong to which bucket parameters to enable efficient streaming. This means parameter-based filtering is limited to equality checks (`=`, `IN`, `IS NULL`). Range operators like `>`, `<`, `>=`, or `<=` are not supported on parameters. + +Additionally, time-based functions like `now()` are not allowed in parameter expressions because the result changes depending on when the query runs, making pre-computation impossible. + +This guide covers a few practical workarounds. + +## Workarounds + +### 1: Pre-Defined Time Ranges + +Add a boolean column to your table that indicates whether a row falls within a specific time range. Keep this column updated in your source database using a scheduled job. + +For example, add an `updated_this_week` column: + +```sql +ALTER TABLE issues ADD COLUMN updated_this_week BOOLEAN DEFAULT false; +``` + +Update it periodically using a cron job (e.g., with `pg_cron`): + +```sql +UPDATE issues SET updated_this_week = (updated_at > now() - interval '7 days'); +``` + +Then filter on the column in a data query: + +```yaml +bucket_definitions: + recent_issues: + data: + - SELECT * FROM issues WHERE updated_this_week = true +``` + +For multiple time ranges, add multiple bucket definitions and let the client choose which bucket to sync: + +```yaml +bucket_definitions: + issues_1week: + parameters: SELECT WHERE request.parameters() ->> 'range' = '1week' + data: + - SELECT * FROM issues WHERE updated_this_week = true + + issues_1month: + parameters: SELECT WHERE request.parameters() ->> 'range' = '1month' + data: + - SELECT * FROM issues WHERE updated_this_month = true +``` + +The client passes the desired range as a [client parameter](/sync/rules/client-parameters): + +```javascript +await db.connect(connector, { + params: { + range: '1week', + }, +}) +``` + +This approach works well when you have a small, fixed set of time ranges. However, it requires schema changes and a scheduled job to keep the columns updated, and it is limited to pre-defined time ranges. + +If you need more flexibility, such as letting users pick arbitrary date ranges, see Workaround 2 below. + +### 2: Buckets Per Date + +Instead of pre-defined ranges, create a bucket for each date and let the client specify which dates to sync. + +Use `substring` to extract the date portion from a timestamp and match it with `=`: + +```yaml +bucket_definitions: + issues_by_update_at: + parameters: SELECT value AS date FROM json_each(request.parameters() ->> 'dates') + data: + - SELECT * FROM issues WHERE substring(updated_at, 1, 10) = bucket.date +``` + +The client passes the dates it wants as client parameters: + +```javascript +await db.connect(connector, { + params: { + dates: ["2026-01-07", "2026-01-08", "2026-01-09"], + }, +}) +``` + +This gives users full control over which dates to sync, with no schema changes or scheduled jobs required. + +The trade-off is granularity. In this example we're using daily buckets. If you need finer precision (hourly), syncing a large range means many buckets, which can degrade sync performance and approach [PowerSync's limit of 1,000 buckets per user](/resources/performance-and-limits#limits). If you use larger buckets (monthly), you lose the ability to filter accurately. + + +You must commit to a single granularity. Daily buckets mean too many buckets for long ranges. Monthly buckets lose precision for recent data. + + +If that is a problem, for example when you want hourly precision for recent data but do not want hundreds of buckets when syncing a full month, see Workaround 3 below. + +### 3: Multiple Granularities + +Combine multiple granularities in a single bucket definition. This lets you use larger buckets (days) for older data and smaller buckets (hours, minutes) for recent data. + +```yaml +bucket_definitions: + issues_by_time: + parameters: SELECT value AS partition FROM json_each(request.parameters() ->> 'partitions') + data: + # By day (e.g., "2026-01-07") + - SELECT * FROM issues WHERE substring(updated_at, 1, 10) = bucket.partition + # By hour (e.g., "2026-01-07T14") + - SELECT * FROM issues WHERE substring(updated_at, 1, 13) = bucket.partition + # By 10 minutes (e.g., "2026-01-07T14:3") + - SELECT * FROM issues WHERE substring(updated_at, 1, 15) = bucket.partition +``` + +The client then mixes granularities as needed: + +```javascript +await db.connect(connector, { + params: { + partitions: [ + "2026-01-05", + "2026-01-06", + "2026-01-07T10", + "2026-01-07T11", + "2026-01-07T12:0", + "2026-01-07T12:1", + "2026-01-07T12:2" + ] + }, +}) +``` + +Each data query acts as a filter based on the length of the partition value: a day-format partition only matches the day query, an hour-format partition only matches the hour query, and so on. + +This syncs January 5–6 by day, the morning of January 7 by hour, and the last 30 minutes in 10-minute chunks, without creating hundreds of buckets. + +The trade-off is complexity. The client must decide which granularity to use for each time segment, and each row belongs to multiple buckets, which increases replication overhead. + + +When using multiple time granularities (e.g., monthly, daily, hourly), rows move between buckets as time passes. Since each granularity creates a different bucket ID, the client must re-download the row from the new bucket even if it already has the data. This re-download overhead can nullify the benefits of granular filtering. For this reason, in some cases it may be better to sync entire months, avoiding the re-sync overhead, even if you sync more data initially. + + +## Conclusion + +Time-based sync is a common need, but PowerSync doesn't support range operators or time-based functions on parameters directly. To recap the workarounds: + +- **Pre-defined time ranges:** Simplest option. Use when you have a fixed set of time ranges and don't mind schema changes. +- **Buckets per date:** More flexible. Use when you need arbitrary date ranges but can live with a single granularity. +- **Multiple granularities:** Most flexible. Use when you need precision for recent data without syncing hundreds of buckets. Be mindful of the re-sync overhead. diff --git a/sync/rules/types.mdx b/sync/rules/types.mdx new file mode 100644 index 000000000..dc722ed1a --- /dev/null +++ b/sync/rules/types.mdx @@ -0,0 +1,16 @@ +--- +title: "Types" +sidebarTitle: "Type Mapping" +description: "How Postgres, MongoDB, MySQL, SQL Server and Convex types map to PowerSync's SQLite-based sync column definitions." +noindex: true +--- + +{/* Wrapper page: the content is snippets/sync-shared/types.mdx, which also renders at sync/types.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} + +import TypeMapping from '/snippets/sync-shared/types.mdx'; + + +Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. + + + diff --git a/sync/streams/bucket-count.mdx b/sync/streams/bucket-count.mdx index 42a563bce..a7ded8057 100644 --- a/sync/streams/bucket-count.mdx +++ b/sync/streams/bucket-count.mdx @@ -318,10 +318,6 @@ This is why a checkpoint log can read `buckets: 7 | param_results: 6`. One globa The parameter limit can stop a sync while the bucket count still looks safe. A user can fail with far fewer than 1,000 buckets, because their parameter lookups returned more than 1,000 rows. Always check both numbers. - -In legacy [Sync Rules](/sync/rules/overview), these two limits were effectively one number, because each parameter-query result became one bucket. In Sync Streams they can diverge. - - ### Total Buckets vs Buckets Per User The 1,000 limit applies to each individual user, not to your whole instance. Your PowerSync Service can track millions of buckets in total, as long as each user syncs fewer than the limit. A large total bucket count is not a problem on its own. diff --git a/sync/streams/client-usage.mdx b/sync/streams/client-usage.mdx index 0765ba9c7..bd3477831 100644 --- a/sync/streams/client-usage.mdx +++ b/sync/streams/client-usage.mdx @@ -532,7 +532,7 @@ When different components subscribe to the same stream with the same parameters ## Connection Parameters -Connection parameters are a more advanced feature for values that apply to all streams in a session. They're the Sync Streams equivalent of [Client Parameters](/sync/rules/client-parameters) in legacy Sync Rules. +Connection parameters are a more advanced feature for values that apply to all streams in a session. For most use cases, **subscription parameters** (passed when subscribing) are more flexible and recommended. Use connection parameters only when you need a single global value across all streams, like an environment flag. diff --git a/sync/streams/examples.mdx b/sync/streams/examples.mdx index 172d3a2cb..991e91c78 100644 --- a/sync/streams/examples.mdx +++ b/sync/streams/examples.mdx @@ -233,7 +233,7 @@ streams: Clients auto-subscribe to all three streams when they connect. Each query joins through `board_subscriptions` to find relevant data: posts in the user's boards, comments on those posts, and other users sharing those boards. -Unlike with legacy [Sync Rules](/sync/rules/many-to-many-join-tables), you don't need to denormalize your schema or maintain array columns to express these relationships. At scale, denormalizing the scope key onto child tables can still be the best way to control how many buckets each user syncs. See [Bucket Count and Limits](/sync/streams/bucket-count). +You don't need to denormalize your schema or maintain array columns to express these relationships. At scale, denormalizing the scope key onto child tables can still be the best way to control how many buckets each user syncs. See [Bucket Count and Limits](/sync/streams/bucket-count). ## Use Case Examples diff --git a/sync/streams/overview.mdx b/sync/streams/overview.mdx index 5fbe3ba2a..4f3d808e5 100644 --- a/sync/streams/overview.mdx +++ b/sync/streams/overview.mdx @@ -8,12 +8,6 @@ With Sync Streams, you write SQL-like queries to define streams of data, and you For example, you might define a stream that syncs only the current user's to-do items, another for shared projects they have access to, and another for reference data that everyone needs. Your app subscribes to these streams on demand, and only that data syncs to the device. Apps that need all relevant data available upfront can set `auto_subscribe: true` so streams sync automatically when clients connect. - -**Are you still using Sync Rules?** Sync Streams support everything Sync Rules do, plus more expressive queries (including JOIN support), on-demand syncing, and a simpler developer experience (e.g. React hooks that manage subscriptions automatically). - -You can migrate in a few clicks. Click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to generate a draft from your current config. See [Migrate to Sync Streams](/sync/rules/migrate-to-sync-streams) for details. - - ## How It Works Each PowerSync Service instance has a deployed Sync Streams configuration: a YAML file that defines the streams that exist. Each stream has a name and a SQL-like query that selects the tables and columns to sync, filters rows by static conditions or by parameters, and can rename or transform columns. The Service uses this configuration in two places: when it replicates data from your source database into buckets, and when it streams those buckets to clients. diff --git a/sync/streams/parameters.mdx b/sync/streams/parameters.mdx index fe3e90d41..aa88f7653 100644 --- a/sync/streams/parameters.mdx +++ b/sync/streams/parameters.mdx @@ -54,7 +54,7 @@ streams: ## Connection Parameters -Specified "globally" at the connection level, before any streams are subscribed. These are the equivalent of [Client Parameters](/sync/rules/client-parameters) in Sync Rules. Use them when you need a value that applies across all streams for the session. +Specified "globally" at the connection level, before any streams are subscribed. Use them when you need a value that applies across all streams for the session. ```yaml streams: diff --git a/sync/streams/prioritized-sync.mdx b/sync/streams/prioritized-sync.mdx index 064c1dd56..15e73407b 100644 --- a/sync/streams/prioritized-sync.mdx +++ b/sync/streams/prioritized-sync.mdx @@ -3,11 +3,13 @@ title: "Prioritized Sync" description: "Prioritize which tables sync first so users can start working immediately while remaining data continues loading in the background." --- +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/prioritized-sync.mdx. That page is deprecated and frozen, but when you fix an error here, check whether it needs the same fix. Do not mention Sync Rules on this page. */} + ## Overview PowerSync supports defining sync priorities, which allows you to control the sync order for different data. This is particularly useful when certain data should be available sooner than others. -In Sync Streams, priorities are assigned to streams and PowerSync manages the underlying buckets internally. (In legacy Sync Rules, priorities were assigned to buckets explicitly.) +In Sync Streams, priorities are assigned to streams and PowerSync manages the underlying buckets internally. ## Why Use Sync Priorities? @@ -19,7 +21,7 @@ While this guarantees consistency, it can lead to delays, especially for large d ## How It Works -Each bucket is assigned a priority value between 0 and 3, where: +Each stream is assigned a priority value between 0 and 3, where: - 0 is the highest priority and has special behavior (detailed below). - 3 is the default and lowest priority. @@ -27,57 +29,9 @@ Each bucket is assigned a priority value between 0 and 3, where: Higher-priority data syncs first, and lower-priority data syncs later. If you only use a single priority, there is no difference between priorities 1-3. The difference only comes in when you use multiple different priorities. - - -In Sync Streams, you assign priorities directly to streams. PowerSync manages buckets internally, so you don't need to think about bucket structure. Each stream with a given priority will have its data synced at that priority level. - -```yaml -streams: - lists: - auto_subscribe: true - query: SELECT * FROM lists WHERE owner_id = auth.user_id() - priority: 1 # Syncs first - - todos: - auto_subscribe: true - query: SELECT * FROM todos WHERE list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) - priority: 2 # Syncs after lists -``` - -Clients can also override the priority when subscribing: - -```js -// Override the stream's default priority for this subscription -const sub = await db.syncStream('todos', { list_id: 'abc' }).subscribe({ priority: 1 }); -``` - -When different components subscribe to the same stream with the same parameters but different priorities, PowerSync uses the highest priority for syncing. That higher priority is kept until the subscription ends (or its TTL expires). Subscriptions with different parameters are independent and do not conflict. - - -In Sync Rules, you assign priorities to bucket definitions. The priority determines when data in that bucket syncs relative to other buckets. - -```yaml -bucket_definitions: - user_lists: - priority: 1 # Syncs first - parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() - data: - - SELECT * FROM lists WHERE id = bucket.list_id - - user_todos: - priority: 2 # Syncs after lists - parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() - data: - - SELECT * FROM todos WHERE list_id = bucket.list_id -``` - - - ## Syntax and Configuration - - -In Sync Streams, set the `priority` option on the stream definition: +Set the `priority` option on the stream definition: ```yaml streams: @@ -91,38 +45,24 @@ streams: query: SELECT * FROM background_table WHERE user_id = auth.user_id() priority: 2 ``` - - -In Sync Rules, priorities can be defined using the `priority` YAML key on bucket definitions, or with the `_priority` attribute inside parameter queries: -```yaml -bucket_definitions: - # Using the `priority` YAML key - user_data: - priority: 1 - parameters: SELECT request.user_id() AS id WHERE ... - data: - # ... - - # Using the `_priority` attribute (useful for multiple parameter queries with different priorities) - project_data: - parameters: SELECT id AS project_id, 2 AS _priority FROM projects WHERE ... - data: - # ... +Clients can also override the priority when subscribing: + +```js +// Override the stream's default priority for this subscription +const sub = await db.syncStream('todos', { list_id: 'abc' }).subscribe({ priority: 1 }); ``` - - + +When different components subscribe to the same stream with the same parameters but different priorities, PowerSync uses the highest priority for syncing. That higher priority is kept until the subscription ends (or its TTL expires). Subscriptions with different parameters are independent and do not conflict. -Priorities must be static and cannot depend on row values within a parameter query. +Priorities are static values that you set in the stream definition or when subscribing. They cannot depend on row values. ## Example: Syncing Lists Before Todos Consider a scenario where you want to display lists immediately while loading todos in the background. This approach allows users to view and interact with lists right away without waiting for todos to sync. - - ```yaml config: edition: 3 @@ -142,38 +82,17 @@ streams: ``` The `lists` stream syncs first (priority 1), allowing users to see and interact with their lists immediately. The `todos` stream syncs afterward (priority 2), loading in the background. - - -```yaml -bucket_definitions: - user_lists: - priority: 1 # Syncs first - parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() - data: - - SELECT * FROM lists WHERE id = bucket.list_id - - user_todos: - priority: 2 # Syncs after lists - parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() - data: - - SELECT * FROM todos WHERE list_id = bucket.list_id -``` - -The `user_lists` bucket syncs first (priority 1), allowing users to see and interact with their lists immediately. The `user_todos` bucket syncs afterward (priority 2), loading in the background. - - - ## Behavioral Considerations -- **Interruption for Higher Priority Data**: Syncing lower-priority data _may_ be interrupted if new data for higher-priority streams/buckets arrives. +- **Interruption for Higher Priority Data**: Syncing lower-priority data _may_ be interrupted if new data for higher-priority streams arrives. - **Local Changes & Consistency**: If local writes fail due to validation or permission issues, they are only reverted after _all_ data has synced. - **Deleted Data**: Deleted data may only be removed after _all_ priorities have completed syncing. Future updates may improve this behavior. - **Data Ordering**: Lower-priority data will never appear before higher-priority data. ## Special Case: Priority 0 -Priority 0 buckets sync regardless of pending uploads. +Priority 0 streams sync regardless of pending uploads. For example, in a collaborative document editing app (e.g., using Yjs), each change is stored as a separate row. Since out-of-order updates don’t affect document integrity, Priority 0 can ensure immediate availability of updates. diff --git a/sync/streams/quickstart.mdx b/sync/streams/quickstart.mdx index c8ae8cd68..36a84d050 100644 --- a/sync/streams/quickstart.mdx +++ b/sync/streams/quickstart.mdx @@ -129,7 +129,6 @@ const sub = await db.syncStream('list_todos', { list_id: 'abc123' }).subscribe() Set `auto_subscribe: true` to sync data automatically when clients connect. This is useful for: - Reference data that all users need, or that is needed in many screens in the app. - User data that should always be available offline. -- Keeping the "sync everything upfront" behavior of legacy [Sync Rules](/sync/rules/overview) when migrating to Sync Streams. ```yaml config: diff --git a/sync/supported-sql.mdx b/sync/supported-sql.mdx index 9287c428c..550e76ab6 100644 --- a/sync/supported-sql.mdx +++ b/sync/supported-sql.mdx @@ -1,11 +1,13 @@ --- title: "Supported SQL" -description: "Reference for SQL syntax, operators, built-in functions, and type casting supported in Sync Streams/Sync Rules queries." +description: "Reference for SQL syntax, operators, built-in functions, and type casting supported in Sync Streams queries." --- -This guide explains the SQL supported in [Sync Streams](/sync/streams/overview) and [Sync Rules (legacy)](/sync/rules/overview): what you can write, with examples and restrictions. +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/supported-sql.mdx. That page is deprecated and frozen, but when you fix an error here, check whether it needs the same fix. Do not mention Sync Rules on this page. */} -For the exact syntax the compiler accepts — railroad diagrams and grammar-rule references — see the [Sync Streams](/sync/grammar/sync-streams/index) or [Sync Rules](/sync/grammar/sync-rules/index) grammar reference. +This guide explains the SQL supported in [Sync Streams](/sync/streams/overview): what you can write, with examples and restrictions. + +For the exact syntax the compiler accepts, with railroad diagrams and grammar-rule references, see the [Sync Streams grammar reference](/sync/grammar/sync-streams/index). Some fundamental restrictions on the usage of SQL expressions are: @@ -19,36 +21,22 @@ For the exact syntax the compiler accepts — railroad diagrams and grammar-rule ## Query Syntax -The supported SQL is based on a subset of the standard SQL syntax. Sync Streams support more SQL features than the legacy Sync Rules. - - - - - `SELECT` with column selection and [`WHERE` filtering](#filtering-where-clause) - - [Subqueries](/sync/streams/queries#using-subqueries) with `IN (SELECT ...)` and nested subqueries - - [`INNER JOIN`](#join-syntax) (selected columns must come from a single table) - - [Common Table Expressions (CTEs)](#cte-and-with-syntax) via the `with:` block - - Multiple queries per stream via `queries:` - - Table-valued functions such as `json_each()` for [expanding arrays](/sync/streams/parameters#expanding-json-arrays) - - `BETWEEN` and `CASE` expressions - - A limited set of [operators](#operators) and [functions](#functions) - - **Not supported**: aggregation, sorting, or set operations (`GROUP BY`, `ORDER BY`, `LIMIT`, `UNION`, etc.). See [Writing Queries](/sync/streams/queries) for details. - - - - Simple `SELECT` with column selection - - `WHERE` filtering on parameters (see [Filtering: WHERE Clause](#filtering-where-clause)) - - A limited set of [operators](#operators) and [functions](#functions) - - **Not supported**: subqueries, JOINs, CTEs, aggregation, sorting, or set operations (`GROUP BY`, `ORDER BY`, `LIMIT`, `UNION`, etc.). - - +The supported SQL is based on a subset of the standard SQL syntax: -## Filtering: WHERE Clause +- `SELECT` with column selection and [`WHERE` filtering](#filtering-where-clause) +- [Subqueries](/sync/streams/queries#using-subqueries) with `IN (SELECT ...)` and nested subqueries +- [`INNER JOIN`](#join-syntax) (selected columns must come from a single table) +- [Common Table Expressions (CTEs)](#cte-and-with-syntax) via the `with:` block +- Multiple queries per stream via `queries:` +- Table-valued functions such as `json_each()` for [expanding arrays](/sync/streams/parameters#expanding-json-arrays) +- `BETWEEN` and `CASE` expressions +- A limited set of [operators](#operators) and [functions](#functions) -Sync queries support a subset of SQL `WHERE` syntax. Allowed operators and combinations differ between Sync Streams and Sync Rules, and are more restrictive than standard SQL. +**Not supported**: aggregation, sorting, or set operations (`GROUP BY`, `ORDER BY`, `LIMIT`, `UNION`, etc.). See [Writing Queries](/sync/streams/queries) for details. - - +## Filtering: WHERE Clause + +Sync Streams queries support a subset of SQL `WHERE` syntax. Allowed operators and combinations are more restrictive than standard SQL. **`=` and `IS NULL`** — Compare a row column to a static value, a parameter, or another column: @@ -109,59 +97,6 @@ WHERE category NOT IN ROW('draft', 'hidden') -- WHERE id NOT IN subscription.parameter('excluded_ids') ``` - - - -**`=` and `IS NULL`** — Compare a row column to a static value or a bucket parameter: - -```sql --- Static value -WHERE status = 'active' -WHERE deleted_at IS NULL - --- Bucket parameter -WHERE owner_id = bucket.user_id -``` - -**`AND`** — Supported in both Parameter Queries and Data Queries. In Parameter Queries, each condition may match a different parameter. However, you cannot combine two `IN` expressions on parameters in the same `AND`; split them into separate Parameter Queries instead. - -```sql --- Supported: parameter condition + row-value condition -WHERE users.id = request.user_id() - AND users.is_admin = true - --- Not supported: two IN expressions on parameters in the same AND --- WHERE bucket.list_id IN lists.allowed_ids --- AND bucket.org_id IN lists.allowed_org_ids -``` - -**`OR`** — Supported when both sides of the `OR` reference the exact same set of parameters. If the two sides use different parameters, use separate parameter queries instead. - -```sql --- Supported: both sides reference the same parameter -WHERE lists.owner_id = request.user_id() - OR lists.shared_with = request.user_id() - --- Not supported: sides reference different parameters --- WHERE lists.owner_id = request.user_id() --- OR lists.org_id = bucket.org_id -``` - -**`NOT`** — Supported for simple row-value conditions. Not supported on parameter-matching expressions. - -```sql --- Supported -WHERE status != 'archived' -WHERE deleted_at IS NOT NULL -WHERE NOT users.is_admin = true - --- Not supported in parameter queries --- WHERE NOT users.id = request.user_id() -``` - - - - ## Operators Operators can be used in `WHERE` clauses and in `SELECT` expressions. When filtering on parameters (e.g. `auth.user_id()`, `subscription.parameter('id')`), some combinations are restricted — see [Filtering: WHERE Clause](#filtering-where-clause). @@ -183,15 +118,12 @@ Operators can be used in `WHERE` clauses and in `SELECT` expressions. When filte - `json ->> 'path'` — Returns the extracted value. - - **Sync Streams:** `left IN right` — `left` can be a row column and `right` a parameter array (e.g. `id IN subscription.parameter('ids')`), or `left` a parameter and `right` a row JSON array column. Also supports subqueries: `id IN (SELECT ...)`. - - **Sync Rules:** Returns true if `left` is in the `right` JSON array. In Data Queries, `left` must be a row column and `right` cannot be a bucket parameter. In Parameter Queries, either side may be a parameter. + - `left IN right` — `left` can be a row column and `right` a parameter array (e.g. `id IN subscription.parameter('ids')`), or `left` a parameter and `right` a row JSON array column. Also supports subqueries: `id IN (SELECT ...)`. - `x BETWEEN a AND b`, `x NOT BETWEEN a AND b` — True if `x` is in the inclusive range `[a, b]`. Usable in `WHERE` or as a `SELECT` expression. If any operand is `null`, the result is `null`. Example: `WHERE price BETWEEN 10 AND 100` - - Supported in Sync Streams only. Not available in Sync Rules. - ` && ` — True if the JSON array in `left` and the set `right` share at least one value. Use when the row stores an array (e.g. a `tagged_users` column). `left` must be a row column (JSON array); `right` must be a subquery or parameter array. @@ -199,8 +131,6 @@ Operators can be used in `WHERE` clauses and in `SELECT` expressions. When filte Example: `WHERE tagged_users && (SELECT id FROM org_members WHERE org_id = auth.parameter('org_id'))` Use `IN` when the row has a single value to check against a set; use `&&` when the row has an array and you want to match any element. - - Supported in Sync Streams only. Not available in Sync Rules. @@ -225,13 +155,11 @@ Most functions are from [SQLite built-in functions](https://www.sqlite.org/lang_ - **[typeof(data)](https://www.sqlite.org/lang_corefunc.html#typeof)** — Returns `text`, `integer`, `real`, `blob`, or `null`. - - **[json_each(data)](https://www.sqlite.org/json1.html#jeach)** — Expands a JSON array into rows. - - **Sync Streams:** Works with auth and connection parameters (e.g. `JOIN json_each(auth.parameter('ids')) AS t` or `WHERE id IN (SELECT value FROM json_each(auth.parameter('ids')))`). Can also be used with columns from joined tables in some cases (e.g. `SELECT * FROM lists WHERE id IN (SELECT lists.value FROM access_control a, json_each(a.allowed_lists) as lists WHERE a.user = auth.user_id())`). See [Expanding JSON arrays](/sync/streams/parameters#expanding-json-arrays). - - **Sync Rules:** Expands a JSON array or object from a request or token parameter into a set of parameter rows. Example: `SELECT value AS project_id FROM json_each(request.jwt() -> 'project_ids')`. + - **[json_each(data)](https://www.sqlite.org/json1.html#jeach)** — Expands a JSON array into rows. Works with auth and connection parameters (e.g. `JOIN json_each(auth.parameter('ids')) AS t` or `WHERE id IN (SELECT value FROM json_each(auth.parameter('ids')))`). Can also be used with columns from joined tables in some cases (e.g. `SELECT * FROM lists WHERE id IN (SELECT lists.value FROM access_control a, json_each(a.allowed_lists) as lists WHERE a.user = auth.user_id())`). See [Expanding JSON arrays](/sync/streams/parameters#expanding-json-arrays). - **[json_extract(data, path)](https://www.sqlite.org/json1.html#jex)** — Same as `->>` operator, but the path must start with `$.` - **[json_array_length(data)](https://www.sqlite.org/json1.html#jarraylen)** — Given a JSON array (as text), returns the length of the array. If data is null, returns null. If the value is not a JSON array, returns 0. - **[json_valid(data)](https://www.sqlite.org/json1.html#jvalid)** — Returns 1 if the data can be parsed as JSON, 0 otherwise. - - **json_keys(data)** — Returns the set of keys of a JSON object as a JSON array. Example: `SELECT * FROM items WHERE bucket.user_id IN json_keys(permissions_json)`. + - **json_keys(data)** — Returns the set of keys of a JSON object as a JSON array. Example: `SELECT id, json_keys(permissions_json) AS permission_keys FROM items`. - **[ifnull(x, y)](https://www.sqlite.org/lang_corefunc.html#ifnull)** — Returns x if non-null, otherwise returns y. @@ -251,7 +179,7 @@ Most functions are from [SQLite built-in functions](https://www.sqlite.org/lang_ - **table_name()** — Returns the name of the table the row was replicated from. This is the source table's name, not the alias or the output table name. - **table_suffix()** — Returns the part of the table name matched by the trailing `%` of a [wildcard table name](/sync/advanced/partitioned-tables). For example, with `FROM "todos_%" AS todos`, `todos.table_suffix()` returns `2024` for rows from the `todos_2024` table. On tables without a wildcard name, the result is always empty and the compiler reports a warning. - Supported in Sync Streams only, with PowerSync Service v1.24.0 or later. + Requires PowerSync Service v1.24.0 or later. - **[ST_AsGeoJSON(geometry)](/client-sdks/advanced/gis-data-postgis)** — Convert [PostGIS](/client-sdks/advanced/gis-data-postgis) (in Postgres) geometry from WKB to GeoJSON. Combine with JSON operators to extract specific fields. @@ -265,8 +193,6 @@ If you need an operator or function not listed, [contact us](/resources/contact- ## JOIN Syntax -Supported in Sync Streams only. Not available in Sync Rules. - Sync Streams support a subset of join syntax. The following rules define what is valid: - **Only inner joins:** Use `JOIN` or `INNER JOIN`. `LEFT`, `RIGHT`, and `OUTER` joins are not supported. @@ -295,8 +221,6 @@ For how to use JOINs in your stream queries (when to use them, patterns, and exa ## CTE and WITH Syntax -Supported in Sync Streams only. Not available in Sync Rules. - Common Table Expressions (CTEs) can be defined in a `with:` block **inside a stream** (stream-level, scoped to that stream) or at the **top level** of the Sync Config (global, shared across all streams). Each CTE is a name and a single `SELECT` query. The following rules apply: - **Stream-level CTEs take precedence over global CTEs.** If a stream defines a CTE with the same name as a global CTE, the stream-level definition is used within that stream. @@ -341,8 +265,6 @@ For how to use CTEs, see [Common Table Expressions (CTEs)](/sync/streams/ctes). ## CASE Expressions -Supported in Sync Streams only. Not available in Sync Rules. - `CASE` is allowed anywhere an expression is allowed — in `SELECT` columns or `WHERE` clauses. **Searched CASE** — Each `WHEN` is an independent boolean condition: diff --git a/sync/types.mdx b/sync/types.mdx index 0e69f3ce0..eae7205d2 100644 --- a/sync/types.mdx +++ b/sync/types.mdx @@ -4,165 +4,8 @@ sidebarTitle: "Type Mapping" description: "How Postgres, MongoDB, MySQL, SQL Server and Convex types map to PowerSync's SQLite-based sync column definitions." --- -import BinaryType from '/snippets/binary-type.mdx'; +{/* Wrapper page: the content is snippets/sync-shared/types.mdx, which also renders at sync/rules/types.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} -The supported client-side SQLite types are: +import TypeMapping from '/snippets/sync-shared/types.mdx'; -1. `null` -2. `integer`: a 64-bit signed integer -3. `real`: a 64-bit floating point number -4. `text`: A UTF-8 text string -5. `blob`: Binary data - - -## Postgres Type Mapping - -Postgres types are mapped to SQLite types as follows: - -| Postgres Data Type | PowerSync / SQLite Column Type | Notes | -|--------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `text`, `varchar` | `text` | | -| `int2`, `int4`, `int8` | `integer` | | -| `numeric` / `decimal` | `text` | These types have arbitrary precision in Postgres, so can only be represented accurately as text in SQLite | -| `bool` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | -| `float4`, `float8` | `real` | | -| `enum` | `text` | | -| `uuid` | `text` | | -| `timestamptz` | `text` | Format: `YYYY-MM-DD hh:mm:ss.sssZ`. This is compatible with ISO8601 and SQLite's functions. Precision matches the precision used in Postgres. `-infinity` becomes `0000-01-01 00:00:00Z` and `infinity` becomes `9999-12-31 23:59:59Z`. | -| `timestamp` | `text` | Format: `YYYY-MM-DD hh:mm:ss.sss`. In most cases, timestamptz should be used instead. `-infinity` becomes `0000-01-01 00:00:00` and `infinity` becomes `9999-12-31 23:59:59`. | -| `date`, `time` | `text` | | -| `json`, `jsonb` | `text` | `json` and `jsonb` values are treated as `text` values in their serialized representation. [JSON functions and operators](/sync/supported-sql#operators) operate directly on these `text` values. | -| `interval` | `text` | | -| `macaddr` | `text` | | -| `inet` | `text` | | -| `bytea` | `blob` | Cannot sync directly to client — convert to hex or base64 first. See [Operators & Functions](/sync/supported-sql). | -| `geometry` (PostGIS) | `text` | Hex string of the binary data. Use the [ST functions](/sync/supported-sql#functions) to convert to other formats | -| Arrays | `text` | JSON array. | -| `DOMAIN` types | `text` / depends | Depending on [compatibility options](/sync/advanced/compatibility#custom_postgres_types), inner type or raw wire representation (legacy). | -| Custom types | `text` | Depending on [compatibility options](/sync/advanced/compatibility#custom_postgres_types), JSON object or raw wire representation (legacy). | -| (Multi-)ranges | `text` | Depending on [compatibility options](/sync/advanced/compatibility#custom_postgres_types), JSON object (array for multi-ranges) or raw wire representation (legacy). | - - - - -## Convex Type Mapping - - - The Convex replicator is currently released as an [experimental feature](/resources/feature-status). APIs and - behavior may change, and we can't yet guarantee continued support or long-term stability. - - -Convex values are mapped to SQLite types as follows: - -| Convex Type | TS/JS Type | PowerSync / SQLite Column Type | Notes | -| ----------- | ---------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `Id` | `string` | `text` | Convex document IDs are exposed as `_id` and can be synced as `text`. For synced client tables, use client-side ID mapping with a stable UUID column as `id` instead of relying on Convex-generated `_id` values. | -| `Null` | `null` | `null` | | -| `Int64` | `base-10 string` | `text` | Cast to `INTEGER` in Sync Streams when you want to sync the value as a SQLite integer. | -| `Float64` | `number` | `real` | | -| `Boolean` | `boolean` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | -| `String` | `string` | `text` | | -| `Bytes` | `base64 string` | `text` | Decode from base64 in your app if you need binary data on the client. | -| `Array` | `Array` | `text` | Converted to a JSON string. | -| `Object` | `Object` | `text` | Converted to a JSON string. | -| `Record` | `Record` | `text` | Converted to a JSON string. | - -- Convex documents are converted to a flat list of columns, one column per top-level field. -- Nested objects and arrays are converted to JSON, and [JSON functions and operators](/sync/supported-sql#operators) can be used to query them in Sync Streams or on the client-side SQLite database. -- Cast Convex `Int64` fields to `INTEGER` in Sync Streams when you want SQLite integer values on the client, for example `CAST(an_int64_column AS INTEGER) AS an_int64_column`. - - -## MongoDB Type Mapping - -MongoDB types are mapped to SQLite types as follows: - -| BSON Type | PowerSync / SQLite Column Type | Notes | -|--------------------|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------| -| `String` | `text` | | -| `Int`, `Long` | `integer` | | -| `Double` | `real` | | -| `Decimal128` | `text` | | -| `Object` | `text` | Converted to a JSON string | -| `Array` | `text` | Converted to a JSON string | -| `ObjectId` | `text` | Lower-case hex string | -| `UUID` | `text` | Lower-case hex string | -| `Boolean` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | -| `Date` | `text` | Format: `YYYY-MM-DD hh:mm:ss.sssZ` | -| `Null` | `null` | | -| `Binary` | `blob` | Cannot sync directly to client — convert to hex or base64 first. See [Operators & Functions](/sync/supported-sql). | -| Regular Expression | `text` | JSON text in the format `{"pattern":"...","options":"..."}` | -| `Timestamp` | `integer` | Converted to a 64-bit integer | -| `Undefined` | `null` | | -| `DBPointer` | `text` | JSON text in the format `{"collection":"...","oid":"...","db":"...","fields":...}` | -| `JavaScript` | `text` | JSON text in the format `{"code": "...", "scope": ...}` | -| `Symbol` | `text` | | -| `MinKey`, `MaxKey` | `null` | | - -* Data is converted to a flat list of columns, one column per top-level field in the MongoDB document. -* Special BSON types are converted to plain SQLite alternatives. For example, `ObjectId`, `Date`, `UUID` are all converted to a plain `TEXT` column. -* Nested objects and arrays are converted to JSON, and [JSON functions and operators](/sync/supported-sql#operators) can be used to query them (in the Sync Streams / Sync Rules and/or on the client-side SQLite statements). -* Binary data nested in objects or arrays is not supported. - - - - -## MySQL Type Mapping - -MySQL support is currently in a [Beta release](/resources/feature-status). - -MySQL types are mapped to SQLite types as follows: - -| MySQL Data Type | PowerSync / SQLite Column Type | Notes | -|----------------------------------------------------|--------------------------------|-----------------------------------------------------------------------------------| -| `tinyint`, `smallint`, `mediumint`, `bigint`, `integer`, `int` | `integer` | | -| `numeric`, `decimal` | `text` | | -| `bool`, `boolean` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | -| `float`, `double`, `real` | `real` | | -| `enum` | `text` | | -| `set` | `text` | Converted to JSON array | -| `char`, `varchar` | `text` | | -| `tinytext`, `text`, `mediumtext`, `longtext` | `text` | | -| `timestamp` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | -| `date` | `text` | Format: `YYYY-MM-DD` | -| `time`, `datetime` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | -| `year` | `text` | | -| `json` | `text` | There is no dedicated JSON type in SQLite — JSON functions operate directly on text values. | -| `bit` | `blob` | * See note below regarding syncing binary types | -| `binary`, `varbinary` | `blob` | | -| `image` | `blob` | | -| `geometry`, `geometrycollection` | `blob` | | -| `point`, `multipoint` | `blob` | | -| `linestring`, `multilinestring` | `blob` | | -| `polygon`, `multipolygon` | `blob` | | - - - - -## SQL Server Type Mapping - -SQL Server support is currently in a [Beta release](/resources/feature-status). - -SQL Server types are mapped to SQLite types as follows: - -| SQL Server Data Type | PowerSync / SQLite Column Type | Notes | -|----------------------------------------------------|--------------------------------|--------------------------------------------------------| -| `tinyint`, `smallint`, `int`, `bigint` | `integer` | | -| `numeric`, `decimal` | `text` | Numeric string | -| `float`, `real` | `real` | | -| `bit` | `integer` | | -| `money`, `smallmoney` | `text` | Numeric string | -| `xml` | `text` | | -| `char`, `nchar`, `ntext` | `text` | | -| `varchar`, `nvarchar`, `text` | `text` | | -| `uniqueidentifier` | `text` | | -| `timestamp` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | -| `date` | `text` | Format: `YYYY-MM-DD` | -| `time` | `text` | Format: `HH:mm:ss.sss` | -| `datetime`, `datetime2`, `smalldatetime`, `datetimeoffset` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | -| `json` | `text` | Only exists for Azure SQL Database and SQL Server 2025 | -| `geometry`, `geography` | `text` | `text` of JSON object describing the spatial data type | -| `binary`, `varbinary`, `image` | `blob` | * See note below regarding binary types | -| `rowversion`, `timestamp` | `blob` | * See note below regarding binary types | -| User Defined Types: `hiearchyid` | `blob` | * See note below regarding binary types | - - + From 7c88c482492beedd64e080e642ad1d4947020228 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Fri, 18 Sep 2026 18:28:11 +0200 Subject: [PATCH 11/20] Simplify the check:links architecture --- .claude/CLAUDE.md | 3 +-- .claude/commands/lint-docs.md | 8 ++++---- package.json | 3 +-- scripts/check-links.mjs | 10 ++++++++++ 4 files changed, 16 insertions(+), 8 deletions(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index ca29c2dab..ec4279b40 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -153,8 +153,7 @@ Sync Rules are deprecated. New documentation and updates cover Sync Streams only - Verify technical claims and run code examples before publication. Select other checks appropriate to the change. - Run `vale ` for changed MDX pages. Add new technical terms to `.github/vale/config/vocabularies/PowerSync/accept.txt`; do not add ordinary misspellings. -- After link or navigation changes, run `npx mintlify broken-links`. Mintlify requires Node 20.17–24; if needed, use `PATH="/opt/homebrew/opt/node@24/bin:$PATH" npx mintlify broken-links`. -- For anchor and snippet checks, use `pnpm check:links`. It wraps the Mintlify anchor check in `scripts/check-links.mjs` so that anchors defined in imported snippets resolve. Validate repository instruction links as file paths, since the site checker does not cover all of them. +- After link or navigation changes, run `pnpm check:links`. It runs the Mintlify path, anchor, and snippet checks through `scripts/check-links.mjs`, resolves anchors defined in imported snippets, and is the same check CI runs. It tells you if your Node version is unsupported. Validate repository instruction links as file paths, since the site checker does not cover them. - Use [the lint command](commands/lint-docs.md) for the check workflow and [the reviewer](agents/document-reviewer.md) for editorial review. Passing linters does not establish technical accuracy or style compliance. ## Git Workflow diff --git a/.claude/commands/lint-docs.md b/.claude/commands/lint-docs.md index e0ac914c1..ca9027388 100644 --- a/.claude/commands/lint-docs.md +++ b/.claude/commands/lint-docs.md @@ -1,9 +1,9 @@ --- -allowed-tools: Read, Bash(npx mintlify *), Bash(PATH=* npx mintlify *), Bash(vale *), Bash(git diff *), Bash(git status *) -description: Run Vale and Mintlify broken-link checks for documentation changes and report failures. +allowed-tools: Read, Bash(pnpm check:links), Bash(PATH=* pnpm check:links), Bash(vale *), Bash(git diff *), Bash(git status *) +description: Run Vale and the link check for documentation changes and report failures. --- -1. Read the canonical [Verification](../CLAUDE.md#verification) section for commands, supported Node versions, and vocabulary rules. +1. Read the canonical [Verification](../CLAUDE.md#verification) section for commands and vocabulary rules. 2. Use the requested file scope. Otherwise, identify changed MDX pages with `git diff main --name-only --diff-filter=ACMR -- '*.mdx'` and `git status --short`, including untracked pages. -3. Run `vale ` for each page and `npx mintlify broken-links` for the site. Use the canonical Node fallback if needed. +3. Run `vale ` for each page and `pnpm check:links` once for the site. 4. Report findings by file, suggested fixes, and totals for errors, warnings, and suggestions. Report failed or unavailable checks separately from content findings. diff --git a/package.json b/package.json index 490f97ce2..3bd14d0be 100644 --- a/package.json +++ b/package.json @@ -4,8 +4,7 @@ "packageManager": "pnpm@11.3.0+sha512.2c403d6594527287672b1f7056343a1f7c3634036a67ffabfcc2b3d7595d843768f8787148d1b57cf7956c90606bbd192857c363af19e96d2d0ec9ec5741d215", "scripts": { "dev": "mintlify dev", - "check:links": "node scripts/check-links.mjs", - "check:links:mintlify": "mintlify broken-links --check-anchors --check-snippets" + "check:links": "node scripts/check-links.mjs" }, "devDependencies": { "mintlify": "^4.2.520" diff --git a/scripts/check-links.mjs b/scripts/check-links.mjs index a2fa6c6c8..e6f939e84 100644 --- a/scripts/check-links.mjs +++ b/scripts/check-links.mjs @@ -16,6 +16,16 @@ import { spawnSync } from 'node:child_process'; import { existsSync, readFileSync } from 'node:fs'; import path from 'node:path'; +const nodeMajor = Number(process.versions.node.split('.')[0]); +if (nodeMajor < 20 || nodeMajor > 24) { + console.error( + `The Mintlify CLI supports Node 20.17 to 24, but this is Node ${process.versions.node}.\n` + + 'Run `nvm use` (the repo pins Node 24 in .nvmrc) or prefix the command with ' + + 'PATH="/opt/homebrew/opt/node@24/bin:$PATH".', + ); + process.exit(1); +} + const root = process.cwd(); const localBin = path.join(root, 'node_modules', '.bin', 'mintlify'); const bin = existsSync(localBin) ? localBin : 'mintlify'; From c65a7e9aa635759455e9cc357b2636f2b65c8371 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Fri, 18 Sep 2026 18:29:28 +0200 Subject: [PATCH 12/20] Sync Rules docs aren't frozen per se --- .claude/CLAUDE.md | 4 ++-- .claude/agents/document-reviewer.md | 2 +- .claude/skills/doc-author/SKILL.md | 2 +- .claude/skills/pr-to-docs/SKILL.md | 2 +- sync/advanced/multiple-client-versions.mdx | 2 +- sync/advanced/partitioned-tables.mdx | 2 +- sync/advanced/sync-data-by-time.mdx | 2 +- sync/rules/multiple-client-versions.mdx | 4 ++-- sync/rules/partitioned-tables.mdx | 4 ++-- sync/rules/prioritized-sync.mdx | 4 ++-- sync/rules/supported-sql.mdx | 4 ++-- sync/rules/sync-data-by-time.mdx | 4 ++-- sync/streams/prioritized-sync.mdx | 2 +- sync/supported-sql.mdx | 2 +- 14 files changed, 20 insertions(+), 20 deletions(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index ec4279b40..a980780d3 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -142,9 +142,9 @@ Update `docs.json` when adding, moving, or removing pages. Add redirects for mov ## Sync Streams and Sync Rules -Sync Rules are deprecated. New documentation and updates cover Sync Streams only. Many customers still use Sync Rules, so their documentation stays online but frozen. +Sync Rules are deprecated, but many customers still run them. New documentation covers Sync Streams. Keep the Sync Rules docs accurate, and add to them when that makes them more accurate or helpful, without prioritizing that work. The goal is to avoid noisy Sync Rules references outside their own section, not to stop maintaining it. -- **Sync Rules docs are frozen.** `sync/rules/` and `sync/grammar/sync-rules/` take error fixes only: no new features, examples, or pages. Each page opens with an `` callout that starts "Sync Rules are deprecated." and links to its Sync Streams version. The sidebar group stays "Sync Rules (Legacy)". +- **Sync Rules section.** `sync/rules/` and `sync/grammar/sync-rules/` hold all Sync Rules content. Each page opens with an `` callout that starts "Sync Rules are deprecated." and links to its Sync Streams version. The sidebar group stays "Sync Rules (Legacy)". - **Keep the engines apart.** Never place Sync Streams and Sync Rules content side by side: no engine tabs, no "(or legacy Sync Rules)" asides, no pointers to a Sync Rules equivalent. Outside `sync/rules/`, mention Sync Rules only to state a behavior difference that affects Sync Rules users, and remove other mentions when you edit a page. - **Shared pages.** A page that applies to both engines keeps one body in `snippets/sync-shared/.mdx`, imported by a Sync Streams wrapper at the original path and a Sync Rules wrapper at `sync/rules/.mdx` that adds the callout and `noindex: true`. Edit the snippet, not the wrappers, and keep it valid for both engines. - **Counterpart comments.** Every split twin, wrapper, and shared snippet starts with an MDX comment naming its counterpart. Read it before editing, apply a fix to both sides where content is shared, and keep the comment when restructuring. diff --git a/.claude/agents/document-reviewer.md b/.claude/agents/document-reviewer.md index 05794244e..74464fbcf 100644 --- a/.claude/agents/document-reviewer.md +++ b/.claude/agents/document-reviewer.md @@ -17,7 +17,7 @@ Apply the canonical standards in three passes and report findings from each: 1. **Accuracy:** claims, platform scope, versions, and consistency with the surrounding page. Before reporting a claim as unverified, check the sources the PR or the user cites, such as the source PR, divergence issue, release notes, or code at the merged commit, and follow the links inside them. If nothing is cited, look up the release the text names. Report a claim as unverified only when no source covers it or a source contradicts it. Give evidence, do not invent problems, and do not approve unverified claims as correct. 2. **Necessity:** list every sentence that describes what the product prints, displays, logs, or says in an error. Treat each one as a finding to remove unless it passes the restating rule in [Content Strategy](../CLAUDE.md#content-strategy), and report it even when the sentence is accurate. Describing visible output is not a mechanism, consequence, or signal. Also flag internal mechanics, rare exceptions, and repetition. Flag missing context only when readers need it, and do not require every entry to explain a mechanism, consequence, signal, action, and trade-off. 3. **Clarity and format:** plain technical English, clear actors and actions, and suitable examples and components. -4. **Sync Rules containment:** flag any Sync Rules mention, example, or tab outside `sync/rules/` and `snippets/sync-shared/` that does not state a behavior difference, and any new Sync Rules content anywhere. Check that split twins, wrappers, and shared snippets keep their counterpart comment. +4. **Sync Rules containment:** flag any Sync Rules mention, example, or tab outside `sync/rules/` and `snippets/sync-shared/` that does not state a behavior difference. Check that split twins, wrappers, and shared snippets keep their counterpart comment. ## Default Output diff --git a/.claude/skills/doc-author/SKILL.md b/.claude/skills/doc-author/SKILL.md index 84de0db5d..c2201dbb4 100644 --- a/.claude/skills/doc-author/SKILL.md +++ b/.claude/skills/doc-author/SKILL.md @@ -18,6 +18,6 @@ Use the canonical Working Process for scope changes and unresolved decisions. 1. Identify the reader, desired outcome, and affected feature or concept. 2. Research the implementation and existing coverage. Read the most relevant related pages and `docs.json`; avoid unnecessary duplication. 3. If a plan is needed, state the proposed pages, structure, and unresolved questions before drafting. -4. Write the update under the canonical standards. Cover Sync Streams only and keep Sync Rules content out of it; Sync Rules pages take error fixes only. Keep the existing structure unless the task requires a change. +4. Write the update under the canonical standards. Write new content for Sync Streams and keep Sync Rules content inside its own section. Keep the existing structure unless the task requires a change. 5. Self-review for accuracy, reader understanding, minimum useful detail, and navigation fit. Run the canonical verification checks relevant to the change. 6. Present the result and any unresolved draft TODOs, or complete the delivery workflow already authorized by the user. diff --git a/.claude/skills/pr-to-docs/SKILL.md b/.claude/skills/pr-to-docs/SKILL.md index b74ea0ed0..b8556e124 100644 --- a/.claude/skills/pr-to-docs/SKILL.md +++ b/.claude/skills/pr-to-docs/SKILL.md @@ -33,4 +33,4 @@ Ask before expanding scope, documenting a deprecation that needs migration decis ## 4. Draft and Verify -Apply the canonical writing standards, navigation requirements, and verification checks. Document new behavior for Sync Streams only; do not add Sync Rules examples or mentions. Preserve existing structure unless restructuring is part of the approved plan. Flag unresolved facts using the canonical draft-TODO convention and report what must be resolved before publication. +Apply the canonical writing standards, navigation requirements, and verification checks. Document new behavior for Sync Streams. Update Sync Rules pages when the change affects them too, and do not add Sync Rules mentions elsewhere. Preserve existing structure unless restructuring is part of the approved plan. Flag unresolved facts using the canonical draft-TODO convention and report what must be resolved before publication. diff --git a/sync/advanced/multiple-client-versions.mdx b/sync/advanced/multiple-client-versions.mdx index 4775b1b9e..4562e6b76 100644 --- a/sync/advanced/multiple-client-versions.mdx +++ b/sync/advanced/multiple-client-versions.mdx @@ -3,7 +3,7 @@ title: "Multiple Client Versions" description: "Handle multiple client app versions that require different output schemas from Sync Streams." --- -{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/multiple-client-versions.mdx. That page is deprecated and frozen, but when you fix an error here, check whether it needs the same fix. Do not mention Sync Rules on this page. */} +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/multiple-client-versions.mdx. When you change shared behavior or fix an error here, check whether that page needs the same change. Do not mention Sync Rules on this page. */} When schema changes are additive, old clients ignore the new tables and columns, and no special handling is required. More drastic changes, such as renaming tables or changing a table's structure, can break older app versions that are still in use. In these cases, define separate versions of the affected [Sync Streams](/sync/streams/overview) so that each client version receives the tables and columns it expects. diff --git a/sync/advanced/partitioned-tables.mdx b/sync/advanced/partitioned-tables.mdx index 0c90f8bb8..0e78e1e63 100644 --- a/sync/advanced/partitioned-tables.mdx +++ b/sync/advanced/partitioned-tables.mdx @@ -3,7 +3,7 @@ title: "Partitioned Tables (Postgres)" description: "Sync data from Postgres partitioned tables using wildcard table name matching." --- -{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/partitioned-tables.mdx. That page is deprecated and frozen, but when you fix an error here, check whether it needs the same fix. Do not mention Sync Rules on this page. */} +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/partitioned-tables.mdx. When you change shared behavior or fix an error here, check whether that page needs the same change. Do not mention Sync Rules on this page. */} For partitioned tables in Postgres, each individual partition is replicated and processed using [Sync Streams](/sync/streams/overview). diff --git a/sync/advanced/sync-data-by-time.mdx b/sync/advanced/sync-data-by-time.mdx index afd0b875c..b0564046e 100644 --- a/sync/advanced/sync-data-by-time.mdx +++ b/sync/advanced/sync-data-by-time.mdx @@ -4,7 +4,7 @@ description: "Filter and sync data based on time ranges using Sync Streams, with sidebarTitle: "Sync Data by Time" --- -{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/sync-data-by-time.mdx. That page is deprecated and frozen, but when you fix an error here, check whether it needs the same fix. Do not mention Sync Rules on this page. */} +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/sync-data-by-time.mdx. When you change shared behavior or fix an error here, check whether that page needs the same change. Do not mention Sync Rules on this page. */} A common need in offline-first apps is syncing data based on time, for example, only syncing issues updated in the last 7 days instead of the entire dataset. You might expect to write something like: diff --git a/sync/rules/multiple-client-versions.mdx b/sync/rules/multiple-client-versions.mdx index 0db532955..f6dd31151 100644 --- a/sync/rules/multiple-client-versions.mdx +++ b/sync/rules/multiple-client-versions.mdx @@ -4,10 +4,10 @@ sidebarTitle: "Multiple Client Versions" description: "Handle multiple client app versions that require different output schemas from legacy Sync Rules." --- -{/* Split page: the Sync Streams version of this page is sync/advanced/multiple-client-versions.mdx. This page is deprecated and frozen: no new features or examples. When you fix an error here, check whether the Sync Streams page needs the same fix. */} +{/* Split page: the Sync Streams version of this page is sync/advanced/multiple-client-versions.mdx. Sync Rules are deprecated: keep this page accurate, but do not prioritize additions. When you fix an error here, check whether the Sync Streams page needs the same fix. */} -Sync Rules are deprecated. This page is not updated with new features. For the Sync Streams version, see [Multiple Client Versions](/sync/advanced/multiple-client-versions). +Sync Rules are deprecated. For the Sync Streams version of this page, see [Multiple Client Versions](/sync/advanced/multiple-client-versions). When schema changes are additive, old clients ignore the new tables and columns, and no special handling is required. More drastic changes, such as renaming tables or changing a table's structure, can break older app versions that are still in use. In these cases, define separate versions of the affected bucket definitions so that each client version receives the tables and columns it expects. diff --git a/sync/rules/partitioned-tables.mdx b/sync/rules/partitioned-tables.mdx index 729f31b4d..c0918acfb 100644 --- a/sync/rules/partitioned-tables.mdx +++ b/sync/rules/partitioned-tables.mdx @@ -4,10 +4,10 @@ sidebarTitle: "Partitioned Tables (Postgres)" description: "Sync data from Postgres partitioned tables in legacy Sync Rules using wildcard table name matching." --- -{/* Split page: the Sync Streams version of this page is sync/advanced/partitioned-tables.mdx. This page is deprecated and frozen: no new features or examples. When you fix an error here, check whether the Sync Streams page needs the same fix. */} +{/* Split page: the Sync Streams version of this page is sync/advanced/partitioned-tables.mdx. Sync Rules are deprecated: keep this page accurate, but do not prioritize additions. When you fix an error here, check whether the Sync Streams page needs the same fix. */} -Sync Rules are deprecated. This page is not updated with new features. For the Sync Streams version, see [Partitioned Tables (Postgres)](/sync/advanced/partitioned-tables). +Sync Rules are deprecated. For the Sync Streams version of this page, see [Partitioned Tables (Postgres)](/sync/advanced/partitioned-tables). For partitioned tables in Postgres, each individual partition is replicated and processed using [Sync Rules](/sync/rules/overview). diff --git a/sync/rules/prioritized-sync.mdx b/sync/rules/prioritized-sync.mdx index e0f397f2e..52f58ff46 100644 --- a/sync/rules/prioritized-sync.mdx +++ b/sync/rules/prioritized-sync.mdx @@ -4,10 +4,10 @@ sidebarTitle: "Prioritized Sync" description: "Assign sync priorities to bucket definitions in legacy Sync Rules so that important data syncs before the rest." --- -{/* Split page: the Sync Streams version of this page is sync/streams/prioritized-sync.mdx. This page is deprecated and frozen: no new features or examples. When you fix an error here, check whether the Sync Streams page needs the same fix. */} +{/* Split page: the Sync Streams version of this page is sync/streams/prioritized-sync.mdx. Sync Rules are deprecated: keep this page accurate, but do not prioritize additions. When you fix an error here, check whether the Sync Streams page needs the same fix. */} -Sync Rules are deprecated. This page is not updated with new features. For the Sync Streams version, see [Prioritized Sync](/sync/streams/prioritized-sync). +Sync Rules are deprecated. For the Sync Streams version of this page, see [Prioritized Sync](/sync/streams/prioritized-sync). ## Overview diff --git a/sync/rules/supported-sql.mdx b/sync/rules/supported-sql.mdx index 3f98c7f1f..910971a42 100644 --- a/sync/rules/supported-sql.mdx +++ b/sync/rules/supported-sql.mdx @@ -4,10 +4,10 @@ sidebarTitle: "Supported SQL" description: "Reference for SQL syntax, operators, built-in functions, and type casting supported in legacy Sync Rules queries." --- -{/* Split page: the Sync Streams version of this page is sync/supported-sql.mdx. This page is deprecated and frozen: no new features or examples. When you fix an error here, check whether the Sync Streams page needs the same fix. */} +{/* Split page: the Sync Streams version of this page is sync/supported-sql.mdx. Sync Rules are deprecated: keep this page accurate, but do not prioritize additions. When you fix an error here, check whether the Sync Streams page needs the same fix. */} -Sync Rules are deprecated. This page is not updated with new features. For the Sync Streams version, see [Supported SQL](/sync/supported-sql). +Sync Rules are deprecated. For the Sync Streams version of this page, see [Supported SQL](/sync/supported-sql). This guide explains the SQL supported in [Sync Rules](/sync/rules/overview) parameter queries and data queries: what you can write, with examples and restrictions. diff --git a/sync/rules/sync-data-by-time.mdx b/sync/rules/sync-data-by-time.mdx index 3860f06ab..d8b32a7e2 100644 --- a/sync/rules/sync-data-by-time.mdx +++ b/sync/rules/sync-data-by-time.mdx @@ -4,10 +4,10 @@ sidebarTitle: "Sync Data by Time" description: "Filter and sync data based on time ranges using legacy Sync Rules, with patterns for recent-only and sliding-window queries." --- -{/* Split page: the Sync Streams version of this page is sync/advanced/sync-data-by-time.mdx. This page is deprecated and frozen: no new features or examples. When you fix an error here, check whether the Sync Streams page needs the same fix. */} +{/* Split page: the Sync Streams version of this page is sync/advanced/sync-data-by-time.mdx. Sync Rules are deprecated: keep this page accurate, but do not prioritize additions. When you fix an error here, check whether the Sync Streams page needs the same fix. */} -Sync Rules are deprecated. This page is not updated with new features. For the Sync Streams version, see [Sync Data by Time](/sync/advanced/sync-data-by-time). +Sync Rules are deprecated. For the Sync Streams version of this page, see [Sync Data by Time](/sync/advanced/sync-data-by-time). A common need is syncing data based on time, for example, only syncing issues updated in the last 7 days instead of the entire dataset. You might expect to write something like: diff --git a/sync/streams/prioritized-sync.mdx b/sync/streams/prioritized-sync.mdx index 15e73407b..fa25a5ccc 100644 --- a/sync/streams/prioritized-sync.mdx +++ b/sync/streams/prioritized-sync.mdx @@ -3,7 +3,7 @@ title: "Prioritized Sync" description: "Prioritize which tables sync first so users can start working immediately while remaining data continues loading in the background." --- -{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/prioritized-sync.mdx. That page is deprecated and frozen, but when you fix an error here, check whether it needs the same fix. Do not mention Sync Rules on this page. */} +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/prioritized-sync.mdx. When you change shared behavior or fix an error here, check whether that page needs the same change. Do not mention Sync Rules on this page. */} ## Overview diff --git a/sync/supported-sql.mdx b/sync/supported-sql.mdx index 550e76ab6..b246d733f 100644 --- a/sync/supported-sql.mdx +++ b/sync/supported-sql.mdx @@ -3,7 +3,7 @@ title: "Supported SQL" description: "Reference for SQL syntax, operators, built-in functions, and type casting supported in Sync Streams queries." --- -{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/supported-sql.mdx. That page is deprecated and frozen, but when you fix an error here, check whether it needs the same fix. Do not mention Sync Rules on this page. */} +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/supported-sql.mdx. When you change shared behavior or fix an error here, check whether that page needs the same change. Do not mention Sync Rules on this page. */} This guide explains the SQL supported in [Sync Streams](/sync/streams/overview): what you can write, with examples and restrictions. From ef173d5778111a8ac911fff2e65724d1a568020c Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Mon, 21 Sep 2026 14:05:02 +0200 Subject: [PATCH 13/20] Remove compability page from sync rules --- docs.json | 1 - snippets/sync-shared/compatibility.mdx | 217 ------------------------- sync/advanced/compatibility.mdx | 216 +++++++++++++++++++++++- sync/rules/compatibility.mdx | 15 -- 4 files changed, 213 insertions(+), 236 deletions(-) delete mode 100644 snippets/sync-shared/compatibility.mdx delete mode 100644 sync/rules/compatibility.mdx diff --git a/docs.json b/docs.json index 7d5d58fa2..8fe96b689 100644 --- a/docs.json +++ b/docs.json @@ -250,7 +250,6 @@ "sync/rules/prioritized-sync", "sync/rules/client-id", "sync/rules/case-sensitivity", - "sync/rules/compatibility", "sync/rules/storage-version-4", "sync/rules/sync-data-by-time", "sync/rules/schemas-and-connections", diff --git a/snippets/sync-shared/compatibility.mdx b/snippets/sync-shared/compatibility.mdx deleted file mode 100644 index 6642437f1..000000000 --- a/snippets/sync-shared/compatibility.mdx +++ /dev/null @@ -1,217 +0,0 @@ -{/* Shared body: rendered by sync/advanced/compatibility.mdx (Sync Streams section) and sync/rules/compatibility.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} - -To ensure consistency, it is important that the PowerSync Service does not interpret the same source row in different ways after updating to a new version. -At the same time, we want to fix bugs or other inaccuracies that have accumulated during the development of the Service. - -## Overview - -To make this trade‑off explicit, you choose whether to keep the existing behavior or turn on newer fixes that slightly change how data is processed. - -Use the `config` block in your Sync Config YAML to choose the behavior. There are two ways to turn fixes on: - -1. Set an `edition` to enable the full set of fixes for that edition. This is the recommended approach for new projects. -2. Toggle individual options for more fine‑grained control. - -For older projects, the previous behavior remains the default. New projects should enable all current fixes. - -### Configuration - -For new projects, it is recommended to enable all current fixes by setting `edition: `: - -```yaml -config: - edition: 3 # Recommended to set to the latest available edition (see 'Supported fixes' table below) - -streams: - # ... -``` - -Or, specify options individually: - -```yaml -config: - timestamps_iso8601: true - versioned_bucket_ids: true - fixed_json_extract: true - custom_postgres_types: true -``` - -## Sync Streams Requirement - -**New Sync Streams configurations should use `edition: 3`**, which enables the new compiler with an expanded SQL feature set (including `JOIN`, CTEs, multiple queries per stream, `BETWEEN`, `CASE`, and more): - -```yaml -config: - edition: 3 - -streams: - my_stream: - query: SELECT * FROM my_table WHERE user_id = auth.user_id() -``` - - -**Upgrading from alpha**: If you have existing Sync Streams using `edition: 2`, upgrade to `edition: 3` to enable the new compiler with an expanded SQL feature set (including `JOIN`, CTEs, multiple queries per stream, `BETWEEN`, `CASE`, and more). See [Supported SQL](/sync/supported-sql) for the full list of supported features. - - -## Storage Version - -A storage version tells the PowerSync Service how to organize prepared sync data in the [bucket storage database](/architecture/powersync-service#bucket-storage). - -Changing the version does not rewrite the current data in place. When you next deploy the Sync Config, PowerSync prepares a new copy using the selected version. Clients continue using the current copy until the new one is ready. This avoids taking the instance offline for a bucket storage migration. - -### Optional `config.storage_version` - -You can choose the bucket storage version in the `config` block: - -```yaml -config: - edition: 3 - storage_version: 4 - -streams: - todos: - query: SELECT * FROM todos WHERE owner_id = auth.user_id() -``` - -When you omit `storage_version`, the PowerSync Service uses its default, which is version 2 in v1.26.0. On PowerSync Cloud, PowerSync manages the default. For self-hosted deployments, set `config.storage_version` explicitly to select a different version. - -Set `storage_version` when you need to: - -- Use [storage version 4](/sync/advanced/storage-version-4), which is in Beta and enables incremental reprocessing and S3 object storage. -- Delay a storage upgrade. When the default moves to a newer version, pin `storage_version` to the version your data already uses. This keeps later Sync Config deployments on that format. Remove the pin when you are ready for the new format. -- Prepare for a Service downgrade. Select a version supported by the older Service, deploy the Sync Config, and wait for the new copy to finish before downgrading. - -### Available Versions - -All PowerSync Cloud instances use MongoDB bucket storage, so they are compatible with all available storage versions. Self-hosted instances with Postgres bucket storage can use versions 1 and 2 only. - -| Version | Bucket storage | Status | -| --- | --- | --- | -| `1` | MongoDB or Postgres | Legacy format, retained for existing deployments. | -| `2` | MongoDB or Postgres | Stable. The default in v1.26.0. | -| `3` | MongoDB | Experimental. The unstable predecessor of version 4, with the same format. Do not use it in production. Deploy with version 4 instead. | -| `4` | MongoDB | Stable. Enables [incremental reprocessing and S3 object storage](/sync/advanced/storage-version-4) (Beta). | - -Version numbers follow a pattern. Even numbers are stable formats: they stay backwards compatible and later Service versions continue to support them. Stable makes no guarantee that a format is bug-free. Odd numbers are experimental formats: their layout can change without notice and support can be removed in a later release, so use them only for testing, never in production. - -## Supported Fixes - -This table lists all fixes currently supported: - -| Name | Explanation | Added in Service version | Fixed in edition | -|----------------------------|------------------------------------|--------------|------------------| -| `timestamps_iso8601` | [Link](#timestamps_iso8601) | 1.15.0 | 2 | -| `versioned_bucket_ids` | [Link](#versioned_bucket_ids) | 1.15.0 | 2 | -| `fixed_json_extract` | [Link](#fixed_json_extract) | 1.15.0 | 2 | -| `custom_postgres_types` | [Link](#custom_postgres_types) | 1.15.3 | 2 | -| `unstable_sqlite_expression_engine` | [Link](#unstable_sqlite_expression_engine). | 1.22.0 | None (unstable) | - -### `timestamps_iso8601` - -PowerSync is supposed to encode timestamps according to the ISO-8601 standard. -Without this fix, the service encoded timestamps from MongoDB and Postgres source databases incorrectly. -To ensure time values from Postgres compare lexicographically, they're also padded to six digits of accuracy when encoded. -Since MongoDB only stores values with an accuracy of milliseconds, only three digits of accuracy are used. - -For instance, the value `2025-09-22T14:29:30` would be encoded as follows: - -- For Postgres: `2025-09-22 14:29:30` without the fix, `2025-09-22T14:29:30.000000` with the fix applied. -- For MongoDB: `2025-09-22 14:29:30.000` without the fix, `2025-09-22T14:29:30.000` with the fix applied. - -Note that MySQL has never been affected by this issue, and thus behaves the same regardless of the option used. - -#### Configurable Sub-Second Datetime Precision - -When the `timestamps_iso8601` option is enabled, PowerSync will sync date and time values with a higher -precision depending on the source database. -You can use the `timestamp_max_precision` option to configure the actual precision to use. -For instance, a Postgres timestamp value would sync as `2025-09-22T14:29:30.000000` by default. -If you don't want that level of precision, you can use the following options to make it sync as `2025-09-22T14:29:30.000`: - -```yaml sync-config.yaml -config: - edition: 3 - timestamp_max_precision: milliseconds -``` - -Valid options for `timestamp_max_precision` are `seconds`, `milliseconds`, `microseconds` and `nanoseconds`. When an explicit -value is given, all synced time values will use that precision. -If a source value has a higher precision, it will be truncated (it is not rounded). -If a source value has a lower precision, it will be padded (so setting the option to `microseconds` with a MongoDB source database -will sync values as `2025-09-22T14:29:30.123000`, with the last three sub-second digits always being set to zero). - -If no option is given, the default precision depends on the source database: - -| Source database | Default precision | Max precision | Notes | -|-----------------|-------------------|---------------|---------------------------------------------------------------------------------------------------------| -| MongoDB | Milliseconds | Milliseconds | | -| Postgres | Microseconds | Microseconds | | -| MySQL | Milliseconds | Microseconds | Defaults to milliseconds, but can be expanded with the option. | -| SQL Server | Nanoseconds | Nanoseconds | SQL Server supports 7 digits of accuracy, the sync service pads values to always use 9 for nanoseconds. | - -### `versioned_bucket_ids` - -Sync Rules define buckets, which rows to sync are then assigned to. When you run a full defragmentation or -redeploy Sync Rules, the same bucket identifiers are re-used when processing data again. - -Because the second iteration uses different checksums for the same bucket ids, clients may sync data -twice before realizing that something is off and starting from scratch. - -Applying this fix improves client-side progress estimation and is more efficient, since data would not get -downloaded twice. - -For how bucket identifiers are represented in bucket storage at the persistence layer (including automatic use of versioned bucket names with newer storage formats), see [Storage version](#storage-version). - -### `fixed_json_extract` - -This fixes the `json_extract` functions as well as the `->` and `->>` operators in Sync Rules to behave similar -to recent SQLite versions: We only split on `.` if the path starts with `$.`. - -For instance, `'json_extract({"foo.bar": "baz"}', 'foo.bar')` would evaluate to: - -1. `baz` with the option enabled. -2. `null` with the option disabled. - -### `custom_postgres_types` - -If you have custom Postgres types in your backend source database schema, older versions of the PowerSync Service -would not recognize these values and sync them with the textual wire representation used by Postgres. -This is especially noticeable when defining `DOMAIN` types with e.g. a `REAL` inner type: The wrapped -`DOMAIN` type should get synced as a real value as well, but it would actually get synced as a string. - -With this fix applied: - -- `DOMAIN TYPE`s are synced as their inner type. -- Array types of custom types get parsed correctly, and sync as a JSON array. -- Custom types get parsed and synced as a JSON object containing their members. -- Ranges sync as a JSON object corresponding to the following TypeScript definition: - ```TypeScript - export type Range = - | { - lower: T | null; - upper: T | null; - lower_exclusive: boolean; - upper_exclusive: boolean; - } - | 'empty'; - ``` -- Multi-ranges sync as an array of ranges. - -### `unstable_sqlite_expression_engine` - - -This option is experimental: When enabled, updates to the PowerSync Service might change how rows are processed -and this option may be removed in a future version of the Service. - - -Sync Streams support scalar SQL operators (like `+`, `-` and `||`) and [functions](/sync/supported-sql#functions). -SQL in Sync Streams should behave exactly as it would in SQLite, but the Service uses a custom implementation which differs -from SQLite for some edge cases. - -To perfectly align the behavior of the Service and SQLite, enabling this option makes the Service use an actual -SQLite database to evaluate Sync Streams. -Some known issues with the JavaScript evaluator that are fixed by this option are: - -- Exact null handling: `NOT NULL` evaluates to `TRUE` without this option, enabling it yields `NULL`. -- Without this option, `substr()` and `length()` operate on UTF-16 code units. Enabling it makes them operate on - Unicode code points. diff --git a/sync/advanced/compatibility.mdx b/sync/advanced/compatibility.mdx index 286263be8..44cc73686 100644 --- a/sync/advanced/compatibility.mdx +++ b/sync/advanced/compatibility.mdx @@ -3,8 +3,218 @@ title: "Compatibility" description: "Configure compatibility editions and bucket storage format version in PowerSync's Sync Config." --- -{/* Wrapper page: the content is snippets/sync-shared/compatibility.mdx, which also renders at sync/rules/compatibility.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} +To ensure consistency, it is important that the PowerSync Service does not interpret the same source row in different ways after updating to a new version. +At the same time, we want to fix bugs or other inaccuracies that have accumulated during the development of the Service. -import Compatibility from '/snippets/sync-shared/compatibility.mdx'; +## Overview - +To make this trade‑off explicit, you choose whether to keep the existing behavior or turn on newer fixes that slightly change how data is processed. + +Use the `config` block in your Sync Config YAML to choose the behavior. There are two ways to turn fixes on: + +1. Set an `edition` to enable the full set of fixes for that edition. This is the recommended approach for new projects. +2. Toggle individual options for more fine‑grained control. + +For older projects, the previous behavior remains the default. New projects should enable all current fixes. + +### Configuration + +For new projects, it is recommended to enable all current fixes by setting `edition: `: + +```yaml +config: + edition: 3 # Recommended to set to the latest available edition (see 'Supported fixes' table below) + +streams: + # ... +``` + +Or, specify options individually: + +```yaml +config: + timestamps_iso8601: true + versioned_bucket_ids: true + fixed_json_extract: true + custom_postgres_types: true +``` + +## Sync Streams Requirement + +**New Sync Streams configurations should use `edition: 3`**, which enables the new compiler with an expanded SQL feature set (including `JOIN`, CTEs, multiple queries per stream, `BETWEEN`, `CASE`, and more): + +```yaml +config: + edition: 3 + +streams: + my_stream: + query: SELECT * FROM my_table WHERE user_id = auth.user_id() +``` + + +**Upgrading from alpha**: If you have existing Sync Streams using `edition: 2`, upgrade to `edition: 3` to enable the new compiler with an expanded SQL feature set (including `JOIN`, CTEs, multiple queries per stream, `BETWEEN`, `CASE`, and more). See [Supported SQL](/sync/supported-sql) for the full list of supported features. + + +## Storage Version + +A storage version tells the PowerSync Service how to organize prepared sync data in the [bucket storage database](/architecture/powersync-service#bucket-storage). + +Changing the version does not rewrite the current data in place. When you next deploy the Sync Config, PowerSync prepares a new copy using the selected version. Clients continue using the current copy until the new one is ready. This avoids taking the instance offline for a bucket storage migration. + +### Optional `config.storage_version` + +You can choose the bucket storage version in the `config` block: + +```yaml +config: + edition: 3 + storage_version: 4 + +streams: + todos: + query: SELECT * FROM todos WHERE owner_id = auth.user_id() +``` + +When you omit `storage_version`, the PowerSync Service uses its default, which is version 2 in v1.26.0. On PowerSync Cloud, PowerSync manages the default. For self-hosted deployments, set `config.storage_version` explicitly to select a different version. + +Set `storage_version` when you need to: + +- Use [storage version 4](/sync/advanced/storage-version-4), which is in Beta and enables incremental reprocessing and S3 object storage. +- Delay a storage upgrade. When the default moves to a newer version, pin `storage_version` to the version your data already uses. This keeps later Sync Config deployments on that format. Remove the pin when you are ready for the new format. +- Prepare for a Service downgrade. Select a version supported by the older Service, deploy the Sync Config, and wait for the new copy to finish before downgrading. + +### Available Versions + +All PowerSync Cloud instances use MongoDB bucket storage, so they are compatible with all available storage versions. Self-hosted instances with Postgres bucket storage can use versions 1 and 2 only. + +| Version | Bucket storage | Status | +| --- | --- | --- | +| `1` | MongoDB or Postgres | Legacy format, retained for existing deployments. | +| `2` | MongoDB or Postgres | Stable. The default in v1.26.0. | +| `3` | MongoDB | Experimental. The unstable predecessor of version 4, with the same format. Do not use it in production. Deploy with version 4 instead. | +| `4` | MongoDB | Stable. Enables [incremental reprocessing and S3 object storage](/sync/advanced/storage-version-4) (Beta). | + +Version numbers follow a pattern. Even numbers are stable formats: they stay backwards compatible and later Service versions continue to support them. Stable makes no guarantee that a format is bug-free. Odd numbers are experimental formats: their layout can change without notice and support can be removed in a later release, so use them only for testing, never in production. + +## Supported Fixes + +This table lists all fixes currently supported: + +| Name | Explanation | Added in Service version | Fixed in edition | +|----------------------------|------------------------------------|--------------|------------------| +| `timestamps_iso8601` | [Link](#timestamps_iso8601) | 1.15.0 | 2 | +| `versioned_bucket_ids` | [Link](#versioned_bucket_ids) | 1.15.0 | 2 | +| `fixed_json_extract` | [Link](#fixed_json_extract) | 1.15.0 | 2 | +| `custom_postgres_types` | [Link](#custom_postgres_types) | 1.15.3 | 2 | +| `unstable_sqlite_expression_engine` | [Link](#unstable_sqlite_expression_engine). | 1.22.0 | None (unstable) | + +### `timestamps_iso8601` + +PowerSync is supposed to encode timestamps according to the ISO-8601 standard. +Without this fix, the service encoded timestamps from MongoDB and Postgres source databases incorrectly. +To ensure time values from Postgres compare lexicographically, they're also padded to six digits of accuracy when encoded. +Since MongoDB only stores values with an accuracy of milliseconds, only three digits of accuracy are used. + +For instance, the value `2025-09-22T14:29:30` would be encoded as follows: + +- For Postgres: `2025-09-22 14:29:30` without the fix, `2025-09-22T14:29:30.000000` with the fix applied. +- For MongoDB: `2025-09-22 14:29:30.000` without the fix, `2025-09-22T14:29:30.000` with the fix applied. + +Note that MySQL has never been affected by this issue, and thus behaves the same regardless of the option used. + +#### Configurable Sub-Second Datetime Precision + +When the `timestamps_iso8601` option is enabled, PowerSync will sync date and time values with a higher +precision depending on the source database. +You can use the `timestamp_max_precision` option to configure the actual precision to use. +For instance, a Postgres timestamp value would sync as `2025-09-22T14:29:30.000000` by default. +If you don't want that level of precision, you can use the following options to make it sync as `2025-09-22T14:29:30.000`: + +```yaml sync-config.yaml +config: + edition: 3 + timestamp_max_precision: milliseconds +``` + +Valid options for `timestamp_max_precision` are `seconds`, `milliseconds`, `microseconds` and `nanoseconds`. When an explicit +value is given, all synced time values will use that precision. +If a source value has a higher precision, it will be truncated (it is not rounded). +If a source value has a lower precision, it will be padded (so setting the option to `microseconds` with a MongoDB source database +will sync values as `2025-09-22T14:29:30.123000`, with the last three sub-second digits always being set to zero). + +If no option is given, the default precision depends on the source database: + +| Source database | Default precision | Max precision | Notes | +|-----------------|-------------------|---------------|---------------------------------------------------------------------------------------------------------| +| MongoDB | Milliseconds | Milliseconds | | +| Postgres | Microseconds | Microseconds | | +| MySQL | Milliseconds | Microseconds | Defaults to milliseconds, but can be expanded with the option. | +| SQL Server | Nanoseconds | Nanoseconds | SQL Server supports 7 digits of accuracy, the sync service pads values to always use 9 for nanoseconds. | + +### `versioned_bucket_ids` + +Streams are compiled into buckets, and rows to sync are assigned to those buckets. When you run a full defragmentation or +redeploy your Sync Config, the same bucket identifiers are re-used when processing data again. + +Because the second iteration uses different checksums for the same bucket ids, clients may sync data +twice before realizing that something is off and starting from scratch. + +Applying this fix improves client-side progress estimation and is more efficient, since data would not get +downloaded twice. + +For how bucket identifiers are represented in bucket storage at the persistence layer (including automatic use of versioned bucket names with newer storage formats), see [Storage version](#storage-version). + +### `fixed_json_extract` + +This fixes the `json_extract` functions as well as the `->` and `->>` operators in stream queries to behave similar +to recent SQLite versions: We only split on `.` if the path starts with `$.`. + +For instance, `'json_extract({"foo.bar": "baz"}', 'foo.bar')` would evaluate to: + +1. `baz` with the option enabled. +2. `null` with the option disabled. + +### `custom_postgres_types` + +If you have custom Postgres types in your backend source database schema, older versions of the PowerSync Service +would not recognize these values and sync them with the textual wire representation used by Postgres. +This is especially noticeable when defining `DOMAIN` types with e.g. a `REAL` inner type: The wrapped +`DOMAIN` type should get synced as a real value as well, but it would actually get synced as a string. + +With this fix applied: + +- `DOMAIN TYPE`s are synced as their inner type. +- Array types of custom types get parsed correctly, and sync as a JSON array. +- Custom types get parsed and synced as a JSON object containing their members. +- Ranges sync as a JSON object corresponding to the following TypeScript definition: + ```TypeScript + export type Range = + | { + lower: T | null; + upper: T | null; + lower_exclusive: boolean; + upper_exclusive: boolean; + } + | 'empty'; + ``` +- Multi-ranges sync as an array of ranges. + +### `unstable_sqlite_expression_engine` + + +This option is experimental: When enabled, updates to the PowerSync Service might change how rows are processed +and this option may be removed in a future version of the Service. + + +Sync Streams support scalar SQL operators (like `+`, `-` and `||`) and [functions](/sync/supported-sql#functions). +SQL in Sync Streams should behave exactly as it would in SQLite, but the Service uses a custom implementation which differs +from SQLite for some edge cases. + +To perfectly align the behavior of the Service and SQLite, enabling this option makes the Service use an actual +SQLite database to evaluate Sync Streams. +Some known issues with the JavaScript evaluator that are fixed by this option are: + +- Exact null handling: `NOT NULL` evaluates to `TRUE` without this option, enabling it yields `NULL`. +- Without this option, `substr()` and `length()` operate on UTF-16 code units. Enabling it makes them operate on + Unicode code points. diff --git a/sync/rules/compatibility.mdx b/sync/rules/compatibility.mdx deleted file mode 100644 index b9fa6991e..000000000 --- a/sync/rules/compatibility.mdx +++ /dev/null @@ -1,15 +0,0 @@ ---- -title: "Compatibility" -description: "Configure compatibility editions and bucket storage format version in PowerSync's Sync Config." -noindex: true ---- - -{/* Wrapper page: the content is snippets/sync-shared/compatibility.mdx, which also renders at sync/advanced/compatibility.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} - -import Compatibility from '/snippets/sync-shared/compatibility.mdx'; - - -Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. Configuration examples use Sync Streams syntax. - - - From 6bf68f5c9443a9de0bfbd4c8506283514b62f19a Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Mon, 21 Sep 2026 14:05:21 +0200 Subject: [PATCH 14/20] More generalized wording for shared pages --- snippets/sync-shared/case-sensitivity.mdx | 10 +++++----- snippets/sync-shared/client-id.mdx | 4 ++-- snippets/sync-shared/schemas-and-connections.mdx | 2 +- snippets/sync-shared/sharded-databases.mdx | 4 ++-- snippets/sync-shared/types.mdx | 2 +- 5 files changed, 11 insertions(+), 11 deletions(-) diff --git a/snippets/sync-shared/case-sensitivity.mdx b/snippets/sync-shared/case-sensitivity.mdx index da99add96..db88d82d3 100644 --- a/snippets/sync-shared/case-sensitivity.mdx +++ b/snippets/sync-shared/case-sensitivity.mdx @@ -1,18 +1,18 @@ {/* Shared body: rendered by sync/advanced/case-sensitivity.mdx (Sync Streams section) and sync/rules/case-sensitivity.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} -### Case in Sync Rules +### Case in Sync Queries -PowerSync converts all table/collection and column/field names to lower-case by default in Sync Rule queries (this is how Postgres also behaves). To preserve the case, surround the names with double quotes, for example: +PowerSync converts all table/collection and column/field names to lower-case by default in sync queries (this is how Postgres also behaves). To preserve the case, surround the names with double quotes, for example: ```sql -SELECT "ID" as id, "Description", "ListID" FROM "TODOs" WHERE "TODOs"."ListID" = bucket.list_id +SELECT "ID" as id, "Description", "ListID" FROM "TODOs" ``` When using `SELECT *`, the original case is preserved for the returned columns/fields. ### Client-Side Case -On the client side, the case of table and column names in the [client-side schema](/intro/setup-guide#define-your-client-side-schema) must match the case produced by Sync Rules exactly. For the above example, use the following in Dart: +On the client side, the case of table and column names in the [client-side schema](/intro/setup-guide#define-your-client-side-schema) must match the case produced by the sync query exactly. For the above example, use the following in Dart: ```dart Table('TODOs', [ @@ -25,7 +25,7 @@ SQLite itself is case-insensitive. When querying and modifying the data on the c Operations (`PUT`/`PATCH`/`DELETE`) are stored in the upload queue using the case as defined in the schema above for table and column names, not the case used in queries. -As another example, in this Sync Rule query: +As another example, in this sync query: ```sql SELECT ID, todo_description as Description FROM todo_items as TODOs diff --git a/snippets/sync-shared/client-id.mdx b/snippets/sync-shared/client-id.mdx index f6ec13cd5..7e88433b7 100644 --- a/snippets/sync-shared/client-id.mdx +++ b/snippets/sync-shared/client-id.mdx @@ -5,14 +5,14 @@ For tables where the client will create new rows: - Postgres, MySQL and SQL Server: use a UUID for `id`. Use the `uuid()` helper to generate a random UUID (v4) on the client. - MongoDB: use an `ObjectId` for `_id`. Generate an `ObjectId()` in your app code and store it in the client's `id` column as a string; this will map to MongoDB's `_id`. -To use a different column/field from the server-side database as the record ID on the client, use a column/field alias in your [Sync Streams](/sync/streams/overview) query (or [Sync Rules](/sync/rules/overview) data query): +To use a different column/field from the server-side database as the record ID on the client, use a column/field alias in your sync query: ```sql SELECT client_id as id FROM my_data ``` - MongoDB uses `_id` as the name of the ID field in collections. You must use `SELECT _id as id` (and include any other columns you need) in [Sync Streams](/sync/streams/overview) queries and [Sync Rules](/sync/rules/overview) data queries when using MongoDB as the backend source database. When inserting new documents from the client, prefer `ObjectId` values for `_id` (stored in the client's `id` column). + MongoDB uses `_id` as the name of the ID field in collections. You must use `SELECT _id as id` (and include any other columns you need) in your sync queries when using MongoDB as the backend source database. When inserting new documents from the client, prefer `ObjectId` values for `_id` (stored in the client's `id` column). Custom transformations can also be used for the ID column. This is useful in certain scenarios for example when dealing with join tables, because PowerSync doesn't currently support composite primary keys. For example: diff --git a/snippets/sync-shared/schemas-and-connections.mdx b/snippets/sync-shared/schemas-and-connections.mdx index f922962c8..592125faf 100644 --- a/snippets/sync-shared/schemas-and-connections.mdx +++ b/snippets/sync-shared/schemas-and-connections.mdx @@ -50,7 +50,7 @@ In the future, it will be possible to configure PowerSync with multiple separate You should not add multiple connections to multiple replicas of the same database — this would cause data duplication. Only use this when the data on each connection does not overlap. -It will be possible for each connection to be configured with a "tag", to distinguish these connections in Sync Rules. The same tag may be used for multiple connections (if the schema is the same in each). +It will be possible for each connection to be configured with a "tag", to distinguish these connections in sync queries. The same tag may be used for multiple connections (if the schema is the same in each). By default, queries will reference the "default" tag. To use a different connection or connections, assign a different tag, and specify it in the query as a schema prefix. In this case, the schema itself must also be specified. diff --git a/snippets/sync-shared/sharded-databases.mdx b/snippets/sync-shared/sharded-databases.mdx index 9ea54cd92..4ee959b44 100644 --- a/snippets/sync-shared/sharded-databases.mdx +++ b/snippets/sync-shared/sharded-databases.mdx @@ -21,7 +21,7 @@ Some specific scenarios: This is common when separate "services" use separate databases, but multiple tables across those databases need to be synced to the same users. -Use a single PowerSync Service instance, with a separate connection for each source database ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release). Use a unique [connection tag](/sync/advanced/schemas-and-connections) for each source database, allowing them to be distinguished in your [Sync Streams](/sync/streams/overview) or [Sync Rules](/sync/rules/overview). +Use a single PowerSync Service instance, with a separate connection for each source database ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release). Use a unique [connection tag](/sync/advanced/schemas-and-connections) for each source database, allowing them to be distinguished in your sync queries. #### 2a. All Data for a Single Customer Is Contained in a Single Shard @@ -41,4 +41,4 @@ This is more complicated than the other cases listed above. Please [reach out to In some cases, most tables would be on a shared server, with only a few large tables being sharded. -For this case, use a single PowerSync Service instance. Add each shard as a new connection on this instance ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release) — all with the same connection tag, so that the same [Sync Streams](/sync/streams/overview) or [Sync Rules](/sync/rules/overview) applies to each. +For this case, use a single PowerSync Service instance. Add each shard as a new connection on this instance ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release) — all with the same connection tag, so that the same Sync Config applies to each. diff --git a/snippets/sync-shared/types.mdx b/snippets/sync-shared/types.mdx index e3ea89acc..51982e582 100644 --- a/snippets/sync-shared/types.mdx +++ b/snippets/sync-shared/types.mdx @@ -96,7 +96,7 @@ MongoDB types are mapped to SQLite types as follows: * Data is converted to a flat list of columns, one column per top-level field in the MongoDB document. * Special BSON types are converted to plain SQLite alternatives. For example, `ObjectId`, `Date`, `UUID` are all converted to a plain `TEXT` column. -* Nested objects and arrays are converted to JSON, and [JSON functions and operators](/sync/supported-sql#operators) can be used to query them (in the Sync Streams / Sync Rules and/or on the client-side SQLite statements). +* Nested objects and arrays are converted to JSON, and [JSON functions and operators](/sync/supported-sql#operators) can be used to query them (in sync queries or in client-side SQLite statements). * Binary data nested in objects or arrays is not supported. From 3caf10b1cd1d1744bc21dc2c201f6c0d54581ef1 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Mon, 21 Sep 2026 15:24:53 +0200 Subject: [PATCH 15/20] Minor fixed based on Claude's review --- .claude/CLAUDE.md | 2 +- snippets/binary-type.mdx | 2 +- snippets/sync-shared/case-sensitivity.mdx | 4 ++-- snippets/sync-shared/sharded-databases.mdx | 2 +- sync/advanced/case-sensitivity.mdx | 2 +- sync/advanced/schemas-and-connections.mdx | 2 +- sync/grammar/sync-rules/index.mdx | 4 ++++ sync/rules/case-sensitivity.mdx | 2 +- sync/rules/client-parameters.mdx | 4 ++++ sync/rules/data-queries.mdx | 4 ++++ sync/rules/global-buckets.mdx | 4 ++++ sync/rules/many-to-many-join-tables.mdx | 8 ++++---- sync/rules/organize-data-into-buckets.mdx | 4 ++++ sync/rules/overview.mdx | 4 ++-- sync/rules/parameter-queries.mdx | 4 ++++ sync/rules/schemas-and-connections.mdx | 2 +- sync/rules/supported-sql.mdx | 4 +++- sync/streams/prioritized-sync.mdx | 2 +- 18 files changed, 43 insertions(+), 17 deletions(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index a980780d3..d28b039d5 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -144,7 +144,7 @@ Update `docs.json` when adding, moving, or removing pages. Add redirects for mov Sync Rules are deprecated, but many customers still run them. New documentation covers Sync Streams. Keep the Sync Rules docs accurate, and add to them when that makes them more accurate or helpful, without prioritizing that work. The goal is to avoid noisy Sync Rules references outside their own section, not to stop maintaining it. -- **Sync Rules section.** `sync/rules/` and `sync/grammar/sync-rules/` hold all Sync Rules content. Each page opens with an `` callout that starts "Sync Rules are deprecated." and links to its Sync Streams version. The sidebar group stays "Sync Rules (Legacy)". +- **Sync Rules section.** `sync/rules/` and `sync/grammar/sync-rules/` hold all Sync Rules content. Each page opens with an `` callout that starts "Sync Rules are deprecated." A split twin or concept page then links to its Sync Streams equivalent. A shared-snippet wrapper instead states that the page applies to both Sync Streams and Sync Rules unless a section says otherwise. The overview carries the full deprecation notice in a ``. The sidebar group stays "Sync Rules (Legacy)". - **Keep the engines apart.** Never place Sync Streams and Sync Rules content side by side: no engine tabs, no "(or legacy Sync Rules)" asides, no pointers to a Sync Rules equivalent. Outside `sync/rules/`, mention Sync Rules only to state a behavior difference that affects Sync Rules users, and remove other mentions when you edit a page. - **Shared pages.** A page that applies to both engines keeps one body in `snippets/sync-shared/.mdx`, imported by a Sync Streams wrapper at the original path and a Sync Rules wrapper at `sync/rules/.mdx` that adds the callout and `noindex: true`. Edit the snippet, not the wrappers, and keep it valid for both engines. - **Counterpart comments.** Every split twin, wrapper, and shared snippet starts with an MDX comment naming its counterpart. Read it before editing, apply a fix to both sides where content is shared, and keep the comment when restructuring. diff --git a/snippets/binary-type.mdx b/snippets/binary-type.mdx index 530cededc..6a999b8a9 100644 --- a/snippets/binary-type.mdx +++ b/snippets/binary-type.mdx @@ -1,3 +1,3 @@ - Binary data can be accessed in Sync Streams (or legacy Sync Rules), but cannot be used as a parameter (see [Sync Streams parameters](/sync/streams/parameters) or [Sync Rules client parameters](/sync/rules/client-parameters)). To sync binary columns/fields to clients, those columns need to be converted to hex or base64 representation using the relevant [functions](/sync/supported-sql#functions). + Binary data can be selected in sync queries, but cannot be used as a parameter. To sync binary columns/fields to clients, convert them to a hex or base64 representation using the relevant [functions](/sync/supported-sql#functions). \ No newline at end of file diff --git a/snippets/sync-shared/case-sensitivity.mdx b/snippets/sync-shared/case-sensitivity.mdx index db88d82d3..f558c208c 100644 --- a/snippets/sync-shared/case-sensitivity.mdx +++ b/snippets/sync-shared/case-sensitivity.mdx @@ -1,6 +1,6 @@ {/* Shared body: rendered by sync/advanced/case-sensitivity.mdx (Sync Streams section) and sync/rules/case-sensitivity.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} -### Case in Sync Queries +## Case in Sync Queries PowerSync converts all table/collection and column/field names to lower-case by default in sync queries (this is how Postgres also behaves). To preserve the case, surround the names with double quotes, for example: @@ -10,7 +10,7 @@ SELECT "ID" as id, "Description", "ListID" FROM "TODOs" When using `SELECT *`, the original case is preserved for the returned columns/fields. -### Client-Side Case +## Client-Side Case On the client side, the case of table and column names in the [client-side schema](/intro/setup-guide#define-your-client-side-schema) must match the case produced by the sync query exactly. For the above example, use the following in Dart: diff --git a/snippets/sync-shared/sharded-databases.mdx b/snippets/sync-shared/sharded-databases.mdx index 4ee959b44..77a0d5536 100644 --- a/snippets/sync-shared/sharded-databases.mdx +++ b/snippets/sync-shared/sharded-databases.mdx @@ -13,7 +13,7 @@ The primary options are: 1. Use a separate PowerSync Service instance per database. 2. Add a connection for each database in the same PowerSync Service instance ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release). -Where feasible, using separate PowerSync Service instances would give better performance and give more control over how changes are rolled out, especially around Sync Rule reprocessing. +Where feasible, using separate PowerSync Service instances would give better performance and give more control over how changes are rolled out, especially around Sync Config reprocessing. Some specific scenarios: diff --git a/sync/advanced/case-sensitivity.mdx b/sync/advanced/case-sensitivity.mdx index 67cf813e7..7cf4b8e38 100644 --- a/sync/advanced/case-sensitivity.mdx +++ b/sync/advanced/case-sensitivity.mdx @@ -1,6 +1,6 @@ --- title: "Case Sensitivity" -description: "Handle case-sensitive table and column names in PowerSync Sync Streams/Rules, with best practices for lowercase identifiers and quoting strategies." +description: "Handle case-sensitive table and column names in PowerSync sync queries, with best practices for lowercase identifiers and quoting strategies." --- {/* Wrapper page: the content is snippets/sync-shared/case-sensitivity.mdx, which also renders at sync/rules/case-sensitivity.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} diff --git a/sync/advanced/schemas-and-connections.mdx b/sync/advanced/schemas-and-connections.mdx index 4a666c980..e3653fd9e 100644 --- a/sync/advanced/schemas-and-connections.mdx +++ b/sync/advanced/schemas-and-connections.mdx @@ -1,6 +1,6 @@ --- title: "Schemas and Connections" -description: "Configure Postgres schema usage in Sync Streams/Rules queries, including wildcard schemas for schema-per-tenant setups, and connect to high-availability replicas." +description: "Configure Postgres schema usage in sync queries, including wildcard schemas for schema-per-tenant setups, and connect to high-availability replicas." --- {/* Wrapper page: the content is snippets/sync-shared/schemas-and-connections.mdx, which also renders at sync/rules/schemas-and-connections.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} diff --git a/sync/grammar/sync-rules/index.mdx b/sync/grammar/sync-rules/index.mdx index c96e0e1bc..1efebf5e5 100644 --- a/sync/grammar/sync-rules/index.mdx +++ b/sync/grammar/sync-rules/index.mdx @@ -3,6 +3,10 @@ title: "Grammar Reference (Sync Rules)" description: "Railroad diagram reference for the SQL grammar supported in legacy Sync Rules queries." --- + +Sync Rules are deprecated. For the Sync Streams version of this page, see [Grammar Reference](/sync/grammar/sync-streams/index). + + This page is a formal grammar reference for Sync Rules: it shows the syntax accepted for parameter queries and data queries using railroad diagrams. This page complements the [Supported SQL](/sync/rules/supported-sql) guide, which explains in prose what you can write, with examples and restrictions. **When to use this page:** If you need to check whether a construct is valid, see how parameter vs data query syntax differs, or you're used to grammar specs, use the diagrams and the "Used by" / "References" links to navigate. For most users just getting started, see [Supported SQL](/sync/rules/supported-sql) and the [Sync Rules](/sync/rules/overview) docs. diff --git a/sync/rules/case-sensitivity.mdx b/sync/rules/case-sensitivity.mdx index c4b61d188..13f660cb1 100644 --- a/sync/rules/case-sensitivity.mdx +++ b/sync/rules/case-sensitivity.mdx @@ -1,6 +1,6 @@ --- title: "Case Sensitivity" -description: "Handle case-sensitive table and column names in PowerSync Sync Streams/Rules, with best practices for lowercase identifiers and quoting strategies." +description: "Handle case-sensitive table and column names in PowerSync sync queries, with best practices for lowercase identifiers and quoting strategies." noindex: true --- diff --git a/sync/rules/client-parameters.mdx b/sync/rules/client-parameters.mdx index cdd33224b..fbab9c7b0 100644 --- a/sync/rules/client-parameters.mdx +++ b/sync/rules/client-parameters.mdx @@ -3,6 +3,10 @@ title: "Client Parameters" description: "Pass dynamic parameters from the client SDK directly into Sync Rules queries to filter data per user, device, or application context." --- + +Sync Rules are deprecated. For the Sync Streams equivalent, see [Connection Parameters](/sync/streams/parameters#connection-parameters). + + Use client parameters with caution. Please make sure to read the [Security consideration](#security-consideration) section below. diff --git a/sync/rules/data-queries.mdx b/sync/rules/data-queries.mdx index b654435fb..5341c67d5 100644 --- a/sync/rules/data-queries.mdx +++ b/sync/rules/data-queries.mdx @@ -3,6 +3,10 @@ title: "Data Queries" description: "Write Data Queries in Sync Rules to select and filter data for buckets using bucket parameters." --- + +Sync Rules are deprecated. For the Sync Streams equivalent, see [Writing Queries](/sync/streams/queries). + + Data Queries select the data that form part of a [bucket](/architecture/powersync-service#bucket-system), using the bucket [parameters](/sync/rules/overview#parameters). Multiple Data Queries can be specified for a single [bucket definition](/sync/rules/overview#bucket-definition). diff --git a/sync/rules/global-buckets.mdx b/sync/rules/global-buckets.mdx index 747727d0a..62d302442 100644 --- a/sync/rules/global-buckets.mdx +++ b/sync/rules/global-buckets.mdx @@ -3,6 +3,10 @@ title: "Global Buckets" description: "Set up global buckets in Sync Rules to sync shared reference data to all connected users without per-user filtering or parameter queries." --- + +Sync Rules are deprecated. For the Sync Streams equivalent, see [Global Data](/sync/streams/quickstart#global-data) in the Sync Streams Quickstart. + + Any bucket with no _Parameter Query_ in the bucket definition is automatically a _Global Bucket_. These buckets will be synced to all clients/users. For example, the following Sync Rules sync all `todos` and only unarchived `lists` to all clients/users: diff --git a/sync/rules/many-to-many-join-tables.mdx b/sync/rules/many-to-many-join-tables.mdx index 21a141ded..e1de45f59 100644 --- a/sync/rules/many-to-many-join-tables.mdx +++ b/sync/rules/many-to-many-join-tables.mdx @@ -4,11 +4,11 @@ sidebarTitle: "Many-to-Many and Join Tables" description: "Handle many-to-many relationships in Sync Rules using join table strategies." --- -Join tables are often used to implement many-to-many relationships between tables. Join queries are not directly supported in PowerSync Sync Rules, and require some workarounds depending on the use case. This guide contains some recommended strategies. + +Sync Rules are deprecated. Sync Streams support [JOINs](/sync/streams/queries#using-joins) and [nested subqueries](/sync/streams/queries#using-subqueries), which handle most many-to-many relationships directly without the workarounds on this page. See [Many-to-Many with Sync Streams](/sync/streams/examples#many-to-many-relationships). + - -**Using Sync Streams?** Sync Streams support [JOINs](/sync/streams/queries#using-joins) and [nested subqueries](/sync/streams/queries#using-subqueries), which handle most many-to-many relationships directly without the workarounds described here. See [Many-to-Many with Sync Streams](/sync/streams/examples#many-to-many-relationships) for examples. - +Join tables are often used to implement many-to-many relationships between tables. Join queries are not directly supported in PowerSync Sync Rules, and require some workarounds depending on the use case. This guide contains some recommended strategies. **Postgres users:** For Postgres source databases, you can use the [`pg_ivm` extension](https://www.powersync.com/blog/using-pg-ivm-to-enable-joins-in-powersync) to create incrementally maintained materialized views with JOINs that can be referenced directly in Sync Rules. This approach avoids the need to denormalize your schema. diff --git a/sync/rules/organize-data-into-buckets.mdx b/sync/rules/organize-data-into-buckets.mdx index 8888b9757..c42b3e454 100644 --- a/sync/rules/organize-data-into-buckets.mdx +++ b/sync/rules/organize-data-into-buckets.mdx @@ -3,6 +3,10 @@ title: "Organize Data Into Buckets" description: "Design Sync Rules to organize data into global and user-filtered buckets." --- + +Sync Rules are deprecated. For the Sync Streams equivalent, see the [Sync Streams Quickstart](/sync/streams/quickstart). + + Designing your Sync Rules is about _organizing data into buckets_, and creating the bucket definitions accordingly. Each [bucket definition](/sync/rules/overview#bucket-definition) defines a set of tables/collections and rows/documents to sync. * If there's some data you want to sync to _all_ your users/clients, you can add bucket definitions for one or more [Global Buckets](/sync/rules/global-buckets). This is the simplest way to get started with PowerSync. diff --git a/sync/rules/overview.mdx b/sync/rules/overview.mdx index 78ae6a770..edeb3c6a2 100644 --- a/sync/rules/overview.mdx +++ b/sync/rules/overview.mdx @@ -4,14 +4,14 @@ sidebarTitle: "Overview & Key Concepts" description: "Understand legacy Sync Rules for controlling which data syncs to each client." --- -Sync Rules are PowerSync's original system for partial sync, using YAML bucket definitions. They are deprecated. Existing instances keep working and stay supported while you migrate to Sync Streams. +Sync Rules are PowerSync's original system for partial sync, using YAML bucket definitions. {/* TODO: Link to the Sync Rules deprecation announcement on releases.powersync.com once it is published. */} **Sync Rules are deprecated** -PowerSync is phasing out Sync Rules in favor of [Sync Streams](/sync/streams/overview), which support everything Sync Rules do and add on-demand syncing, JOINs, CTEs, and subqueries. Nothing changes for your instance today: Sync Rules keep working and stay supported while you migrate. New sync config features are added to Sync Streams only. +PowerSync is phasing out Sync Rules in favor of [Sync Streams](/sync/streams/overview), which support everything Sync Rules do and add on-demand syncing, JOINs, CTEs, and subqueries. Nothing changes for your instance today: Sync Rules keep working and stay supported while you migrate. New Sync Config features are added to Sync Streams only. To migrate, click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to convert your current config. Migrating does not change what your app syncs. See [Migrate to Sync Streams](/sync/rules/migrate-to-sync-streams). diff --git a/sync/rules/parameter-queries.mdx b/sync/rules/parameter-queries.mdx index c8c3972a9..2b532b489 100644 --- a/sync/rules/parameter-queries.mdx +++ b/sync/rules/parameter-queries.mdx @@ -3,6 +3,10 @@ title: "Parameter Queries" description: "Define bucket parameters in Sync Rules using Parameter Queries with JWT claims and client parameters." --- + +Sync Rules are deprecated. For the Sync Streams equivalent, see [Using Parameters](/sync/streams/parameters). + + _Parameter Queries_ allow [parameters](/sync/rules/overview#parameters) to be defined on a [bucket](/sync/rules/overview#bucket-definition) to group data. Each [bucket](/sync/rules/overview#bucket-definition) can have zero or more Parameter Queries. diff --git a/sync/rules/schemas-and-connections.mdx b/sync/rules/schemas-and-connections.mdx index f6a70c675..46f347846 100644 --- a/sync/rules/schemas-and-connections.mdx +++ b/sync/rules/schemas-and-connections.mdx @@ -1,6 +1,6 @@ --- title: "Schemas and Connections" -description: "Configure Postgres schema usage in Sync Streams/Rules queries, including wildcard schemas for schema-per-tenant setups, and connect to high-availability replicas." +description: "Configure Postgres schema usage in sync queries, including wildcard schemas for schema-per-tenant setups, and connect to high-availability replicas." noindex: true --- diff --git a/sync/rules/supported-sql.mdx b/sync/rules/supported-sql.mdx index 910971a42..ea7ef2908 100644 --- a/sync/rules/supported-sql.mdx +++ b/sync/rules/supported-sql.mdx @@ -1,5 +1,5 @@ --- -title: "Supported SQL in Sync Rules" +title: "Supported SQL with Sync Rules" sidebarTitle: "Supported SQL" description: "Reference for SQL syntax, operators, built-in functions, and type casting supported in legacy Sync Rules queries." --- @@ -155,3 +155,5 @@ Most functions are from [SQLite built-in functions](https://www.sqlite.org/lang_ - **[ST_Y(point)](/client-sdks/advanced/gis-data-postgis)** — Get the Y coordinate of a [PostGIS](/client-sdks/advanced/gis-data-postgis) point (in Postgres). + +If you need an operator or function not listed, [contact us](/resources/contact-us) so we can consider adding it. diff --git a/sync/streams/prioritized-sync.mdx b/sync/streams/prioritized-sync.mdx index fa25a5ccc..6b9a96500 100644 --- a/sync/streams/prioritized-sync.mdx +++ b/sync/streams/prioritized-sync.mdx @@ -9,7 +9,7 @@ description: "Prioritize which tables sync first so users can start working imme PowerSync supports defining sync priorities, which allows you to control the sync order for different data. This is particularly useful when certain data should be available sooner than others. -In Sync Streams, priorities are assigned to streams and PowerSync manages the underlying buckets internally. +Priorities are assigned to streams, and PowerSync manages the underlying buckets internally. ## Why Use Sync Priorities? From 8b63aaffcb75c0e4b1ae76dc8fa4830e2f28e6f3 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Mon, 21 Sep 2026 15:38:09 +0200 Subject: [PATCH 16/20] The Reducing buckets page needs no sync rules equivalent --- .../sync-shared/reducing-bucket-count.mdx | 240 ------------------ sync/advanced/reducing-bucket-count.mdx | 239 ++++++++++++++++- sync/rules/reducing-bucket-count.mdx | 16 -- 3 files changed, 236 insertions(+), 259 deletions(-) delete mode 100644 snippets/sync-shared/reducing-bucket-count.mdx delete mode 100644 sync/rules/reducing-bucket-count.mdx diff --git a/snippets/sync-shared/reducing-bucket-count.mdx b/snippets/sync-shared/reducing-bucket-count.mdx deleted file mode 100644 index 84568d37c..000000000 --- a/snippets/sync-shared/reducing-bucket-count.mdx +++ /dev/null @@ -1,240 +0,0 @@ -{/* Shared body: rendered by sync/advanced/reducing-bucket-count.mdx (Sync Streams section) and sync/rules/reducing-bucket-count.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} - -import BucketCountExampleApp from '/snippets/bucket-count-example-app.mdx'; - -If a user syncs too many buckets, or you hit a `PSYNC_S2305` error, this page shows how to find the cause and bring the count down. For how buckets are counted in the first place, see [Bucket Count](/sync/streams/bucket-count). - -PowerSync enforces two limits per user, both with a default of 1,000. One is the number of unique buckets. The other is the number of parameter query results, counted before duplicates are removed. Exceeding either fails the sync with a `PSYNC_S2305` error. The fix is different for each, so start by finding out which one you hit from the error message. See [Limits](/sync/streams/bucket-count#limits) for the full difference. - -## Diagnosing High Bucket Count - -### Reading the Error Message First - -The `PSYNC_S2305` message tells you which limit you reached. The fix is different for each, so read it first. - -- `Too many buckets` means you reached the bucket limit. Reduce the number of unique buckets. Any strategy below helps. -- `Too many parameter query results` means you reached the parameter limit. Reduce the rows your parameter lookups return. Only some strategies help here: [Denormalizing the Scope Key](#denormalizing-the-scope-key) and [Querying the Membership Table Directly](#querying-the-membership-table-directly) cut the lookups themselves, so they lower both counts. - -```mermaid -flowchart TD - E["PSYNC_S2305 error"] --> M{"Which message?"} - M -->|"Too many buckets"| Bk["Reduce unique buckets"] - M -->|"Too many parameter query results"| Pr["Reduce parameter rows"] - Bk --> D["Denormalize the scope key,
or merge streams"] - Pr --> D -``` - -### The Contributor Breakdown - -The `PSYNC_S2305` log includes a breakdown of the streams that contribute the most. - -- For a bucket-limit error, it lists streams by bucket count, highest first. -- For a parameter-limit error, it lists the streams that returned the most rows, and then the stream that exceeded the limit. Each listed stream shows how many rows it returned. The failing stream instead shows how much budget was left when it failed. - - -For a parameter-limit error, the last stream in the breakdown is the one that ran when the limit was reached. This stream is not always the cause. PowerSync adds up parameter results across streams in order. The last stream is only the one that exceeded the limit. Check every stream in the breakdown, not just the last one. - - -### Checkpoint Logs - -Checkpoint logs record the counts for each connection. Find them in your [instance logs](/maintenance-ops/monitoring-and-alerting). For example: - -```text -New checkpoint: 800178 | write: null | buckets: 7 | param_results: 6 ["5#org_data|0[\"ef718ff3...\"]","5#org_data|1[\"1ddeddba...\"]", ...] -``` - -- `buckets` is the number of unique buckets for this connection. -- `param_results` is the total number of parameter rows for this connection. -- The array lists the bucket names. Each name already includes its parameter value. The list stops after 20 names. - -### Sync Diagnostics Client - -The [Sync Diagnostics Client](/tools/diagnostics-client) shows the buckets for one user. It does not load for a user who is over the limit, because that user's sync fails before the data loads. Use the instance logs and the error breakdown for those users. The client shows the bucket count, which may not be the limit you reached. Confirm the limit from the error message. - - - -## Reducing Bucket Count - -Start with the strategy that matches your query pattern. Most high counts come from hierarchical or many-to-many data, where denormalizing the scope key gives the biggest reduction. - -### Multiple Queries per Stream - -**Reduces:** bucket count. - -Use `queries` instead of separate streams to group related tables. All queries in a stream that filter the same way share one bucket per value. See [multiple queries per stream](/sync/streams/queries#multiple-queries-per-stream). - -**Before**: 5 separate streams, each with a direct `auth.user_id()` filter, create 5 buckets per user. - -**After**: 1 stream with 5 queries creates 1 bucket per user. - -```yaml -streams: - user_settings: # [!code --] - query: SELECT * FROM settings WHERE user_id = auth.user_id() # [!code --] - user_prefs: # [!code --] - query: SELECT * FROM preferences WHERE user_id = auth.user_id() # [!code --] - user_org_list: # [!code --] - query: SELECT * FROM org_membership WHERE user_id = auth.user_id() # [!code --] - user_region: # [!code --] - query: SELECT * FROM region_members WHERE user_id = auth.user_id() # [!code --] - user_profile: # [!code --] - query: SELECT * FROM profiles WHERE user_id = auth.user_id() # [!code --] - user_data: # [!code ++] - queries: # [!code ++] - - SELECT * FROM settings WHERE user_id = auth.user_id() # [!code ++] - - SELECT * FROM preferences WHERE user_id = auth.user_id() # [!code ++] - - SELECT * FROM org_membership WHERE user_id = auth.user_id() # [!code ++] - - SELECT * FROM region_members WHERE user_id = auth.user_id() # [!code ++] - - SELECT * FROM profiles WHERE user_id = auth.user_id() # [!code ++] -``` - -### Denormalizing the Scope Key - -**Reduces:** bucket count and parameter query results. - -This is the most effective fix for parent-child data. When chained queries through org → project → task create too many buckets, filter every table with the same top-level parameter, such as `org_id`. A bucket's key must be a column on the table you sync (see [The Partition Key Must Exist on the Row](#the-partition-key-must-exist-on-the-row) below). So this works only if the child tables have that column. If tasks only have `project_id`, add `org_id` to the tasks table. - -**Before**: chained queries create 10 + 500 = 510 buckets for 10 orgs with 50 projects each. Projects and tasks share buckets because they use the same filter. Orgs use a different filter, so they add their own buckets. - -**After**: add `org_id` to the tasks table, drop the `user_projects` CTE, and filter every table by org. This creates 10 buckets. - -```yaml -streams: - org_projects_tasks: - with: - user_orgs: SELECT org_id FROM org_membership WHERE user_id = auth.user_id() - user_projects: SELECT id FROM projects WHERE org_id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) # [!code --] - queries: - - SELECT * FROM orgs WHERE id IN user_orgs - - SELECT * FROM projects WHERE id IN user_projects # [!code --] - - SELECT * FROM projects WHERE org_id IN user_orgs # [!code ++] - - SELECT * FROM tasks WHERE project_id IN user_projects # [!code --] - - SELECT * FROM tasks WHERE org_id IN user_orgs # [!code ++] -``` - -### Querying the Membership Table Directly - -**Reduces:** bucket count and parameter query results. - -When a subquery or JOIN through a membership table creates N buckets, query the membership table directly with a direct auth filter. Use no subquery and no JOIN. You often need fields from the related table, such as the org name, alongside each membership row. Denormalize those fields onto the membership table so they are available without a JOIN. - -**Before**: N org memberships create N buckets. - -**After**: 1 bucket per user, with org fields denormalized onto `org_membership`. - -```yaml -streams: - org_data: # [!code --] - query: SELECT * FROM orgs WHERE id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) # [!code --] - my_org_memberships: # [!code ++] - query: SELECT * FROM org_membership WHERE user_id = auth.user_id() # [!code ++] -``` - -### Many-to-Many via a JSON Array Column - -**Reduces:** bucket count. - -A join through a link table creates one bucket per row of the table you select from. For assets linked to projects through `project_assets`, you get one bucket per asset. - -Add a denormalized `project_ids` JSON array column to `assets`, maintained with database triggers. Then use `json_each()` to traverse it. This lets PowerSync key the bucket by project ID instead of asset ID. - -**Before**: one bucket per asset. 2,000 assets create 2,000 buckets. - -**After**: key by project. 50 projects create 50 buckets. - -```yaml -streams: - assets_in_projects: - with: - user_projects: SELECT id FROM projects WHERE org_id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) - query: SELECT assets.* FROM assets JOIN project_assets ON project_assets.asset_id = assets.id WHERE project_assets.project_id IN user_projects # [!code --] - query: SELECT assets.* FROM assets INNER JOIN json_each(assets.project_ids) AS p INNER JOIN user_projects ON p.value = user_projects.id # [!code ++] -``` - -The `INNER JOIN user_projects` syncs only assets that belong to at least one of the user's projects. The bucket key is the project ID, so the count matches the number of projects, not assets. - -### Subscription Parameters for On-Demand Sync - -**Reduces:** bucket count. - -Buckets are created per active subscription, not from every possible value. Use `subscription.parameter('project_id')` so the count is bounded by how many subscriptions the client has active. - -**Before**: a subquery returns all of the user's projects. 50 projects create 50 buckets. - -**After**: the client subscribes per project on demand. 3 open projects create 3 buckets. - -```yaml -streams: - project_tasks: - with: - user_projects: SELECT id FROM projects WHERE org_id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) - query: SELECT * FROM tasks WHERE project_id IN user_projects # [!code --] - query: SELECT * FROM tasks WHERE project_id = subscription.parameter('project_id') AND project_id IN user_projects # [!code ++] -``` - -The client subscribes when the user opens a project and unsubscribes when they leave. This works only when the user does not need every record available offline at the same time. - -## Edge Cases and Gotchas - -### The Partition Key Must Exist on the Row - -A bucket's key must be a value that physically exists on a row of the table you sync. You cannot split a table into buckets by a column it does not have. This is why denormalizing the scope key onto child tables is the standard fix. If tasks only have `project_id`, you cannot key their buckets by `org_id` until you add `org_id` to the tasks table. - -### Subscription Parameters Choose Buckets, Not Re-Partition Them - -A subscription parameter lets the client choose which existing buckets to sync. It does not change how those buckets are defined. - -For a parameter to select a bucket, its value must match a value on the row being synced. For example, each task has a `project_id`, so you can use that column to group tasks into project buckets: - -```yaml -streams: - project_tasks: - query: SELECT * FROM tasks WHERE project_id = subscription.parameter('project_id') -``` - -Assets are different. An asset can belong to multiple projects, so the asset row does not have a single `project_id`. Passing a `project_id` as a subscription parameter therefore cannot make PowerSync group those assets by project. The asset row has no project ID to match against. - -If you want to sync assets by project, the asset row needs to contain a project reference first. For example, you could add a `project_ids` array column as described in [Reducing Bucket Count](#reducing-bucket-count). - -### Correlated Joins Behave Like Subqueries - -A correlated JOIN and an `IN (subquery)` compile to the same internal form. They create the same number of buckets. Rewriting one as the other does not reduce the count. - -### CTEs Cannot Reference Each Other - -Each CTE must be self-contained. A CTE cannot reference another CTE by name. If it does, the deploy fails. Inline the nested subquery instead. See [CTE limitations](/sync/streams/ctes#limitations). - -### Global Buckets Multiply Storage and Cost - -A stream with no filter creates one global bucket that every user syncs. Under `auto_subscribe: true`, every write to that table fans out to every user. This drives up synced data volume and cost. Scope global buckets carefully, and only mark truly shared reference data as global. - -### Bucket Storage Does Not Shrink When You Archive - -Buckets are append-only. Marking a row as archived does not remove it from bucket storage on its own. A row leaves storage only when it stops matching the data query, through a hard delete or a filter on the table's own column. Storage reclaims space during [compaction](/maintenance-ops/compacting-buckets). Filtering through a parent table does not shrink a child table's stored data. - -## Increasing the Limit - -Raise the limit only after you exhaust the reduction strategies above. - -Before you raise it, weigh the cost. Sync overhead scales roughly linearly with the number of buckets per user. Doubling the bucket count roughly doubles sync latency for a single operation. It also roughly doubles CPU and memory use on the server and the client. Many operations inside a single bucket scale much more efficiently than many buckets. The 1,000 default exists to encourage fewer, larger buckets and to protect the service from excessive counts. - -On PowerSync Cloud, you can request a higher limit on [Team and Enterprise](https://www.powersync.com/pricing) plans, up to 10,000. The limit applies per user, so your instance can still track far more buckets in total. - -For self-hosted deployments, set the limits under `api.parameters`: - -```yaml service.yaml -api: - parameters: - max_buckets_per_connection: 5000 - max_parameter_query_results: 5000 -``` - -Set both. Raising one without the other still leaves you capped by the limit you did not change. - -## Related Pages - -- [Bucket Count](/sync/streams/bucket-count) explains how buckets are counted and the two limits. -- [Writing Queries](/sync/streams/queries) covers the query syntax that determines your bucket count. -- [Common Table Expressions (CTEs)](/sync/streams/ctes) covers shared filtering logic. -- [Troubleshooting](/debugging/troubleshooting#psync_s2305-too-many-buckets-/-parameter-query-results) covers the `PSYNC_S2305` error. -- [Performance and Limits](/resources/performance-and-limits) lists the Service limits. diff --git a/sync/advanced/reducing-bucket-count.mdx b/sync/advanced/reducing-bucket-count.mdx index 37162620d..16c42f2f2 100644 --- a/sync/advanced/reducing-bucket-count.mdx +++ b/sync/advanced/reducing-bucket-count.mdx @@ -4,8 +4,241 @@ description: "Diagnose a high bucket count, reduce the number of buckets a user sidebarTitle: "Reducing Buckets" --- -{/* Wrapper page: the content is snippets/sync-shared/reducing-bucket-count.mdx, which also renders at sync/rules/reducing-bucket-count.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} +import BucketCountExampleApp from '/snippets/bucket-count-example-app.mdx'; -import ReducingBucketCount from '/snippets/sync-shared/reducing-bucket-count.mdx'; +If a user syncs too many buckets, or you hit a `PSYNC_S2305` error, this page shows how to find the cause and bring the count down. For how buckets are counted in the first place, see [Bucket Count](/sync/streams/bucket-count). - +PowerSync enforces two limits per user, both with a default of 1,000. One is the number of unique buckets. The other is the number of parameter query results, counted before duplicates are removed. Exceeding either fails the sync with a `PSYNC_S2305` error. The fix is different for each, so start by finding out which one you hit from the error message. See [Limits](/sync/streams/bucket-count#limits) for the full difference. + +## Diagnosing High Bucket Count + +### Reading the Error Message First + +The `PSYNC_S2305` message tells you which limit you reached. The fix is different for each, so read it first. + +- `Too many buckets` means you reached the bucket limit. Reduce the number of unique buckets. Any strategy below helps. +- `Too many parameter query results` means you reached the parameter limit. Reduce the rows your parameter lookups return. Only some strategies help here: [Denormalizing the Scope Key](#denormalizing-the-scope-key) and [Querying the Membership Table Directly](#querying-the-membership-table-directly) cut the lookups themselves, so they lower both counts. + +```mermaid +flowchart TD + E["PSYNC_S2305 error"] --> M{"Which message?"} + M -->|"Too many buckets"| Bk["Reduce unique buckets"] + M -->|"Too many parameter query results"| Pr["Reduce parameter rows"] + Bk --> D["Denormalize the scope key,
or merge streams"] + Pr --> D +``` + +### The Contributor Breakdown + +The `PSYNC_S2305` log includes a breakdown of the streams that contribute the most. + +- For a bucket-limit error, it lists streams by bucket count, highest first. +- For a parameter-limit error, it lists the streams that returned the most rows, and then the stream that exceeded the limit. Each listed stream shows how many rows it returned. The failing stream instead shows how much budget was left when it failed. + + +For a parameter-limit error, the last stream in the breakdown is the one that ran when the limit was reached. This stream is not always the cause. PowerSync adds up parameter results across streams in order. The last stream is only the one that exceeded the limit. Check every stream in the breakdown, not just the last one. + + +### Checkpoint Logs + +Checkpoint logs record the counts for each connection. Find them in your [instance logs](/maintenance-ops/monitoring-and-alerting). For example: + +```text +New checkpoint: 800178 | write: null | buckets: 7 | param_results: 6 ["5#org_data|0[\"ef718ff3...\"]","5#org_data|1[\"1ddeddba...\"]", ...] +``` + +- `buckets` is the number of unique buckets for this connection. +- `param_results` is the total number of parameter rows for this connection. +- The array lists the bucket names. Each name already includes its parameter value. The list stops after 20 names. + +### Sync Diagnostics Client + +The [Sync Diagnostics Client](/tools/diagnostics-client) shows the buckets for one user. It does not load for a user who is over the limit, because that user's sync fails before the data loads. Use the instance logs and the error breakdown for those users. The client shows the bucket count, which may not be the limit you reached. Confirm the limit from the error message. + + + +## Reducing Bucket Count + +Start with the strategy that matches your query pattern. Most high counts come from hierarchical or many-to-many data, where denormalizing the scope key gives the biggest reduction. + +### Multiple Queries per Stream + +**Reduces:** bucket count. + +Use `queries` instead of separate streams to group related tables. All queries in a stream that filter the same way share one bucket per value. See [multiple queries per stream](/sync/streams/queries#multiple-queries-per-stream). + +**Before**: 5 separate streams, each with a direct `auth.user_id()` filter, create 5 buckets per user. + +**After**: 1 stream with 5 queries creates 1 bucket per user. + +```yaml +streams: + user_settings: # [!code --] + query: SELECT * FROM settings WHERE user_id = auth.user_id() # [!code --] + user_prefs: # [!code --] + query: SELECT * FROM preferences WHERE user_id = auth.user_id() # [!code --] + user_org_list: # [!code --] + query: SELECT * FROM org_membership WHERE user_id = auth.user_id() # [!code --] + user_region: # [!code --] + query: SELECT * FROM region_members WHERE user_id = auth.user_id() # [!code --] + user_profile: # [!code --] + query: SELECT * FROM profiles WHERE user_id = auth.user_id() # [!code --] + user_data: # [!code ++] + queries: # [!code ++] + - SELECT * FROM settings WHERE user_id = auth.user_id() # [!code ++] + - SELECT * FROM preferences WHERE user_id = auth.user_id() # [!code ++] + - SELECT * FROM org_membership WHERE user_id = auth.user_id() # [!code ++] + - SELECT * FROM region_members WHERE user_id = auth.user_id() # [!code ++] + - SELECT * FROM profiles WHERE user_id = auth.user_id() # [!code ++] +``` + +### Denormalizing the Scope Key + +**Reduces:** bucket count and parameter query results. + +This is the most effective fix for parent-child data. When chained queries through org → project → task create too many buckets, filter every table with the same top-level parameter, such as `org_id`. A bucket's key must be a column on the table you sync (see [The Partition Key Must Exist on the Row](#the-partition-key-must-exist-on-the-row) below). So this works only if the child tables have that column. If tasks only have `project_id`, add `org_id` to the tasks table. + +**Before**: chained queries create 10 + 500 = 510 buckets for 10 orgs with 50 projects each. Projects and tasks share buckets because they use the same filter. Orgs use a different filter, so they add their own buckets. + +**After**: add `org_id` to the tasks table, drop the `user_projects` CTE, and filter every table by org. This creates 10 buckets. + +```yaml +streams: + org_projects_tasks: + with: + user_orgs: SELECT org_id FROM org_membership WHERE user_id = auth.user_id() + user_projects: SELECT id FROM projects WHERE org_id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) # [!code --] + queries: + - SELECT * FROM orgs WHERE id IN user_orgs + - SELECT * FROM projects WHERE id IN user_projects # [!code --] + - SELECT * FROM projects WHERE org_id IN user_orgs # [!code ++] + - SELECT * FROM tasks WHERE project_id IN user_projects # [!code --] + - SELECT * FROM tasks WHERE org_id IN user_orgs # [!code ++] +``` + +### Querying the Membership Table Directly + +**Reduces:** bucket count and parameter query results. + +When a subquery or JOIN through a membership table creates N buckets, query the membership table directly with a direct auth filter. Use no subquery and no JOIN. You often need fields from the related table, such as the org name, alongside each membership row. Denormalize those fields onto the membership table so they are available without a JOIN. + +**Before**: N org memberships create N buckets. + +**After**: 1 bucket per user, with org fields denormalized onto `org_membership`. + +```yaml +streams: + org_data: # [!code --] + query: SELECT * FROM orgs WHERE id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) # [!code --] + my_org_memberships: # [!code ++] + query: SELECT * FROM org_membership WHERE user_id = auth.user_id() # [!code ++] +``` + +### Many-to-Many via a JSON Array Column + +**Reduces:** bucket count. + +A join through a link table creates one bucket per row of the table you select from. For assets linked to projects through `project_assets`, you get one bucket per asset. + +Add a denormalized `project_ids` JSON array column to `assets`, maintained with database triggers. Then use `json_each()` to traverse it. This lets PowerSync key the bucket by project ID instead of asset ID. + +**Before**: one bucket per asset. 2,000 assets create 2,000 buckets. + +**After**: key by project. 50 projects create 50 buckets. + +```yaml +streams: + assets_in_projects: + with: + user_projects: SELECT id FROM projects WHERE org_id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) + query: SELECT assets.* FROM assets JOIN project_assets ON project_assets.asset_id = assets.id WHERE project_assets.project_id IN user_projects # [!code --] + query: SELECT assets.* FROM assets INNER JOIN json_each(assets.project_ids) AS p INNER JOIN user_projects ON p.value = user_projects.id # [!code ++] +``` + +The `INNER JOIN user_projects` syncs only assets that belong to at least one of the user's projects. The bucket key is the project ID, so the count matches the number of projects, not assets. + +### Subscription Parameters for On-Demand Sync + +**Reduces:** bucket count. + +Buckets are created per active subscription, not from every possible value. Use `subscription.parameter('project_id')` so the count is bounded by how many subscriptions the client has active. + +**Before**: a subquery returns all of the user's projects. 50 projects create 50 buckets. + +**After**: the client subscribes per project on demand. 3 open projects create 3 buckets. + +```yaml +streams: + project_tasks: + with: + user_projects: SELECT id FROM projects WHERE org_id IN (SELECT org_id FROM org_membership WHERE user_id = auth.user_id()) + query: SELECT * FROM tasks WHERE project_id IN user_projects # [!code --] + query: SELECT * FROM tasks WHERE project_id = subscription.parameter('project_id') AND project_id IN user_projects # [!code ++] +``` + +The client subscribes when the user opens a project and unsubscribes when they leave. This works only when the user does not need every record available offline at the same time. + +## Edge Cases and Gotchas + +### The Partition Key Must Exist on the Row + +A bucket's key must be a value that physically exists on a row of the table you sync. You cannot split a table into buckets by a column it does not have. This is why denormalizing the scope key onto child tables is the standard fix. If tasks only have `project_id`, you cannot key their buckets by `org_id` until you add `org_id` to the tasks table. + +### Subscription Parameters Choose Buckets, Not Re-Partition Them + +A subscription parameter lets the client choose which existing buckets to sync. It does not change how those buckets are defined. + +For a parameter to select a bucket, its value must match a value on the row being synced. For example, each task has a `project_id`, so you can use that column to group tasks into project buckets: + +```yaml +streams: + project_tasks: + query: SELECT * FROM tasks WHERE project_id = subscription.parameter('project_id') +``` + +Assets are different. An asset can belong to multiple projects, so the asset row does not have a single `project_id`. Passing a `project_id` as a subscription parameter therefore cannot make PowerSync group those assets by project. The asset row has no project ID to match against. + +If you want to sync assets by project, the asset row needs to contain a project reference first. For example, you could add a `project_ids` array column as described in [Reducing Bucket Count](#reducing-bucket-count). + +### Correlated Joins Behave Like Subqueries + +A correlated JOIN and an `IN (subquery)` compile to the same internal form. They create the same number of buckets. Rewriting one as the other does not reduce the count. + +### CTEs Cannot Reference Each Other + +Each CTE must be self-contained. A CTE cannot reference another CTE by name. If it does, the deploy fails. Inline the nested subquery instead. See [CTE limitations](/sync/streams/ctes#limitations). + +### Global Buckets Multiply Storage and Cost + +A stream with no filter creates one global bucket that every user syncs. Under `auto_subscribe: true`, every write to that table fans out to every user. This drives up synced data volume and cost. Scope global buckets carefully, and only mark truly shared reference data as global. + +### Bucket Storage Does Not Shrink When You Archive + +Buckets are append-only. Marking a row as archived does not remove it from bucket storage on its own. A row leaves storage only when it stops matching the data query, through a hard delete or a filter on the table's own column. Storage reclaims space during [compaction](/maintenance-ops/compacting-buckets). Filtering through a parent table does not shrink a child table's stored data. + +## Increasing the Limit + +Raise the limit only after you exhaust the reduction strategies above. + +Before you raise it, weigh the cost. Sync overhead scales roughly linearly with the number of buckets per user. Doubling the bucket count roughly doubles sync latency for a single operation. It also roughly doubles CPU and memory use on the server and the client. Many operations inside a single bucket scale much more efficiently than many buckets. The 1,000 default exists to encourage fewer, larger buckets and to protect the service from excessive counts. + +On PowerSync Cloud, you can request a higher limit on [Team and Enterprise](https://www.powersync.com/pricing) plans, up to 10,000. The limit applies per user, so your instance can still track far more buckets in total. + +For self-hosted deployments, set the limits under `api.parameters`: + +```yaml service.yaml +api: + parameters: + max_buckets_per_connection: 5000 + max_parameter_query_results: 5000 +``` + +Set both. Raising one without the other still leaves you capped by the limit you did not change. + +## Related Pages + +- [Bucket Count](/sync/streams/bucket-count) explains how buckets are counted and the two limits. +- [Writing Queries](/sync/streams/queries) covers the query syntax that determines your bucket count. +- [Common Table Expressions (CTEs)](/sync/streams/ctes) covers shared filtering logic. +- [Troubleshooting](/debugging/troubleshooting#psync_s2305-too-many-buckets-/-parameter-query-results) covers the `PSYNC_S2305` error. +- [Performance and Limits](/resources/performance-and-limits) lists the Service limits. diff --git a/sync/rules/reducing-bucket-count.mdx b/sync/rules/reducing-bucket-count.mdx deleted file mode 100644 index c20a2ec30..000000000 --- a/sync/rules/reducing-bucket-count.mdx +++ /dev/null @@ -1,16 +0,0 @@ ---- -title: "Reducing Bucket Count" -description: "Diagnose a high bucket count, reduce the number of buckets a user syncs, and raise the per-user limits when needed." -sidebarTitle: "Reducing Buckets" -noindex: true ---- - -{/* Wrapper page: the content is snippets/sync-shared/reducing-bucket-count.mdx, which also renders at sync/advanced/reducing-bucket-count.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} - -import ReducingBucketCount from '/snippets/sync-shared/reducing-bucket-count.mdx'; - - -Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. Configuration examples use Sync Streams syntax. - - - From 7315366cfbfa0074fae22b09ef125866c86fe667 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Mon, 21 Sep 2026 15:38:24 +0200 Subject: [PATCH 17/20] Missed part of last commit --- docs.json | 1 - 1 file changed, 1 deletion(-) diff --git a/docs.json b/docs.json index 8fe96b689..cb659808b 100644 --- a/docs.json +++ b/docs.json @@ -246,7 +246,6 @@ { "group": "Advanced", "pages": [ - "sync/rules/reducing-bucket-count", "sync/rules/prioritized-sync", "sync/rules/client-id", "sync/rules/case-sensitivity", From 733103d068956950b5dacd416e11322b7e2f1087 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Mon, 21 Sep 2026 15:46:58 +0200 Subject: [PATCH 18/20] Much thinner page for Storage V4 under Sync Rules --- snippets/sync-shared/storage-version-4.mdx | 170 -------------------- sync/advanced/storage-version-4.mdx | 171 ++++++++++++++++++++- sync/rules/storage-version-4.mdx | 40 ++++- 3 files changed, 200 insertions(+), 181 deletions(-) delete mode 100644 snippets/sync-shared/storage-version-4.mdx diff --git a/snippets/sync-shared/storage-version-4.mdx b/snippets/sync-shared/storage-version-4.mdx deleted file mode 100644 index 17a4df47d..000000000 --- a/snippets/sync-shared/storage-version-4.mdx +++ /dev/null @@ -1,170 +0,0 @@ -{/* Shared body: rendered by sync/advanced/storage-version-4.mdx (Sync Streams section) and sync/rules/storage-version-4.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} - -Storage version 4 is a new version of the format the PowerSync Service uses to store the data it syncs to clients. It is in [Beta](/resources/feature-status) as of PowerSync Service v1.26.0. - -Compared to version 2, it provides: - -- Faster sync and faster reprocessing after a deployment. -- [Incremental reprocessing](#incremental-reprocessing): a Sync Streams deployment reprocesses only the streams you added or changed. Clients no longer download all their data again after every deployment. -- [S3 object storage](#s3-object-storage): larger blocks of synced data move from the storage database to S3. This reduces load on the storage database when many clients sync at once or sync large amounts of data. - -## Availability - -Storage version 4 is compatible with all PowerSync Cloud instances, which already use MongoDB [bucket storage](/architecture/powersync-service#bucket-storage). Self-hosted instances must also use MongoDB bucket storage. Postgres bucket storage is not currently supported. - -The PowerSync Cloud and self-hosted columns below apply during the Beta only. Once storage version 4 is generally available, it will become the default for all supported instances. S3 object storage is then also enabled on all PowerSync Cloud instances. For self-hosted deployments, follow the [S3 setup instructions](#self-hosted-s3-setup). - -| | Source database | Sync Config | PowerSync Cloud (Beta) | Self-hosted (Beta) | -| --- | --- | --- | --- | --- | -| Storage version 4 | Any | Sync Streams or Sync Rules | Free plan: automatic. Other plans: [opt in](#opt-in). | [Opt in](#opt-in) | -| Incremental reprocessing | MongoDB | Sync Streams | Included with version 4 | Included with version 4 | -| S3 object storage | Any | Sync Streams or Sync Rules | Enabled per instance by PowerSync on request | [Set up S3 object storage](#self-hosted-s3-setup) | - -Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/rules/migrate-to-sync-streams). - -## Opt In - -Version 4 is not the default in PowerSync Service v1.26.0. Moving a Sync Config to version 4 runs like any other deployment: - -1. PowerSync reprocesses all data selected by your Sync Config in the background. The current version keeps serving clients, so there is no downtime. -2. When the new copy is ready, PowerSync switches to it. On PowerSync Cloud, this appears as a new deploy event in the PowerSync Dashboard. -3. Clients download their data again once, as after any deployment before version 4. On self-hosted deployments with many clients, scale out the API before the switch to absorb the re-sync. - -After this first deployment, later Sync Streams deployments use incremental reprocessing automatically when your instance meets its requirements. There is no separate setting. - -### PowerSync Cloud - -Free plan instances are upgraded automatically during the Beta. No action is needed. - -On other plans, add `storage_version: 4` to the `config` block of your Sync Config and deploy it: - -```yaml -config: - edition: 3 - storage_version: 4 - -streams: - todos: - query: SELECT * FROM todos WHERE owner_id = auth.user_id() -``` - -### Self-Hosted - - - Postgres bucket storage is not supported with version 4. - - -Add `storage_version: 4` to the `config` block of each Sync Config as shown above, then deploy or redeploy it to use version 4. - -To move a Sync Config back to version 2, set `storage_version: 2` and deploy again. This is another full reprocess. - -To also enable S3 object storage, follow the [self-hosted S3 setup instructions](#self-hosted-s3-setup) to prepare a bucket and configure the Service. - -## Incremental Reprocessing - -Incremental reprocessing is active when you use a MongoDB source database, Sync Streams, and storage version 4. - - - Self-hosted instances with Postgres bucket storage are not supported. - - -Without it, every deployment reads all data selected by the Sync Config from your source database and prepares a complete new copy. Clients then download all their data again, even if only one stream changed. - -With incremental reprocessing, PowerSync compares the new Sync Config with the current one and reprocesses only the streams you added or changed. Unchanged streams keep their data on the PowerSync Service and on clients. Deployments finish faster, your source database does less work, and clients download only the data for affected streams they subscribe to. - -- Adding a stream reads only the data that stream selects. -- Removing a stream requires no new source reads. PowerSync cleans up stored definitions when no active Sync Config still uses them. -- Renaming a stream counts as removing it and adding a new one, so its data is rebuilt. -- Changing a stream's queries may reprocess affected definitions. Changes that only affect how request parameters select existing buckets do not require reprocessing. - -For example, changing `SELECT * FROM projects WHERE user_id = auth.user_id()` to `SELECT * FROM projects WHERE user_id = auth.jwt() ->> 'owner'` reuses the existing bucket data. The data is still grouped by `user_id`; only the JWT field used to select buckets changes. - -The time saved depends on how your data is split across streams. If one stream selects most of your data, changing that stream still takes about as long as a full reprocess. - -PowerSync favors correctness over reuse. When it cannot confirm that a change leaves a stream's data unchanged, it rebuilds that stream. A deployment that reprocesses more than you expect is not an error. - -Event definitions for [Custom Write Checkpoints](/handling-writes/custom-write-checkpoints) follow the same rules. Unchanged events keep their data, and new or changed events are read again. - -### When PowerSync Reprocesses Everything - -Some changes start a full reprocess, after which clients download all their data again: - -- The first deployment on storage version 4. -- Changes to the `config` block of the Sync Config, such as `edition`, compatibility fixes, or `storage_version`. -- The **Defragment** action in the PowerSync Dashboard, which exists to rebuild all data. See [Defragmenting](/maintenance-ops/compacting-buckets#defragmenting). -- Replication failures, for example when PowerSync loses its position in the MongoDB change stream and has to start over. - -### Sync Config Versions and Replication Streams - -Each deployment has a Sync Config version. With incremental reprocessing, multiple versions can share a replication stream, the replication process and stored state. A full reprocess creates a new replication stream. - -See the [Log Reference](/debugging/log-reference#message-prefixes) for how to identify these versions and streams in your logs. - -For implementation details, see the [storage design](https://github.com/powersync-ja/powersync-service/blob/main/docs/storage/storage-v3.md). The document describes the design introduced in version 3 and carried into version 4. - -### Checking What a Deployment Reprocessed - -If a deployment takes longer or reprocesses more than you expect, see [Checking What a Deployment Reprocessed](/debugging/log-reference#checking-what-a-deployment-reprocessed) in the Log Reference for what to look for in your logs. - -## S3 Object Storage - -Your instance keeps the data it syncs to clients in its bucket storage database, alongside everything else it needs to run. With S3 object storage, larger blocks of that data move to Amazon S3 or an S3-compatible object store, and the PowerSync Service syncs them to clients directly from there. Smaller blocks, and the metadata that locates each block, stay in MongoDB. - -Reading larger blocks from S3 reduces the data MongoDB must read and transfer during sync. When those reads limit performance, offloading them can speed up initial sync and let an instance serve more concurrent clients. The benefit is most noticeable when clients sync large amounts of data or many clients connect at once. The PowerSync Service still handles every client connection, so its CPU and memory capacity also limit concurrency. - -For self-hosted instances, offloading bucket data to S3 can reduce storage and data transfer costs. Compare the reduction in database costs with the object store's storage, request, and data transfer charges for your workload. - -Clients connect only to the PowerSync Service and never to the object store, so no client changes are needed. If the object store becomes unreachable, sync is interrupted until it recovers. Clients reconnect and resume automatically. - -S3 object storage requires storage version 4 and works with Sync Streams and legacy Sync Rules. It is compatible with all PowerSync Cloud instances. - - - S3 object storage holds PowerSync's internal sync data. To store files uploaded by your app, use [Attachments](/client-sdks/advanced/attachments). - - -### PowerSync Cloud - -During the Beta, PowerSync enables S3 object storage per instance. [Contact us](/resources/contact-us) if you want it on your instance before we enable it for all instances. - -### Self-Hosted S3 Setup - - - Self-hosted instances with Postgres bucket storage are not supported. - - - - - Create a bucket. Use the same region as the PowerSync Service where possible, to keep latency low and avoid cross-region data transfer charges. Use a dedicated bucket, or a unique `prefix` per PowerSync instance, so that instances never read or delete each other's files. Give the PowerSync Service permission to list the bucket and to read, write, and delete objects under the prefix. - - Leave object versioning off, or suspend it if the bucket already has it, and leave Object Lock off. PowerSync deletes files itself once they are no longer needed, so versioning keeps charging for old versions and locked objects cannot be cleaned up. Do not add an expiration lifecycle rule: an expired object may still be referenced by MongoDB, which breaks sync for that data. - - - Add `object_storage` to the `storage` section of `service.yaml`: - - ```yaml service.yaml - storage: - type: mongodb - uri: !env PS_MONGO_STORAGE_URI - object_storage: - type: s3 - bucket: powersync-bucket-data - region: us-east-1 - prefix: production - ``` - - Without `access_key_id` and `secret_access_key`, PowerSync uses the AWS credentials available to the process, such as an IAM role. For S3-compatible providers such as MinIO or Cloudflare R2, also set `endpoint`, and set `force_path_style: true` if the provider requires path-style requests. - - Restart or redeploy the PowerSync Service to load the updated `service.yaml`. If you run replication, API, and compacting in separate containers or jobs, apply the same object storage configuration to each. - - - Deploy your Sync Configs on storage version 4 as described in [Opt In](#opt-in). Sync Configs on version 2 keep all data in MongoDB, even when `object_storage` is configured. - - Once replication reaches a healthy checkpoint, confirm that objects appear under the prefix, run a test initial sync, and run `compact` once to surface permission errors early. - - - -After enabling S3 object storage, you can raise [`max_concurrent_connections`](/configuration/powersync-service/self-hosted-instances#param-max-concurrent-connections) from its default of 200 per API process. With storage version 4 and S3 object storage, each API process can handle up to 1,000 concurrent client connections. Performance degrades if a large share of those clients run an initial sync at the same time, so scale out the API before a deployment that makes all clients download their data again. More concurrent connections also increase CPU and memory usage. - -The [S3 object storage configuration reference](/configuration/powersync-service/self-hosted-instances#param-object-storage) lists all supported settings, including timeouts, request concurrency, and the size threshold below which blocks stay in MongoDB. - -Keep the scheduled [compact](/maintenance-ops/compacting-buckets) job running. It removes files that are no longer needed. The `teardown` command deletes PowerSync's files under the prefix before it drops the storage database. The `powersync_object_storage_size_bytes` [metric](/maintenance-ops/self-hosting/monitoring) reports how much object storage PowerSync uses. diff --git a/sync/advanced/storage-version-4.mdx b/sync/advanced/storage-version-4.mdx index e2b71e31a..d09c12d8a 100644 --- a/sync/advanced/storage-version-4.mdx +++ b/sync/advanced/storage-version-4.mdx @@ -3,8 +3,173 @@ title: "Storage Version 4" description: "Opt in to storage version 4 for faster sync, incremental reprocessing of Sync Streams changes, and S3 object storage." --- -{/* Wrapper page: the content is snippets/sync-shared/storage-version-4.mdx, which also renders at sync/rules/storage-version-4.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/storage-version-4.mdx. It is a short summary that links here for details, so keep the anchors opt-in, incremental-reprocessing, s3-object-storage, and self-hosted-s3-setup stable. Mention Sync Rules here only to state that incremental reprocessing requires Sync Streams. */} -import StorageVersion4 from '/snippets/sync-shared/storage-version-4.mdx'; +Storage version 4 is a new version of the format the PowerSync Service uses to store the data it syncs to clients. It is in [Beta](/resources/feature-status) as of PowerSync Service v1.26.0. - +Compared to version 2, it provides: + +- Faster sync and faster reprocessing after a deployment. +- [Incremental reprocessing](#incremental-reprocessing): a Sync Streams deployment reprocesses only the streams you added or changed. Clients no longer download all their data again after every deployment. +- [S3 object storage](#s3-object-storage): larger blocks of synced data move from the storage database to S3. This reduces load on the storage database when many clients sync at once or sync large amounts of data. + +## Availability + +Storage version 4 is compatible with all PowerSync Cloud instances, which already use MongoDB [bucket storage](/architecture/powersync-service#bucket-storage). Self-hosted instances must also use MongoDB bucket storage. Postgres bucket storage is not currently supported. + +The PowerSync Cloud and self-hosted columns below apply during the Beta only. Once storage version 4 is generally available, it will become the default for all supported instances. S3 object storage is then also enabled on all PowerSync Cloud instances. For self-hosted deployments, follow the [S3 setup instructions](#self-hosted-s3-setup). + +| | Source database | Sync Config | PowerSync Cloud (Beta) | Self-hosted (Beta) | +| --- | --- | --- | --- | --- | +| Storage version 4 | Any | Sync Streams or Sync Rules | Free plan: automatic. Other plans: [opt in](#opt-in). | [Opt in](#opt-in) | +| Incremental reprocessing | MongoDB | Sync Streams | Included with version 4 | Included with version 4 | +| S3 object storage | Any | Sync Streams or Sync Rules | Enabled per instance by PowerSync on request | [Set up S3 object storage](#self-hosted-s3-setup) | + +Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/rules/migrate-to-sync-streams). + +## Opt In + +Version 4 is not the default in PowerSync Service v1.26.0. Moving a Sync Config to version 4 runs like any other deployment: + +1. PowerSync reprocesses all data selected by your Sync Config in the background. The current version keeps serving clients, so there is no downtime. +2. When the new copy is ready, PowerSync switches to it. On PowerSync Cloud, this appears as a new deploy event in the PowerSync Dashboard. +3. Clients download their data again once, as after any deployment before version 4. On self-hosted deployments with many clients, scale out the API before the switch to absorb the re-sync. + +After this first deployment, later Sync Streams deployments use incremental reprocessing automatically when your instance meets its requirements. There is no separate setting. + +### PowerSync Cloud + +Free plan instances are upgraded automatically during the Beta. No action is needed. + +On other plans, add `storage_version: 4` to the `config` block of your Sync Config and deploy it: + +```yaml +config: + edition: 3 + storage_version: 4 + +streams: + todos: + query: SELECT * FROM todos WHERE owner_id = auth.user_id() +``` + +### Self-Hosted + + + Postgres bucket storage is not supported with version 4. + + +Add `storage_version: 4` to the `config` block of each Sync Config as shown above, then deploy or redeploy it to use version 4. + +To move a Sync Config back to version 2, set `storage_version: 2` and deploy again. This is another full reprocess. + +To also enable S3 object storage, follow the [self-hosted S3 setup instructions](#self-hosted-s3-setup) to prepare a bucket and configure the Service. + +## Incremental Reprocessing + +Incremental reprocessing is active when you use a MongoDB source database, Sync Streams, and storage version 4. + + + Self-hosted instances with Postgres bucket storage are not supported. + + +Without it, every deployment reads all data selected by the Sync Config from your source database and prepares a complete new copy. Clients then download all their data again, even if only one stream changed. + +With incremental reprocessing, PowerSync compares the new Sync Config with the current one and reprocesses only the streams you added or changed. Unchanged streams keep their data on the PowerSync Service and on clients. Deployments finish faster, your source database does less work, and clients download only the data for affected streams they subscribe to. + +- Adding a stream reads only the data that stream selects. +- Removing a stream requires no new source reads. PowerSync cleans up stored definitions when no active Sync Config still uses them. +- Renaming a stream counts as removing it and adding a new one, so its data is rebuilt. +- Changing a stream's queries may reprocess affected definitions. Changes that only affect how request parameters select existing buckets do not require reprocessing. + +For example, changing `SELECT * FROM projects WHERE user_id = auth.user_id()` to `SELECT * FROM projects WHERE user_id = auth.jwt() ->> 'owner'` reuses the existing bucket data. The data is still grouped by `user_id`; only the JWT field used to select buckets changes. + +The time saved depends on how your data is split across streams. If one stream selects most of your data, changing that stream still takes about as long as a full reprocess. + +PowerSync favors correctness over reuse. When it cannot confirm that a change leaves a stream's data unchanged, it rebuilds that stream. A deployment that reprocesses more than you expect is not an error. + +Event definitions for [Custom Write Checkpoints](/handling-writes/custom-write-checkpoints) follow the same rules. Unchanged events keep their data, and new or changed events are read again. + +### When PowerSync Reprocesses Everything + +Some changes start a full reprocess, after which clients download all their data again: + +- The first deployment on storage version 4. +- Changes to the `config` block of the Sync Config, such as `edition`, compatibility fixes, or `storage_version`. +- The **Defragment** action in the PowerSync Dashboard, which exists to rebuild all data. See [Defragmenting](/maintenance-ops/compacting-buckets#defragmenting). +- Replication failures, for example when PowerSync loses its position in the MongoDB change stream and has to start over. + +### Sync Config Versions and Replication Streams + +Each deployment has a Sync Config version. With incremental reprocessing, multiple versions can share a replication stream, the replication process and stored state. A full reprocess creates a new replication stream. + +See the [Log Reference](/debugging/log-reference#message-prefixes) for how to identify these versions and streams in your logs. + +For implementation details, see the [storage design](https://github.com/powersync-ja/powersync-service/blob/main/docs/storage/storage-v3.md). The document describes the design introduced in version 3 and carried into version 4. + +### Checking What a Deployment Reprocessed + +If a deployment takes longer or reprocesses more than you expect, see [Checking What a Deployment Reprocessed](/debugging/log-reference#checking-what-a-deployment-reprocessed) in the Log Reference for what to look for in your logs. + +## S3 Object Storage + +Your instance keeps the data it syncs to clients in its bucket storage database, alongside everything else it needs to run. With S3 object storage, larger blocks of that data move to Amazon S3 or an S3-compatible object store, and the PowerSync Service syncs them to clients directly from there. Smaller blocks, and the metadata that locates each block, stay in MongoDB. + +Reading larger blocks from S3 reduces the data MongoDB must read and transfer during sync. When those reads limit performance, offloading them can speed up initial sync and let an instance serve more concurrent clients. The benefit is most noticeable when clients sync large amounts of data or many clients connect at once. The PowerSync Service still handles every client connection, so its CPU and memory capacity also limit concurrency. + +For self-hosted instances, offloading bucket data to S3 can reduce storage and data transfer costs. Compare the reduction in database costs with the object store's storage, request, and data transfer charges for your workload. + +Clients connect only to the PowerSync Service and never to the object store, so no client changes are needed. If the object store becomes unreachable, sync is interrupted until it recovers. Clients reconnect and resume automatically. + +S3 object storage requires storage version 4 and works with Sync Streams and legacy Sync Rules. It is compatible with all PowerSync Cloud instances. + + + S3 object storage holds PowerSync's internal sync data. To store files uploaded by your app, use [Attachments](/client-sdks/advanced/attachments). + + +### PowerSync Cloud + +During the Beta, PowerSync enables S3 object storage per instance. [Contact us](/resources/contact-us) if you want it on your instance before we enable it for all instances. + +### Self-Hosted S3 Setup + + + Self-hosted instances with Postgres bucket storage are not supported. + + + + + Create a bucket. Use the same region as the PowerSync Service where possible, to keep latency low and avoid cross-region data transfer charges. Use a dedicated bucket, or a unique `prefix` per PowerSync instance, so that instances never read or delete each other's files. Give the PowerSync Service permission to list the bucket and to read, write, and delete objects under the prefix. + + Leave object versioning off, or suspend it if the bucket already has it, and leave Object Lock off. PowerSync deletes files itself once they are no longer needed, so versioning keeps charging for old versions and locked objects cannot be cleaned up. Do not add an expiration lifecycle rule: an expired object may still be referenced by MongoDB, which breaks sync for that data. + + + Add `object_storage` to the `storage` section of `service.yaml`: + + ```yaml service.yaml + storage: + type: mongodb + uri: !env PS_MONGO_STORAGE_URI + object_storage: + type: s3 + bucket: powersync-bucket-data + region: us-east-1 + prefix: production + ``` + + Without `access_key_id` and `secret_access_key`, PowerSync uses the AWS credentials available to the process, such as an IAM role. For S3-compatible providers such as MinIO or Cloudflare R2, also set `endpoint`, and set `force_path_style: true` if the provider requires path-style requests. + + Restart or redeploy the PowerSync Service to load the updated `service.yaml`. If you run replication, API, and compacting in separate containers or jobs, apply the same object storage configuration to each. + + + Deploy your Sync Configs on storage version 4 as described in [Opt In](#opt-in). Sync Configs on version 2 keep all data in MongoDB, even when `object_storage` is configured. + + Once replication reaches a healthy checkpoint, confirm that objects appear under the prefix, run a test initial sync, and run `compact` once to surface permission errors early. + + + +After enabling S3 object storage, you can raise [`max_concurrent_connections`](/configuration/powersync-service/self-hosted-instances#param-max-concurrent-connections) from its default of 200 per API process. With storage version 4 and S3 object storage, each API process can handle up to 1,000 concurrent client connections. Performance degrades if a large share of those clients run an initial sync at the same time, so scale out the API before a deployment that makes all clients download their data again. More concurrent connections also increase CPU and memory usage. + +The [S3 object storage configuration reference](/configuration/powersync-service/self-hosted-instances#param-object-storage) lists all supported settings, including timeouts, request concurrency, and the size threshold below which blocks stay in MongoDB. + +Keep the scheduled [compact](/maintenance-ops/compacting-buckets) job running. It removes files that are no longer needed. The `teardown` command deletes PowerSync's files under the prefix before it drops the storage database. The `powersync_object_storage_size_bytes` [metric](/maintenance-ops/self-hosting/monitoring) reports how much object storage PowerSync uses. diff --git a/sync/rules/storage-version-4.mdx b/sync/rules/storage-version-4.mdx index e5e974de9..10d461f9d 100644 --- a/sync/rules/storage-version-4.mdx +++ b/sync/rules/storage-version-4.mdx @@ -1,15 +1,39 @@ --- -title: "Storage Version 4" -description: "Opt in to storage version 4 for faster sync, incremental reprocessing of Sync Streams changes, and S3 object storage." -noindex: true +title: "Storage Version 4 with Sync Rules" +sidebarTitle: "Storage Version 4" +description: "What storage version 4 and S3 object storage mean for instances that still use Sync Rules." --- -{/* Wrapper page: the content is snippets/sync-shared/storage-version-4.mdx, which also renders at sync/advanced/storage-version-4.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} - -import StorageVersion4 from '/snippets/sync-shared/storage-version-4.mdx'; +{/* Split page: the Sync Streams version of this page is sync/advanced/storage-version-4.mdx. Sync Rules are deprecated: keep this page accurate, but do not prioritize additions. This page stays short on purpose and links to the Sync Streams page for details, because incremental reprocessing requires Sync Streams. When you fix an error here, check whether the Sync Streams page needs the same fix. */} -Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. Configuration examples use Sync Streams syntax. +Sync Rules are deprecated. For the Sync Streams version of this page, see [Storage Version 4](/sync/advanced/storage-version-4). - +Storage version 4 is a new version of the format the PowerSync Service uses to store the data it syncs to clients. It is in [Beta](/resources/feature-status) as of PowerSync Service v1.26.0. Once it is generally available, it becomes the default for all supported instances. + +With Sync Rules, storage version 4 provides: + +- Faster sync and faster reprocessing after a deployment. +- [S3 object storage](/sync/advanced/storage-version-4#s3-object-storage): larger blocks of synced data move from the storage database to S3. This reduces load on the storage database when many clients sync at once or sync large amounts of data. During the Beta, PowerSync enables it per instance on PowerSync Cloud. Self-hosted instances follow the [S3 setup instructions](/sync/advanced/storage-version-4#self-hosted-s3-setup). + +[Incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing) requires Sync Streams. With Sync Rules, every deployment still reprocesses all data, and clients download all their data again. To use incremental reprocessing, [migrate to Sync Streams](/sync/rules/migrate-to-sync-streams). + +## Opt In + +Free plan instances on PowerSync Cloud are upgraded automatically during the Beta. On other plans, and on self-hosted instances, add `storage_version: 4` to the `config` block of your Sync Rules and deploy them: + +```yaml +config: + storage_version: 4 + +bucket_definitions: + user_lists: + parameters: SELECT request.user_id() AS user_id + data: + - SELECT * FROM lists WHERE owner_id = bucket.user_id +``` + +Self-hosted instances must use MongoDB [bucket storage](/architecture/powersync-service#bucket-storage). Postgres bucket storage is not supported with version 4. + +The first deployment on version 4 reprocesses all data in the background, and clients download their data again once. From 931cd60a64efc243585b4c8802fbbb6a3d652e43 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Mon, 21 Sep 2026 16:05:31 +0200 Subject: [PATCH 19/20] Other polish according to Claude --- snippets/sync-shared/client-id.mdx | 2 +- sync/advanced/compatibility.mdx | 2 +- sync/advanced/storage-version-4.mdx | 12 ++++++------ 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/snippets/sync-shared/client-id.mdx b/snippets/sync-shared/client-id.mdx index 7e88433b7..d1a2884ba 100644 --- a/snippets/sync-shared/client-id.mdx +++ b/snippets/sync-shared/client-id.mdx @@ -34,7 +34,7 @@ PowerSync does not perform any validation that IDs are unique. Duplicate IDs on 1. A non-unique column is used for the ID. 2. Multiple table partitions are used (Postgres), with the same ID present in different partitions. -3. Multiple data queries returning the same record. This is typically not an issue if the queries return the same values (same transformations used in each query). +3. Multiple queries returning the same record. This is typically not an issue if the queries return the same values (same transformations used in each query). We recommend using a unique index on the fields in the source database to ensure uniqueness — this will prevent (1) at least. diff --git a/sync/advanced/compatibility.mdx b/sync/advanced/compatibility.mdx index 44cc73686..f5d3fc8a8 100644 --- a/sync/advanced/compatibility.mdx +++ b/sync/advanced/compatibility.mdx @@ -154,7 +154,7 @@ If no option is given, the default precision depends on the source database: ### `versioned_bucket_ids` -Streams are compiled into buckets, and rows to sync are assigned to those buckets. When you run a full defragmentation or +Streams define buckets, and rows to sync are assigned to those buckets. When you run a full defragmentation or redeploy your Sync Config, the same bucket identifiers are re-used when processing data again. Because the second iteration uses different checksums for the same bucket ids, clients may sync data diff --git a/sync/advanced/storage-version-4.mdx b/sync/advanced/storage-version-4.mdx index d09c12d8a..ac263241b 100644 --- a/sync/advanced/storage-version-4.mdx +++ b/sync/advanced/storage-version-4.mdx @@ -19,11 +19,11 @@ Storage version 4 is compatible with all PowerSync Cloud instances, which alread The PowerSync Cloud and self-hosted columns below apply during the Beta only. Once storage version 4 is generally available, it will become the default for all supported instances. S3 object storage is then also enabled on all PowerSync Cloud instances. For self-hosted deployments, follow the [S3 setup instructions](#self-hosted-s3-setup). -| | Source database | Sync Config | PowerSync Cloud (Beta) | Self-hosted (Beta) | -| --- | --- | --- | --- | --- | -| Storage version 4 | Any | Sync Streams or Sync Rules | Free plan: automatic. Other plans: [opt in](#opt-in). | [Opt in](#opt-in) | -| Incremental reprocessing | MongoDB | Sync Streams | Included with version 4 | Included with version 4 | -| S3 object storage | Any | Sync Streams or Sync Rules | Enabled per instance by PowerSync on request | [Set up S3 object storage](#self-hosted-s3-setup) | +| | Source database | PowerSync Cloud (Beta) | Self-hosted (Beta) | +| --- | --- | --- | --- | +| Storage version 4 | Any | Free plan: automatic. Other plans: [opt in](#opt-in). | [Opt in](#opt-in) | +| Incremental reprocessing | MongoDB | Included with version 4 | Included with version 4 | +| S3 object storage | Any | Enabled per instance by PowerSync on request | [Set up S3 object storage](#self-hosted-s3-setup) | Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/rules/migrate-to-sync-streams). @@ -121,7 +121,7 @@ For self-hosted instances, offloading bucket data to S3 can reduce storage and d Clients connect only to the PowerSync Service and never to the object store, so no client changes are needed. If the object store becomes unreachable, sync is interrupted until it recovers. Clients reconnect and resume automatically. -S3 object storage requires storage version 4 and works with Sync Streams and legacy Sync Rules. It is compatible with all PowerSync Cloud instances. +S3 object storage requires storage version 4. It is compatible with all PowerSync Cloud instances. S3 object storage holds PowerSync's internal sync data. To store files uploaded by your app, use [Attachments](/client-sdks/advanced/attachments). From 263482ba78559fdaa8038818956d5b5656c81a56 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Tue, 22 Sep 2026 11:36:29 +0200 Subject: [PATCH 20/20] Remove sync rules mentions from guides --- .../advanced/custom-types-arrays-and-json.mdx | 114 +++++------------- client-sdks/advanced/gis-data-postgis.mdx | 33 ++--- client-sdks/advanced/pre-seeded-sqlite.mdx | 35 ++---- client-sdks/advanced/raw-tables.mdx | 2 +- .../advanced/sequential-id-mapping.mdx | 44 +++---- client-sdks/frameworks/expo-go-support.mdx | 2 +- client-sdks/infinite-scrolling.mdx | 10 +- .../custom-conflict-resolution.mdx | 45 +++---- integrations/neon.mdx | 77 ++++-------- integrations/serverpod.mdx | 2 +- integrations/supabase/guide.mdx | 43 ++----- .../supabase/rls-and-sync-streams.mdx | 10 +- maintenance-ops/self-hosting/aws-ecs.mdx | 2 +- maintenance-ops/self-hosting/coolify.mdx | 2 +- migration-guides/atlas-device-sync.mdx | 2 +- migration-guides/electric.mdx | 2 +- snippets/postgres-powersync-publication.mdx | 2 +- sync/rules/client-parameters.mdx | 2 +- 18 files changed, 138 insertions(+), 291 deletions(-) diff --git a/client-sdks/advanced/custom-types-arrays-and-json.mdx b/client-sdks/advanced/custom-types-arrays-and-json.mdx index edca592fe..ad163c5bc 100644 --- a/client-sdks/advanced/custom-types-arrays-and-json.mdx +++ b/client-sdks/advanced/custom-types-arrays-and-json.mdx @@ -7,7 +7,7 @@ PowerSync supports JSON/JSONB and array columns. They are synced as JSON text an ## JSON and JSONB -The PowerSync Service treats JSON and JSONB columns as text and provides many helpers for working with JSON in [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)). +The PowerSync Service treats JSON and JSONB columns as text and provides many helpers for working with JSON in [Sync Streams](/sync/streams/overview). **Note:** Native Postgres arrays, JSON arrays, and JSONB arrays are effectively all equivalent in PowerSync. @@ -22,36 +22,20 @@ ADD COLUMN custom_payload json; ### Sync Streams - - - PowerSync treats JSON columns as text. Use `json_extract()` and other JSON functions in stream queries. Subscribe per list to sync only that list's todos: - - ```yaml - config: - edition: 3 - streams: - my_json_todos: - auto_subscribe: true - with: - owned_lists: SELECT id AS list_id FROM lists WHERE owner_id = auth.user_id() - query: SELECT * FROM todos WHERE json_extract(custom_payload, '$.json_list') IN owned_lists - ``` +PowerSync treats JSON columns as text. Use `json_extract()` and other JSON functions in stream queries. Subscribe per list to sync only that list's todos: + +```yaml +config: + edition: 3 +streams: + my_json_todos: + auto_subscribe: true + with: + owned_lists: SELECT id AS list_id FROM lists WHERE owner_id = auth.user_id() + query: SELECT * FROM todos WHERE json_extract(custom_payload, '$.json_list') IN owned_lists +``` - The client subscribes once per list (e.g. `db.syncStream('my_json_todos', { list_id: listId }).subscribe()`). - - - PowerSync treats JSON columns as text and provides transformation functions in Sync Rules such as `json_extract()`. - - ```yaml - bucket_definitions: - my_json_todos: - # Separate bucket per To-Do list - parameters: SELECT id AS list_id FROM lists WHERE owner_id = request.user_id() - data: - - SELECT * FROM todos WHERE json_extract(custom_payload, '$.json_list') = bucket.list_id - ``` - - +The client subscribes once per list (e.g. `db.syncStream('my_json_todos', { list_id: listId }).subscribe()`). ### Client SDK @@ -198,7 +182,7 @@ You can write the entire updated column value as a string, or, with `trackPrevio PowerSync treats array columns as JSON text. This means that the SQLite JSON operators can be used on any array columns. -Additionally, array membership is supported in [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)) so you can sync rows based on whether a parameter value appears in an array column. +Additionally, array membership is supported in [Sync Streams](/sync/streams/overview) so you can sync rows based on whether a parameter value appears in an array column. **Note:** Native Postgres arrays, JSON arrays, and JSONB arrays are effectively all equivalent in PowerSync. @@ -219,33 +203,17 @@ Array columns are converted to text by the PowerSync Service. A text array as de **Array Membership** - - - Sync rows where a subscription parameter value is in the row's array column using `IN`: - - ```yaml - config: - edition: 3 - streams: - custom_todos: - query: SELECT * FROM todos WHERE subscription.parameter('list_id') IN unique_identifiers - ``` +Sync rows where a subscription parameter value is in the row's array column using `IN`: - The client subscribes per list (e.g. `db.syncStream('custom_todos', { list_id: listId }).subscribe()`). - - - It's possible to sync rows dynamically based on the contents of array columns using the `IN` operator: - - ```yaml - bucket_definitions: - custom_todos: - # Separate bucket per To-Do list - parameters: SELECT id AS list_id FROM lists WHERE owner_id = request.user_id() - data: - - SELECT * FROM todos WHERE bucket.list_id IN unique_identifiers - ``` - - +```yaml +config: + edition: 3 +streams: + custom_todos: + query: SELECT * FROM todos WHERE subscription.parameter('list_id') IN unique_identifiers +``` + +The client subscribes per list (e.g. `db.syncStream('custom_todos', { list_id: listId }).subscribe()`). See these additional details when using the `IN` operator: [Operators](/sync/supported-sql#operators) @@ -416,31 +384,15 @@ create type location_address AS ( ### Sync Streams - - - The custom type column is serialized as JSON and you can use `json_extract()` and other JSON functions in stream queries: - - ```yaml - config: - edition: 3 - streams: - todos_by_city: - query: SELECT * FROM todos WHERE json_extract(location, '$.city') = subscription.parameter('city') - ``` - - - Custom type columns are converted to text by the PowerSync Service. - Depending on whether the `custom_postgres_types` [compatibility option](/sync/advanced/compatibility) is enabled, - PowerSync would sync the row as: - - - `{"street":"1000 S Colorado Blvd.","city":"Denver","state":"CO","zip":80211}` if the option is enabled. - - `("1000 S Colorado Blvd.",Denver,CO,80211)` if the option is disabled. +The custom type column is serialized as JSON and you can use `json_extract()` and other JSON functions in stream queries: - You can use regular string and JSON manipulation functions in Sync Rules. This means that individual values of the type - can be synced with `json_extract` if the `custom_postgres_types` compatibility option is enabled. - Without the option, the entire column must be synced as text. - - +```yaml +config: + edition: 3 +streams: + todos_by_city: + query: SELECT * FROM todos WHERE json_extract(location, '$.city') = subscription.parameter('city') +``` ### Client SDK diff --git a/client-sdks/advanced/gis-data-postgis.mdx b/client-sdks/advanced/gis-data-postgis.mdx index f3ac3fd7d..7d0ccef38 100644 --- a/client-sdks/advanced/gis-data-postgis.mdx +++ b/client-sdks/advanced/gis-data-postgis.mdx @@ -115,7 +115,7 @@ The data looks exactly how it’s stored in the Postgres database i.e. Example use case: Extract x (long) and y (lat) values from a PostGIS type, to use these values independently in an application. -PowerSync supports the following PostGIS functions in Sync Streams (or legacy Sync Rules): [Operators and Functions](/sync/supported-sql#functions) +PowerSync supports the following PostGIS functions in Sync Streams: [Operators and Functions](/sync/supported-sql#functions) 1. `ST_AsGeoJSON` 2. `ST_AsText` @@ -126,25 +126,12 @@ PowerSync supports the following PostGIS functions in Sync Streams (or legacy Sy IMPORTANT NOTE: These functions will only work if your Postgres instance has the PostGIS extension installed and you’re storing values as type `geography` or `geometry`. - - - ```yaml - config: - edition: 3 - streams: - global: - queries: - - SELECT * FROM lists - - SELECT *, st_x(location) as longitude, st_y(location) as latitude FROM todos - ``` - - - ```yaml - bucket_definitions: - global: - data: - - SELECT * FROM lists - - SELECT *, st_x(location) as longitude, st_y(location) as latitude from todos - ``` - - +```yaml +config: + edition: 3 +streams: + global: + queries: + - SELECT * FROM lists + - SELECT *, st_x(location) as longitude, st_y(location) as latitude FROM todos +``` diff --git a/client-sdks/advanced/pre-seeded-sqlite.mdx b/client-sdks/advanced/pre-seeded-sqlite.mdx index b431f7687..dbaaefab5 100644 --- a/client-sdks/advanced/pre-seeded-sqlite.mdx +++ b/client-sdks/advanced/pre-seeded-sqlite.mdx @@ -20,34 +20,19 @@ If you're interested in seeing an end-to-end example, we've prepared a demo repo ## Main Concepts ### Generate a Scoped JWT Token -In most cases you'd want to pre-seed the SQLite database with user specific data and not all data from the source database, as you normally would when using PowerSync. For this you would need to generate JWT tokens that include the necessary properties to satisfy the conditions of the queries in your Sync Streams (or legacy Sync Rules). +In most cases you'd want to pre-seed the SQLite database with user specific data and not all data from the source database, as you normally would when using PowerSync. For this you would need to generate JWT tokens that include the necessary properties to satisfy the conditions of the queries in your Sync Streams. Let's say we have the following Sync Config: - - - ```yaml - sync_config: - content: | - config: - edition: 3 - streams: - store_products: - query: SELECT * FROM products WHERE store_id = auth.parameter('store_id') - ``` - - - ```yaml - sync_config: - content: | - bucket_definitions: - store_products: - parameters: SELECT id as store_id FROM stores WHERE id = request.jwt() ->> 'store_id' - data: - - SELECT * FROM products WHERE store_id = bucket.store_id - ``` - - +```yaml +sync_config: + content: | + config: + edition: 3 + streams: + store_products: + query: SELECT * FROM products WHERE store_id = auth.parameter('store_id') +``` In the example above the `store_id` is part of the JWT payload and is used to filter products by store for a user. Given this we would want to do the following: 1. Query the source database, directly from the Node.js application, for all the store ids you'd want a pre-seeded SQLite database for. diff --git a/client-sdks/advanced/raw-tables.mdx b/client-sdks/advanced/raw-tables.mdx index 44fdb9d8a..636a32a65 100644 --- a/client-sdks/advanced/raw-tables.mdx +++ b/client-sdks/advanced/raw-tables.mdx @@ -761,7 +761,7 @@ In PowerSync's [JSON-based view system](/architecture/client-architecture#client ### Adding Raw Tables as a New Table -When you're adding new tables to your Sync Streams (or legacy Sync Rules), clients will start to sync data on those tables - even if the tables aren't mentioned in the client's schema yet. So at the time you're introducing a new raw table to your app, it's possible that PowerSync has already synced some data for that table, which would be stored in `ps_untyped`. When adding regular tables, PowerSync will automatically extract rows from `ps_untyped`. With raw tables, that step is your responsibility. To copy data, run these statements in a transaction after creating the table: +When you're adding new tables to your Sync Streams, clients will start to sync data on those tables - even if the tables aren't mentioned in the client's schema yet. So at the time you're introducing a new raw table to your app, it's possible that PowerSync has already synced some data for that table, which would be stored in `ps_untyped`. When adding regular tables, PowerSync will automatically extract rows from `ps_untyped`. With raw tables, that step is your responsibility. To copy data, run these statements in a transaction after creating the table: ``` INSERT INTO my_table (id, my_column, ...) diff --git a/client-sdks/advanced/sequential-id-mapping.mdx b/client-sdks/advanced/sequential-id-mapping.mdx index d3d5aa57f..632c4324e 100644 --- a/client-sdks/advanced/sequential-id-mapping.mdx +++ b/client-sdks/advanced/sequential-id-mapping.mdx @@ -29,7 +29,7 @@ Before we get started, let's outline the changes we will have to make: - Update your Sync Streams (or legacy Sync Rules) to use the UUID column instead of the integer ID. + Update your Sync Streams to use the UUID column instead of the integer ID. @@ -183,40 +183,24 @@ We will create the following two triggers that cover either scenario of updating We now have triggers in place that will handle the mapping for our updated schema and -can move on to updating your Sync Streams/Sync Rules to use the UUID column instead of the integer ID. +can move on to updating your Sync Streams to use the UUID column instead of the integer ID. ## Update Sync Streams As sequential IDs can only be created on the backend source database, we need to use UUIDs in the client. The Sync Config is updated to use the `uuid` column as the `id` column for the `lists` and `todos` tables, explicitly defining which columns to select so that `list_id` (the integer ID) is no longer exposed to the client. - - - ```yaml - config: - edition: 3 - streams: - user_lists: - auto_subscribe: true - with: - user_lists_param: SELECT id FROM lists WHERE owner_id = auth.user_id() - queries: - - "SELECT lists.uuid AS id, lists.created_at, lists.name, lists.owner_id FROM lists WHERE lists.id IN user_lists_param" - - "SELECT todos.uuid AS id, todos.created_at, todos.completed_at, todos.description, todos.completed, todos.created_by, todos.list_uuid FROM todos WHERE todos.list_id = user_lists_param" - ``` - - - ```yaml sync-config.yaml {4, 7-8} - bucket_definitions: - user_lists: - # Separate bucket per todo list - parameters: select id from lists where owner_id = request.user_id() - data: - # Explicitly define all the columns - - select uuid as id, created_at, name, owner_id from lists where id = bucket.id - - select uuid as id, created_at, completed_at, description, completed, created_by, list_uuid from todos where list_id = bucket.id - ``` - - +```yaml +config: + edition: 3 +streams: + user_lists: + auto_subscribe: true + with: + user_lists_param: SELECT id FROM lists WHERE owner_id = auth.user_id() + queries: + - "SELECT lists.uuid AS id, lists.created_at, lists.name, lists.owner_id FROM lists WHERE lists.id IN user_lists_param" + - "SELECT todos.uuid AS id, todos.created_at, todos.completed_at, todos.description, todos.completed, todos.created_by, todos.list_uuid FROM todos WHERE todos.list_id = user_lists_param" +``` We can now move on to updating the client to use UUIDs. diff --git a/client-sdks/frameworks/expo-go-support.mdx b/client-sdks/frameworks/expo-go-support.mdx index 00980919b..2f658c08d 100644 --- a/client-sdks/frameworks/expo-go-support.mdx +++ b/client-sdks/frameworks/expo-go-support.mdx @@ -109,7 +109,7 @@ export default function HomeScreen() { After adding PowerSync to your app: -1. [**Define what data to sync by setting up Sync Rules**](/sync/rules/overview) +1. [**Define what data to sync by setting up Sync Streams**](/sync/streams/overview) 2. [**Implement your SQLite client schema**](/client-sdks/reference/react-native-and-expo#1-define-the-client-side-schema) 3. [**Connect to PowerSync and your backend**](/client-sdks/reference/react-native-and-expo#3-integrate-with-your-backend) diff --git a/client-sdks/infinite-scrolling.mdx b/client-sdks/infinite-scrolling.mdx index 591bf382a..b5f9d2ed4 100644 --- a/client-sdks/infinite-scrolling.mdx +++ b/client-sdks/infinite-scrolling.mdx @@ -17,13 +17,11 @@ This means that in many cases, you can sync a sufficient amount of data to let a | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | It works offline and is low-latency (data loads quickly from the local database). We don't need to load data from the backend via the network when the user reaches the bottom of the page/feed/list. | There will be cases where this approach won't work because the total volume of data might become too large for the local database - for example, when there's a wide range of tables that the user needs to be able to infinite scroll. Your app allows the user to apply filters to the displayed data, which results in fewer pages displayed from a large dataset, and therefore limited scrolling. | -### 2) Control data sync using subscription or client parameters +### 2) Control data sync using subscription parameters -**Sync Streams** (recommended): Use [subscription parameters](/sync/streams/parameters#subscription-parameters) to subscribe to specific data on demand. For example, a client can subscribe to a specific "page" of data when the user scrolls to it. This is more flexible than client parameters — each subscription is independent and multiple tabs/views can subscribe with different parameters simultaneously. +Use [subscription parameters](/sync/streams/parameters#subscription-parameters) to subscribe to specific data on demand. For example, a client can subscribe to a specific "page" of data when the user scrolls to it. Each subscription is independent, so multiple tabs or views can subscribe with different parameters at the same time. Subscription parameters come from the client, so use them to select data, not for access control. Keep filtering by [auth parameters](/sync/streams/parameters#auth-parameters) such as `auth.user_id()`. -**Sync Rules** (legacy): PowerSync supports the use of [client parameters](/sync/rules/client-parameters) which are specified directly by the client. The app can dynamically change these parameters on the client-side and they can be accessed in Sync Rules on the server-side. The developer can use these parameters to limit/control which data is synced, but since they are not trusted (because they are not passed via the JWT authentication token) they should not be used for access control. You should still filter data by e.g. user ID for access control purposes (using [token parameters](/sync/rules/parameter-queries) from the JWT). - -Usage example: To lazy-load/lazy-sync data for infinite scrolling, you could split your data into 'pages' and use a subscription parameter (Sync Streams) or client parameter (Sync Rules) to specify which pages to sync to a user. +Usage example: To lazy-load/lazy-sync data for infinite scrolling, you could split your data into 'pages' and use a subscription parameter to specify which pages to sync to a user. | Pros | Cons | | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | @@ -39,7 +37,7 @@ In this scenario we can sync a smaller number of rows to the user initially. If ### 4) Client-side triggers a server-side function to flag data to sync -You could add a flag to certain records in your backend source database which are used by your [Sync Streams](/sync/streams/overview) or [Sync Rules](/sync/rules/overview) to determine which records to sync to specific users. Then your app could make an API call which triggers a function that updates the flags on certain records, causing more records to be synced to the user. +You could add a flag to certain records in your backend source database which are used by your [Sync Streams](/sync/streams/overview) to determine which records to sync to specific users. Then your app could make an API call which triggers a function that updates the flags on certain records, causing more records to be synced to the user. ## Questions? diff --git a/handling-writes/custom-conflict-resolution.mdx b/handling-writes/custom-conflict-resolution.mdx index ce369b90c..b5e932608 100644 --- a/handling-writes/custom-conflict-resolution.mdx +++ b/handling-writes/custom-conflict-resolution.mdx @@ -34,7 +34,7 @@ When data changes on the server: 1. **Source database updates** - Direct writes or changes from other clients 2. **PowerSync Service detects changes** - Through replication stream -3. **Clients download updates** - Based on their Sync Streams (or legacy Sync Rules) +3. **Clients download updates** - Based on their Sync Streams 4. **Local SQLite updates** - Changes merge into the client's database **Conflicts arise when**: Multiple clients modify the same row (or fields) before syncing, or when a client's changes conflict with server-side rules. @@ -505,32 +505,17 @@ CREATE TABLE write_conflicts ( ### Step 2: Sync Conflicts to Clients -**Sync Streams / Sync Rules:** - - - - ```yaml - config: - edition: 3 - streams: - user_data: - queries: - - SELECT * FROM tasks WHERE user_id = auth.user_id() - - SELECT * FROM write_conflicts WHERE user_id = auth.user_id() AND NOT resolved - ``` - - - ```yaml - bucket_definitions: - user_data: - parameters: - - SELECT request.user_id() as user_id - data: - - SELECT * FROM tasks WHERE user_id = bucket.user_id - - SELECT * FROM write_conflicts WHERE user_id = bucket.user_id AND resolved = FALSE - ``` - - +**Sync Streams:** + +```yaml +config: + edition: 3 +streams: + user_data: + queries: + - SELECT * FROM tasks WHERE user_id = auth.user_id() + - SELECT * FROM write_conflicts WHERE user_id = auth.user_id() AND NOT resolved +``` ### Step 3: Record Conflicts in Backend @@ -863,7 +848,7 @@ For scenarios where you just need to record changes without tracking their statu How it works: - Mark the table as `insertOnly: true` in your client schema -- Don't include the `field_changes` table in your Sync Rules +- Don't include the `field_changes` table in your Sync Streams - Changes are uploaded to the server but never downloaded back to clients **Client schema:** @@ -893,7 +878,7 @@ For scenarios where you want to show sync status temporarily but don't need a pe How it works: - Use a normal table on the client (not `insertOnly`) -- Don't include the `field_changes` table in your Sync Rules +- Don't include the `field_changes` table in your Sync Streams - Pending changes stay on the client until they're uploaded and the server processes them - Once the server processes a change and PowerSync syncs the next checkpoint, the change automatically disappears from the client @@ -932,7 +917,7 @@ function SyncIndicator({ taskId }: { taskId: string }) { **When to use:** Showing "syncing..." indicators, temporary status tracking without long-term storage overhead, cases where you want automatic cleanup after sync. -**Tradeoff:** Can't show detailed server-side error messages (unless the server writes to a separate errors table that *is* in Sync Rules). No long-term history on the client. +**Tradeoff:** Can't show detailed server-side error messages (unless the server writes to a separate errors table that *is* in Sync Streams). No long-term history on the client. ## Strategy 7: Cumulative Operations (Inventory) diff --git a/integrations/neon.mdx b/integrations/neon.mdx index 554d57f77..2230273bf 100644 --- a/integrations/neon.mdx +++ b/integrations/neon.mdx @@ -137,65 +137,40 @@ PowerSync uses logical replication to sync data from your Neon database. ### Configure Sync Streams -[Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)) allow developers to control which data gets synced to which user devices using a SQL-like syntax in a YAML file. For the demo app, we're going to specify that each user can only see their own notes (plus any shared notes). +[Sync Streams](/sync/streams/overview) allow developers to control which data gets synced to which user devices using a SQL-like syntax in a YAML file. For the demo app, we're going to specify that each user can only see their own notes (plus any shared notes). -1. In the PowerSync Dashboard, select your project and instance and go to the **Sync Streams** view (shown as **Sync Rules** if using legacy Sync Rules). +1. In the PowerSync Dashboard, select your project and instance and go to the **Sync Streams** view. 2. Edit the Sync Config in the editor and replace the contents with the below: - - - ```yaml - config: - edition: 3 - - streams: - user_notes: - auto_subscribe: true - # Sync notes and paragraphs belonging to the authenticated user - queries: - - SELECT * FROM notes WHERE owner_id = auth.user_id() - - SELECT paragraphs.* FROM paragraphs - INNER JOIN notes ON notes.id = paragraphs.note_id - WHERE notes.owner_id = auth.user_id() - shared_notes: - auto_subscribe: true - # Sync all shared notes to all users (not recommended for production) - queries: - - SELECT * FROM notes WHERE shared = TRUE - - SELECT paragraphs.* FROM paragraphs - INNER JOIN notes ON notes.id = paragraphs.note_id - WHERE notes.shared = TRUE - ``` - - - ```yaml - config: - edition: 2 - - bucket_definitions: - by_user: - # Only sync rows belonging to the user - parameters: SELECT id as note_id FROM notes WHERE owner_id = request.user_id() - data: - - SELECT * FROM notes WHERE id = bucket.note_id - - SELECT * FROM paragraphs WHERE note_id = bucket.note_id - # Sync all shared notes to all users (not recommended for production) - shared_notes: - parameters: SELECT id as note_id from notes where shared = TRUE - data: - - SELECT * FROM notes WHERE id = bucket.note_id - - SELECT * FROM paragraphs WHERE note_id = bucket.note_id - ``` - - +```yaml +config: + edition: 3 + +streams: + user_notes: + auto_subscribe: true + # Sync notes and paragraphs belonging to the authenticated user + queries: + - SELECT * FROM notes WHERE owner_id = auth.user_id() + - SELECT paragraphs.* FROM paragraphs + INNER JOIN notes ON notes.id = paragraphs.note_id + WHERE notes.owner_id = auth.user_id() + shared_notes: + auto_subscribe: true + # Sync all shared notes to all users (not recommended for production) + queries: + - SELECT * FROM notes WHERE shared = TRUE + - SELECT paragraphs.* FROM paragraphs + INNER JOIN notes ON notes.id = paragraphs.note_id + WHERE notes.shared = TRUE +``` 3. Click **"Validate"** and ensure there are no errors. This validates your Sync Config against your Postgres database. 4. Click **"Deploy"** to deploy your Sync Config. - For additional information on PowerSync's Sync Streams, refer to the [Sync Streams](/sync/streams/overview) documentation. -- For legacy Sync Rules, refer to the [Sync Rules](/sync/rules/overview) documentation. ## Test Everything (Using Our Demo App) @@ -239,11 +214,11 @@ Once signed in to the demo app, you should see a blank list of notes, so go ahea ### Test Sync (Optional) -During development, you can use the **Sync Test** feature in the PowerSync Dashboard to validate your Sync Rules: +During development, you can use the **Sync Test** feature in the PowerSync Dashboard to validate your Sync Streams: 1. Click on **"Sync Test"** in the PowerSync Dashboard. 2. Enter the UUID of a user in your Neon Auth database to generate a test JWT. -3. Click **"Launch Sync Diagnostics Client"** to test the Sync Rules. +3. Click **"Launch Sync Diagnostics Client"** to test the Sync Streams. For more information, explore the [PowerSync docs](/) or join us on [our community Discord](https://discord.gg/powersync) where our team is always available to answer questions. diff --git a/integrations/serverpod.mdx b/integrations/serverpod.mdx index 05f404b9b..029208004 100644 --- a/integrations/serverpod.mdx +++ b/integrations/serverpod.mdx @@ -265,7 +265,7 @@ For security, it is crucial each user only has access to their own bucket. This 1. When a client connects to PowerSync, it fetches an authentication token from your Serverpod instance. 2. Your Dart backend logic returns a JWT describing what data the user should have access to. -3. In the `sync_rules` section, you reference properties of the created JWTs to control data visible to the connecting clients. +3. In the `sync_config` section, you reference properties of the created JWTs to control data visible to the connecting clients. In this guide, we will use a single virtual user for everything. For real projects, follow [Serverpod documentation on authentication](https://docs.serverpod.dev/concepts/authentication/setup). diff --git a/integrations/supabase/guide.mdx b/integrations/supabase/guide.mdx index 5ce4b29b3..55dfe23fb 100644 --- a/integrations/supabase/guide.mdx +++ b/integrations/supabase/guide.mdx @@ -9,7 +9,7 @@ import SupabaseConnection from '/snippets/supabase-database-connection.mdx'; import PostgresPowerSyncUser from '/snippets/postgres-powersync-user.mdx'; import PostgresPowerSyncPublication from '/snippets/postgres-powersync-publication.mdx'; -This guide shows you how to configure PowerSync with [Supabase](https://supabase.com/). PowerSync syncs selected Supabase Postgres data into client-side SQLite. Client writes (mutations) are commited locally, queued by the SDK, and uploaded via Supabase's client libraries. Your app reads and writes from this local SQLite, so it stays highly responsive even in poor network conditions. +This guide shows you how to configure PowerSync with [Supabase](https://supabase.com/). PowerSync syncs selected Supabase Postgres data into client-side SQLite. Client writes (mutations) are committed locally, queued by the SDK, and uploaded via Supabase's client libraries. Your app reads and writes from this local SQLite, so it stays highly responsive even in poor network conditions.