diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 17b261d2..d28b039d 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -142,18 +142,18 @@ Update `docs.json` when adding, moving, or removing pages. Add redirects for mov ## Sync Streams and Sync Rules -Sync Streams are the default for new documentation. Keep legacy Sync Rules documentation accurate, but do not add new Sync Rules teaching, examples, or proactive references. +Sync Rules are deprecated, but many customers still run them. New documentation covers Sync Streams. Keep the Sync Rules docs accurate, and add to them when that makes them more accurate or helpful, without prioritizing that work. The goal is to avoid noisy Sync Rules references outside their own section, not to stop maintaining it. -When existing content shows both in tabs, preserve equivalent results and filters. Do not add new parallel Sync Rules examples. - -For existing prose that mentions both, use "[Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview))" once per page or major section. Later mentions should omit Sync Rules. +- **Sync Rules section.** `sync/rules/` and `sync/grammar/sync-rules/` hold all Sync Rules content. Each page opens with an `` callout that starts "Sync Rules are deprecated." A split twin or concept page then links to its Sync Streams equivalent. A shared-snippet wrapper instead states that the page applies to both Sync Streams and Sync Rules unless a section says otherwise. The overview carries the full deprecation notice in a ``. The sidebar group stays "Sync Rules (Legacy)". +- **Keep the engines apart.** Never place Sync Streams and Sync Rules content side by side: no engine tabs, no "(or legacy Sync Rules)" asides, no pointers to a Sync Rules equivalent. Outside `sync/rules/`, mention Sync Rules only to state a behavior difference that affects Sync Rules users, and remove other mentions when you edit a page. +- **Shared pages.** A page that applies to both engines keeps one body in `snippets/sync-shared/.mdx`, imported by a Sync Streams wrapper at the original path and a Sync Rules wrapper at `sync/rules/.mdx` that adds the callout and `noindex: true`. Edit the snippet, not the wrappers, and keep it valid for both engines. +- **Counterpart comments.** Every split twin, wrapper, and shared snippet starts with an MDX comment naming its counterpart. Read it before editing, apply a fix to both sides where content is shared, and keep the comment when restructuring. ## Verification - Verify technical claims and run code examples before publication. Select other checks appropriate to the change. - Run `vale ` for changed MDX pages. Add new technical terms to `.github/vale/config/vocabularies/PowerSync/accept.txt`; do not add ordinary misspellings. -- After link or navigation changes, run `npx mintlify broken-links`. Mintlify requires Node 20.17–24; if needed, use `PATH="/opt/homebrew/opt/node@24/bin:$PATH" npx mintlify broken-links`. -- For anchor and snippet checks, use `pnpm check:links`. Validate repository instruction links as file paths, since the site checker does not cover all of them. +- After link or navigation changes, run `pnpm check:links`. It runs the Mintlify path, anchor, and snippet checks through `scripts/check-links.mjs`, resolves anchors defined in imported snippets, and is the same check CI runs. It tells you if your Node version is unsupported. Validate repository instruction links as file paths, since the site checker does not cover them. - Use [the lint command](commands/lint-docs.md) for the check workflow and [the reviewer](agents/document-reviewer.md) for editorial review. Passing linters does not establish technical accuracy or style compliance. ## Git Workflow diff --git a/.claude/agents/document-reviewer.md b/.claude/agents/document-reviewer.md index 06f904ea..74464fbc 100644 --- a/.claude/agents/document-reviewer.md +++ b/.claude/agents/document-reviewer.md @@ -17,6 +17,7 @@ Apply the canonical standards in three passes and report findings from each: 1. **Accuracy:** claims, platform scope, versions, and consistency with the surrounding page. Before reporting a claim as unverified, check the sources the PR or the user cites, such as the source PR, divergence issue, release notes, or code at the merged commit, and follow the links inside them. If nothing is cited, look up the release the text names. Report a claim as unverified only when no source covers it or a source contradicts it. Give evidence, do not invent problems, and do not approve unverified claims as correct. 2. **Necessity:** list every sentence that describes what the product prints, displays, logs, or says in an error. Treat each one as a finding to remove unless it passes the restating rule in [Content Strategy](../CLAUDE.md#content-strategy), and report it even when the sentence is accurate. Describing visible output is not a mechanism, consequence, or signal. Also flag internal mechanics, rare exceptions, and repetition. Flag missing context only when readers need it, and do not require every entry to explain a mechanism, consequence, signal, action, and trade-off. 3. **Clarity and format:** plain technical English, clear actors and actions, and suitable examples and components. +4. **Sync Rules containment:** flag any Sync Rules mention, example, or tab outside `sync/rules/` and `snippets/sync-shared/` that does not state a behavior difference. Check that split twins, wrappers, and shared snippets keep their counterpart comment. ## Default Output diff --git a/.claude/commands/lint-docs.md b/.claude/commands/lint-docs.md index e0ac914c..ca902738 100644 --- a/.claude/commands/lint-docs.md +++ b/.claude/commands/lint-docs.md @@ -1,9 +1,9 @@ --- -allowed-tools: Read, Bash(npx mintlify *), Bash(PATH=* npx mintlify *), Bash(vale *), Bash(git diff *), Bash(git status *) -description: Run Vale and Mintlify broken-link checks for documentation changes and report failures. +allowed-tools: Read, Bash(pnpm check:links), Bash(PATH=* pnpm check:links), Bash(vale *), Bash(git diff *), Bash(git status *) +description: Run Vale and the link check for documentation changes and report failures. --- -1. Read the canonical [Verification](../CLAUDE.md#verification) section for commands, supported Node versions, and vocabulary rules. +1. Read the canonical [Verification](../CLAUDE.md#verification) section for commands and vocabulary rules. 2. Use the requested file scope. Otherwise, identify changed MDX pages with `git diff main --name-only --diff-filter=ACMR -- '*.mdx'` and `git status --short`, including untracked pages. -3. Run `vale ` for each page and `npx mintlify broken-links` for the site. Use the canonical Node fallback if needed. +3. Run `vale ` for each page and `pnpm check:links` once for the site. 4. Report findings by file, suggested fixes, and totals for errors, warnings, and suggestions. Report failed or unavailable checks separately from content findings. diff --git a/.claude/skills/doc-author/SKILL.md b/.claude/skills/doc-author/SKILL.md index 6ac236dc..c2201dbb 100644 --- a/.claude/skills/doc-author/SKILL.md +++ b/.claude/skills/doc-author/SKILL.md @@ -18,6 +18,6 @@ Use the canonical Working Process for scope changes and unresolved decisions. 1. Identify the reader, desired outcome, and affected feature or concept. 2. Research the implementation and existing coverage. Read the most relevant related pages and `docs.json`; avoid unnecessary duplication. 3. If a plan is needed, state the proposed pages, structure, and unresolved questions before drafting. -4. Write the update under the canonical standards. Keep the existing structure unless the task requires a change. +4. Write the update under the canonical standards. Write new content for Sync Streams and keep Sync Rules content inside its own section. Keep the existing structure unless the task requires a change. 5. Self-review for accuracy, reader understanding, minimum useful detail, and navigation fit. Run the canonical verification checks relevant to the change. 6. Present the result and any unresolved draft TODOs, or complete the delivery workflow already authorized by the user. diff --git a/.claude/skills/pr-to-docs/SKILL.md b/.claude/skills/pr-to-docs/SKILL.md index d5203a4a..b8556e12 100644 --- a/.claude/skills/pr-to-docs/SKILL.md +++ b/.claude/skills/pr-to-docs/SKILL.md @@ -33,4 +33,4 @@ Ask before expanding scope, documenting a deprecation that needs migration decis ## 4. Draft and Verify -Apply the canonical writing standards, navigation requirements, and verification checks. Preserve existing structure unless restructuring is part of the approved plan. Flag unresolved facts using the canonical draft-TODO convention and report what must be resolved before publication. +Apply the canonical writing standards, navigation requirements, and verification checks. Document new behavior for Sync Streams. Update Sync Rules pages when the change affects them too, and do not add Sync Rules mentions elsewhere. Preserve existing structure unless restructuring is part of the approved plan. Flag unresolved facts using the canonical draft-TODO convention and report what must be resolved before publication. diff --git a/docs.json b/docs.json index 50a4eec9..cb659808 100644 --- a/docs.json +++ b/docs.json @@ -235,7 +235,28 @@ "sync/rules/data-queries", "sync/rules/many-to-many-join-tables", "sync/rules/client-parameters", - "sync/grammar/sync-rules/index" + { + "group": "Reference", + "pages": [ + "sync/rules/supported-sql", + "sync/grammar/sync-rules/index", + "sync/rules/types" + ] + }, + { + "group": "Advanced", + "pages": [ + "sync/rules/prioritized-sync", + "sync/rules/client-id", + "sync/rules/case-sensitivity", + "sync/rules/storage-version-4", + "sync/rules/sync-data-by-time", + "sync/rules/schemas-and-connections", + "sync/rules/multiple-client-versions", + "sync/rules/partitioned-tables", + "sync/rules/sharded-databases" + ] + } ] } ] @@ -800,10 +821,6 @@ "source": "/usage/sync-rules/operators-and-functions", "destination": "/sync/supported-sql" }, - { - "source": "/sync/rules/supported-sql", - "destination": "/sync/supported-sql" - }, { "source": "/usage/sync-rules/advanced-topics", "destination": "/sync/advanced/overview" diff --git a/package.json b/package.json index d83b56d3..3bd14d0b 100644 --- a/package.json +++ b/package.json @@ -4,7 +4,7 @@ "packageManager": "pnpm@11.3.0+sha512.2c403d6594527287672b1f7056343a1f7c3634036a67ffabfcc2b3d7595d843768f8787148d1b57cf7956c90606bbd192857c363af19e96d2d0ec9ec5741d215", "scripts": { "dev": "mintlify dev", - "check:links": "mintlify broken-links --check-anchors --check-snippets" + "check:links": "node scripts/check-links.mjs" }, "devDependencies": { "mintlify": "^4.2.520" diff --git a/scripts/check-links.mjs b/scripts/check-links.mjs new file mode 100644 index 00000000..e6f939e8 --- /dev/null +++ b/scripts/check-links.mjs @@ -0,0 +1,187 @@ +#!/usr/bin/env node +/** + * Snippet-aware link check. + * + * Runs `mintlify broken-links --check-anchors --check-snippets`, then re-validates + * every reported `#anchor` against the headings of the target page *including* + * headings that come from snippets the page imports. The Mintlify checker only + * reads page files, so a page whose body lives in a snippet (for example the pages + * shared between the Sync Streams and Sync Rules sections, see snippets/sync-shared/) + * would otherwise fail for every inbound anchor link. + * + * Reported links without a fragment, and anchors that still cannot be found, are + * printed in the Mintlify format and make the script exit with status 1. + */ +import { spawnSync } from 'node:child_process'; +import { existsSync, readFileSync } from 'node:fs'; +import path from 'node:path'; + +const nodeMajor = Number(process.versions.node.split('.')[0]); +if (nodeMajor < 20 || nodeMajor > 24) { + console.error( + `The Mintlify CLI supports Node 20.17 to 24, but this is Node ${process.versions.node}.\n` + + 'Run `nvm use` (the repo pins Node 24 in .nvmrc) or prefix the command with ' + + 'PATH="/opt/homebrew/opt/node@24/bin:$PATH".', + ); + process.exit(1); +} + +const root = process.cwd(); +const localBin = path.join(root, 'node_modules', '.bin', 'mintlify'); +const bin = existsSync(localBin) ? localBin : 'mintlify'; + +const run = spawnSync(bin, ['broken-links', '--check-anchors', '--check-snippets'], { + cwd: root, + encoding: 'utf8', + maxBuffer: 64 * 1024 * 1024, +}); +if (run.error) { + console.error(`Could not run ${bin}: ${run.error.message}`); + process.exit(1); +} + +const output = ((run.stdout ?? '') + (run.stderr ?? '')) + .replace(/\x1B\[[0-9;?]*[A-Za-z]/g, '') + .replace(/\r/g, ''); + +// Parse blocks of "\n ⎿ \n ⎿ ". +const flagged = []; +let currentFile = null; +for (const rawLine of output.split('\n')) { + const line = rawLine.trim(); + if (!line) continue; + if (/checking for broken links/.test(line)) continue; + if (/^found \d+ broken link/.test(line)) continue; + if (/^success/i.test(line)) continue; + const link = line.match(/^⎿\s*(\S.*)$/); + if (link) { + if (currentFile) flagged.push({ file: currentFile, link: link[1].trim() }); + continue; + } + currentFile = line; +} + +if (run.status === 0 && flagged.length === 0) { + console.log('success no broken links found'); + process.exit(0); +} +if (flagged.length === 0) { + // Non-zero exit without a parsable report: show what Mintlify printed. + console.log(output.trim()); + process.exit(run.status ?? 1); +} + +function pageFile(urlPath) { + const p = urlPath.replace(/^\//, '').replace(/\/$/, ''); + for (const candidate of [`${p}.mdx`, `${p}.md`, `${p}/index.mdx`, `${p}/index.md`]) { + if (existsSync(path.join(root, candidate))) return candidate; + } + return null; +} + +function snippetFile(spec) { + const rel = spec.startsWith('/') ? spec.slice(1) : path.posix.join('snippets', spec); + return existsSync(path.join(root, rel)) ? rel : null; +} + +// Page source plus the source of every snippet it imports, recursively. +function collectSource(file, seen = new Set()) { + if (seen.has(file)) return ''; + seen.add(file); + const src = readFileSync(path.join(root, file), 'utf8').replace(/^---\n[\s\S]*?\n---\n/, ''); + let out = src; + for (const m of src.matchAll(/^import\s+\w+\s+from\s+['"]([^'"]+)['"]/gm)) { + const f = snippetFile(m[1]); + if (f) out += `\n${collectSource(f, seen)}`; + } + for (const m of src.matchAll(/]*>/g, ' ') + .replace(/\[([^\]]*)\]\([^)]*\)/g, '$1') + .replace(/[`*]/g, '') + .trim() + .toLowerCase(); + const strict = base.replace(/[^a-z0-9\s_-]/g, '').trim().replace(/\s+/g, '-'); + const keepPunctuation = base.replace(/[^a-z0-9\s_/:-]/g, '').trim().replace(/\s+/g, '-'); + return new Set([strict, keepPunctuation, strict.replace(/-+/g, '-'), keepPunctuation.replace(/-+/g, '-')]); +} + +const anchorCache = new Map(); +function anchorsFor(file) { + if (anchorCache.has(file)) return anchorCache.get(file); + const src = collectSource(file).replace(/```[\s\S]*?```/g, ''); + const anchors = new Set(); + for (const m of src.matchAll(/^#{1,6}\s+(.+?)\s*$/gm)) { + let heading = m[1]; + const explicit = heading.match(/\{#([^}]+)\}\s*$/); + if (explicit) { + anchors.add(explicit[1]); + heading = heading.replace(/\{#[^}]+\}\s*$/, ''); + } + for (const slug of slugCandidates(heading)) anchors.add(slug); + } + for (const m of src.matchAll(/]*\btitle=["']([^"']+)["']/g)) { + for (const slug of slugCandidates(m[1])) anchors.add(slug); + } + for (const m of src.matchAll(/\bid=["']([^"']+)["']/g)) anchors.add(m[1]); + for (const m of src.matchAll(/<(?:ResponseField|ParamField)\b[^>]*\bname=["']([^"']+)["']/g)) { + anchors.add(`param-${m[1].replace(/_/g, '-')}`); + } + anchorCache.set(file, anchors); + return anchors; +} + +const unresolved = []; +for (const item of flagged) { + const hash = item.link.indexOf('#'); + if (hash < 0) { + unresolved.push(item); // a path problem, not an anchor problem + continue; + } + const target = item.link.slice(0, hash).split('?')[0]; + let anchor = item.link.slice(hash + 1); + try { + anchor = decodeURIComponent(anchor); + } catch { + // keep the raw fragment + } + const file = target === '' ? item.file : pageFile(target); + if (!file) { + unresolved.push(item); + continue; + } + const anchors = anchorsFor(file); + const withoutSuffix = anchor.replace(/-\d+$/, ''); + if (anchors.has(anchor) || (withoutSuffix !== anchor && anchors.has(withoutSuffix))) continue; + unresolved.push(item); +} + +const plural = (n, word) => `${n} ${word}${n === 1 ? '' : 's'}`; +if (unresolved.length === 0) { + console.log(`success no broken links found (${plural(flagged.length, 'anchor')} resolved through imported snippets)`); + process.exit(0); +} + +const byFile = new Map(); +for (const item of unresolved) { + if (!byFile.has(item.file)) byFile.set(item.file, []); + byFile.get(item.file).push(item.link); +} +console.log(`found ${plural(unresolved.length, 'broken link')} in ${plural(byFile.size, 'file')}\n`); +for (const [file, links] of byFile) { + console.log(file); + for (const link of links) console.log(` ⎿ ${link}`); + console.log(); +} +process.exit(1); diff --git a/snippets/binary-type.mdx b/snippets/binary-type.mdx index 530ceded..6a999b8a 100644 --- a/snippets/binary-type.mdx +++ b/snippets/binary-type.mdx @@ -1,3 +1,3 @@ - Binary data can be accessed in Sync Streams (or legacy Sync Rules), but cannot be used as a parameter (see [Sync Streams parameters](/sync/streams/parameters) or [Sync Rules client parameters](/sync/rules/client-parameters)). To sync binary columns/fields to clients, those columns need to be converted to hex or base64 representation using the relevant [functions](/sync/supported-sql#functions). + Binary data can be selected in sync queries, but cannot be used as a parameter. To sync binary columns/fields to clients, convert them to a hex or base64 representation using the relevant [functions](/sync/supported-sql#functions). \ No newline at end of file diff --git a/snippets/sync-shared/case-sensitivity.mdx b/snippets/sync-shared/case-sensitivity.mdx new file mode 100644 index 00000000..f558c208 --- /dev/null +++ b/snippets/sync-shared/case-sensitivity.mdx @@ -0,0 +1,40 @@ +{/* Shared body: rendered by sync/advanced/case-sensitivity.mdx (Sync Streams section) and sync/rules/case-sensitivity.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} + +## Case in Sync Queries + +PowerSync converts all table/collection and column/field names to lower-case by default in sync queries (this is how Postgres also behaves). To preserve the case, surround the names with double quotes, for example: + +```sql +SELECT "ID" as id, "Description", "ListID" FROM "TODOs" +``` + +When using `SELECT *`, the original case is preserved for the returned columns/fields. + +## Client-Side Case + +On the client side, the case of table and column names in the [client-side schema](/intro/setup-guide#define-your-client-side-schema) must match the case produced by the sync query exactly. For the above example, use the following in Dart: + +```dart + Table('TODOs', [ + Column.text('Description'), + Column.text('ListID') + ]) +``` + +SQLite itself is case-insensitive. When querying and modifying the data on the client, any case may be used. For example, the above table may be queried using `SELECT description FROM todos WHERE listid = ?`. + +Operations (`PUT`/`PATCH`/`DELETE`) are stored in the upload queue using the case as defined in the schema above for table and column names, not the case used in queries. + +As another example, in this sync query: + +```sql +SELECT ID, todo_description as Description FROM todo_items as TODOs +``` + +Each identifier in the example is unquoted and converted to lower case. That means the client-side schema would be: + +```dart +Table('todos', [ + Column.text('description') +]) +``` diff --git a/snippets/sync-shared/client-id.mdx b/snippets/sync-shared/client-id.mdx new file mode 100644 index 00000000..d1a2884b --- /dev/null +++ b/snippets/sync-shared/client-id.mdx @@ -0,0 +1,65 @@ +{/* Shared body: rendered by sync/advanced/client-id.mdx (Sync Streams section) and sync/rules/client-id.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} + +For tables where the client will create new rows: + +- Postgres, MySQL and SQL Server: use a UUID for `id`. Use the `uuid()` helper to generate a random UUID (v4) on the client. +- MongoDB: use an `ObjectId` for `_id`. Generate an `ObjectId()` in your app code and store it in the client's `id` column as a string; this will map to MongoDB's `_id`. + +To use a different column/field from the server-side database as the record ID on the client, use a column/field alias in your sync query: + +```sql +SELECT client_id as id FROM my_data +``` + + + MongoDB uses `_id` as the name of the ID field in collections. You must use `SELECT _id as id` (and include any other columns you need) in your sync queries when using MongoDB as the backend source database. When inserting new documents from the client, prefer `ObjectId` values for `_id` (stored in the client's `id` column). + + +Custom transformations can also be used for the ID column. This is useful in certain scenarios for example when dealing with join tables, because PowerSync doesn't currently support composite primary keys. For example: + +```sql +-- Concatenate multiple columns into a single id column +SELECT *, item_id || '.' || category_id as id FROM item_categories + +-- the source database schema for the above example is CREATE TABLE item_categories(item_id uuid, category_id uuid, PRIMARY KEY(item_id, category_id)); +``` + + + For multiple columns with the same name (e.g. if there was an `id` column in `*`), the last column wins. Prefer writing the `*` before other columns for this reason. + + If you want to upload data to a table with a custom record ID, ensure that `uploadData()` isn't blindly using a field named `id` when handling CRUD operations. See the [Sequential ID mapping tutorial](/client-sdks/advanced/sequential-id-mapping#update-client-to-use-uuids) for an example where the record ID is aliased to `uuid` on the backend. + + +PowerSync does not perform any validation that IDs are unique. Duplicate IDs on a client could occur in any of these scenarios: + +1. A non-unique column is used for the ID. +2. Multiple table partitions are used (Postgres), with the same ID present in different partitions. +3. Multiple queries returning the same record. This is typically not an issue if the queries return the same values (same transformations used in each query). + +We recommend using a unique index on the fields in the source database to ensure uniqueness — this will prevent (1) at least. + +If the client does sync multiple records with the same ID, only one will be present in the final database. This would typically be the one modified last, but this is subject to change — do not depend on any specific record being picked. + +### Postgres: Strategies for Auto-Incrementing IDs + +With auto-incrementing / sequential IDs (e.g. `sequence` type in Postgres), the issue is that the ID can only be generated on the server, and not on the client while offline. If this _must_ be used, there are some options, depending on the use case. + +#### Option 1: Generate ID when server receives record + +If the client does not use the ID as a reference (foreign key) elsewhere, insert any unique value on the client in the `id` field, then generate a new ID when the server receives it. + +#### Option 2: Pre-create records on the server + +For some use cases, it could work to have the server pre-create a set of e.g. 100 draft records for each user. While offline, the client can populate these records without needing to generate new IDs. This is similar to providing an employee with a paper book of blank invoices — each with an invoice number pre-printed. + +This does mean that a user has a limit on how many records can be populated while offline. + +Care must be taken if a user can populate the same records from different devices while offline — ideally each device must have a unique set of pre-created records. + +#### Option 3: Use an ID mapping + +Use UUIDs on the client, then map them to sequential IDs when performing an update on the server. This allows using a sequential primary key for each record, with a UUID as a secondary ID. + +This mapping must be performed wherever the UUIDs are referenced, including for every foreign key column. + +For more information, have a look at [Sequential ID Mapping](/client-sdks/advanced/sequential-id-mapping). diff --git a/snippets/sync-shared/schemas-and-connections.mdx b/snippets/sync-shared/schemas-and-connections.mdx new file mode 100644 index 00000000..592125fa --- /dev/null +++ b/snippets/sync-shared/schemas-and-connections.mdx @@ -0,0 +1,60 @@ +{/* Shared body: rendered by sync/advanced/schemas-and-connections.mdx (Sync Streams section) and sync/rules/schemas-and-connections.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} + +## Schemas (Postgres) + +When no schema is specified, the Postgres `public` schema is used for every query. A different schema can be specified as a prefix: + +```sql +-- Note: the schema must be in double quotes +SELECT * FROM "other"."assets" +``` + +## Wildcard Schemas (Postgres) + + + Wildcard schemas require [Sync Streams](/sync/streams/overview) and PowerSync Service v1.24.0 or later. They are currently only supported for Postgres connections. + + +Use `%` as a wildcard in the schema name to match tables with the same name across multiple schemas. `"%"` matches every schema, and a prefix such as `"tenant_%"` matches every schema whose name starts with `tenant_`. The wildcard can only be the last character of the schema name. Postgres system schemas (`pg_*` and `information_schema`) are never matched. + +Combine a wildcard schema with the [`schema()` function](/sync/supported-sql#functions), which returns the schema each row was replicated from, to filter rows by schema. This supports schema-per-tenant databases (a single database with one identical schema per tenant): one stream covers every tenant schema, and each client syncs only its own tenant's data, resolved from a JWT claim. + +```yaml +config: + edition: 3 + +streams: + work_orders: + query: SELECT * FROM "%".work_orders WHERE work_orders.schema() = auth.parameter('tenant_schema') +``` + +In this example, rows are grouped into a bucket per schema, and each client syncs only the bucket matching the `tenant_schema` claim in its JWT. Rows from all matched schemas sync into a single client-side table, named after the table in the query (`work_orders` here). + + + Each matched table must be part of the [PowerSync publication](/configuration/source-db/setup#3-create-powersync-publication). Tables that are not in the publication are skipped. + + +## High Availability / Replicated Databases (Postgres) + +When the source Postgres database is replicated, for example with Amazon RDS Multi-AZ deployments, specify a single connection with multiple host endpoints. Each host endpoint will be tried in sequence, with the first available primary connection being used. + +For this, each endpoint must point to the same physical database, with the same replication slots. This is the case when block-level replication is used between the databases, but not when streaming physical or logical replication is used. In those cases, replication slots are unique on each host, and all data would be re-synced in a fail-over event. + +## Multiple Separate Database Connections (Planned) + + + This feature will be available in a future release. See this [item on our roadmap](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections). + + +In the future, it will be possible to configure PowerSync with multiple separate source database connections, where each connection is concurrently replicated. + +You should not add multiple connections to multiple replicas of the same database — this would cause data duplication. Only use this when the data on each connection does not overlap. + +It will be possible for each connection to be configured with a "tag", to distinguish these connections in sync queries. The same tag may be used for multiple connections (if the schema is the same in each). + +By default, queries will reference the "default" tag. To use a different connection or connections, assign a different tag, and specify it in the query as a schema prefix. In this case, the schema itself must also be specified. + +```sql +-- Note the usage of quotes here +SELECT * FROM "secondconnection.public"."assets" +``` diff --git a/snippets/sync-shared/sharded-databases.mdx b/snippets/sync-shared/sharded-databases.mdx new file mode 100644 index 00000000..77a0d553 --- /dev/null +++ b/snippets/sync-shared/sharded-databases.mdx @@ -0,0 +1,44 @@ +{/* Shared body: rendered by sync/advanced/sharded-databases.mdx (Sync Streams section) and sync/rules/sharded-databases.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} + +In the case of Postgres, PowerSync cannot replicate Postgres [foreign tables](https://www.postgresql.org/docs/current/ddl-foreign-data.html). + +However, PowerSync does have options available to support sharded databases in general. + + + When using MongoDB, MySQL, or SQL Server as the backend source database, PowerSync does not currently support connecting to sharded clusters. + + +The primary options are: + +1. Use a separate PowerSync Service instance per database. +2. Add a connection for each database in the same PowerSync Service instance ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release). + +Where feasible, using separate PowerSync Service instances would give better performance and give more control over how changes are rolled out, especially around Sync Config reprocessing. + +Some specific scenarios: + +#### 1\. Different Tables on Different Databases + +This is common when separate "services" use separate databases, but multiple tables across those databases need to be synced to the same users. + +Use a single PowerSync Service instance, with a separate connection for each source database ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release). Use a unique [connection tag](/sync/advanced/schemas-and-connections) for each source database, allowing them to be distinguished in your sync queries. + +#### 2a. All Data for a Single Customer Is Contained in a Single Shard + +This is common when sharding per customer account / organization. + +In this case, use a separate PowerSync Service instance for each database. + +#### 2b. Most Customer Data Is in a Single Shard, but Some Data Is in a Shared Database + +If the amount of shared data is small, still use a separate PowerSync Service instance for each database, but also add the shared database connection to each PowerSync Service instance using a separate connection tag ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release). + +#### 2c. Data for a Single Customer Is Split Across Multiple Shards + +This is more complicated than the other cases listed above. Please [reach out to us](/resources/contact-us) if this is your architecture. + +#### 3\. Only Some Tables Are Sharded + +In some cases, most tables would be on a shared server, with only a few large tables being sharded. + +For this case, use a single PowerSync Service instance. Add each shard as a new connection on this instance ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release) — all with the same connection tag, so that the same Sync Config applies to each. diff --git a/snippets/sync-shared/types.mdx b/snippets/sync-shared/types.mdx new file mode 100644 index 00000000..51982e58 --- /dev/null +++ b/snippets/sync-shared/types.mdx @@ -0,0 +1,164 @@ +{/* Shared body: rendered by sync/types.mdx (Sync Streams section) and sync/rules/types.mdx (Sync Rules (Legacy) section). Keep the content valid for both engines. */} + +import BinaryType from '/snippets/binary-type.mdx'; + +The supported client-side SQLite types are: + +1. `null` +2. `integer`: a 64-bit signed integer +3. `real`: a 64-bit floating point number +4. `text`: A UTF-8 text string +5. `blob`: Binary data + + +## Postgres Type Mapping + +Postgres types are mapped to SQLite types as follows: + +| Postgres Data Type | PowerSync / SQLite Column Type | Notes | +|--------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `text`, `varchar` | `text` | | +| `int2`, `int4`, `int8` | `integer` | | +| `numeric` / `decimal` | `text` | These types have arbitrary precision in Postgres, so can only be represented accurately as text in SQLite | +| `bool` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | +| `float4`, `float8` | `real` | | +| `enum` | `text` | | +| `uuid` | `text` | | +| `timestamptz` | `text` | Format: `YYYY-MM-DD hh:mm:ss.sssZ`. This is compatible with ISO8601 and SQLite's functions. Precision matches the precision used in Postgres. `-infinity` becomes `0000-01-01 00:00:00Z` and `infinity` becomes `9999-12-31 23:59:59Z`. | +| `timestamp` | `text` | Format: `YYYY-MM-DD hh:mm:ss.sss`. In most cases, timestamptz should be used instead. `-infinity` becomes `0000-01-01 00:00:00` and `infinity` becomes `9999-12-31 23:59:59`. | +| `date`, `time` | `text` | | +| `json`, `jsonb` | `text` | `json` and `jsonb` values are treated as `text` values in their serialized representation. [JSON functions and operators](/sync/supported-sql#operators) operate directly on these `text` values. | +| `interval` | `text` | | +| `macaddr` | `text` | | +| `inet` | `text` | | +| `bytea` | `blob` | Cannot sync directly to client — convert to hex or base64 first. See [Operators & Functions](/sync/supported-sql). | +| `geometry` (PostGIS) | `text` | Hex string of the binary data. Use the [ST functions](/sync/supported-sql#functions) to convert to other formats | +| Arrays | `text` | JSON array. | +| `DOMAIN` types | `text` / depends | Depending on [compatibility options](/sync/advanced/compatibility#custom_postgres_types), inner type or raw wire representation (legacy). | +| Custom types | `text` | Depending on [compatibility options](/sync/advanced/compatibility#custom_postgres_types), JSON object or raw wire representation (legacy). | +| (Multi-)ranges | `text` | Depending on [compatibility options](/sync/advanced/compatibility#custom_postgres_types), JSON object (array for multi-ranges) or raw wire representation (legacy). | + + + + +## Convex Type Mapping + + + The Convex replicator is currently released as an [experimental feature](/resources/feature-status). APIs and + behavior may change, and we can't yet guarantee continued support or long-term stability. + + +Convex values are mapped to SQLite types as follows: + +| Convex Type | TS/JS Type | PowerSync / SQLite Column Type | Notes | +| ----------- | ---------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `Id` | `string` | `text` | Convex document IDs are exposed as `_id` and can be synced as `text`. For synced client tables, use client-side ID mapping with a stable UUID column as `id` instead of relying on Convex-generated `_id` values. | +| `Null` | `null` | `null` | | +| `Int64` | `base-10 string` | `text` | Cast to `INTEGER` in Sync Streams when you want to sync the value as a SQLite integer. | +| `Float64` | `number` | `real` | | +| `Boolean` | `boolean` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | +| `String` | `string` | `text` | | +| `Bytes` | `base64 string` | `text` | Decode from base64 in your app if you need binary data on the client. | +| `Array` | `Array` | `text` | Converted to a JSON string. | +| `Object` | `Object` | `text` | Converted to a JSON string. | +| `Record` | `Record` | `text` | Converted to a JSON string. | + +- Convex documents are converted to a flat list of columns, one column per top-level field. +- Nested objects and arrays are converted to JSON, and [JSON functions and operators](/sync/supported-sql#operators) can be used to query them in Sync Streams or on the client-side SQLite database. +- Cast Convex `Int64` fields to `INTEGER` in Sync Streams when you want SQLite integer values on the client, for example `CAST(an_int64_column AS INTEGER) AS an_int64_column`. + + +## MongoDB Type Mapping + +MongoDB types are mapped to SQLite types as follows: + +| BSON Type | PowerSync / SQLite Column Type | Notes | +|--------------------|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------| +| `String` | `text` | | +| `Int`, `Long` | `integer` | | +| `Double` | `real` | | +| `Decimal128` | `text` | | +| `Object` | `text` | Converted to a JSON string | +| `Array` | `text` | Converted to a JSON string | +| `ObjectId` | `text` | Lower-case hex string | +| `UUID` | `text` | Lower-case hex string | +| `Boolean` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | +| `Date` | `text` | Format: `YYYY-MM-DD hh:mm:ss.sssZ` | +| `Null` | `null` | | +| `Binary` | `blob` | Cannot sync directly to client — convert to hex or base64 first. See [Operators & Functions](/sync/supported-sql). | +| Regular Expression | `text` | JSON text in the format `{"pattern":"...","options":"..."}` | +| `Timestamp` | `integer` | Converted to a 64-bit integer | +| `Undefined` | `null` | | +| `DBPointer` | `text` | JSON text in the format `{"collection":"...","oid":"...","db":"...","fields":...}` | +| `JavaScript` | `text` | JSON text in the format `{"code": "...", "scope": ...}` | +| `Symbol` | `text` | | +| `MinKey`, `MaxKey` | `null` | | + +* Data is converted to a flat list of columns, one column per top-level field in the MongoDB document. +* Special BSON types are converted to plain SQLite alternatives. For example, `ObjectId`, `Date`, `UUID` are all converted to a plain `TEXT` column. +* Nested objects and arrays are converted to JSON, and [JSON functions and operators](/sync/supported-sql#operators) can be used to query them (in sync queries or in client-side SQLite statements). +* Binary data nested in objects or arrays is not supported. + + + + +## MySQL Type Mapping + +MySQL support is currently in a [Beta release](/resources/feature-status). + +MySQL types are mapped to SQLite types as follows: + +| MySQL Data Type | PowerSync / SQLite Column Type | Notes | +|----------------------------------------------------|--------------------------------|-----------------------------------------------------------------------------------| +| `tinyint`, `smallint`, `mediumint`, `bigint`, `integer`, `int` | `integer` | | +| `numeric`, `decimal` | `text` | | +| `bool`, `boolean` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | +| `float`, `double`, `real` | `real` | | +| `enum` | `text` | | +| `set` | `text` | Converted to JSON array | +| `char`, `varchar` | `text` | | +| `tinytext`, `text`, `mediumtext`, `longtext` | `text` | | +| `timestamp` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | +| `date` | `text` | Format: `YYYY-MM-DD` | +| `time`, `datetime` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | +| `year` | `text` | | +| `json` | `text` | There is no dedicated JSON type in SQLite — JSON functions operate directly on text values. | +| `bit` | `blob` | * See note below regarding syncing binary types | +| `binary`, `varbinary` | `blob` | | +| `image` | `blob` | | +| `geometry`, `geometrycollection` | `blob` | | +| `point`, `multipoint` | `blob` | | +| `linestring`, `multilinestring` | `blob` | | +| `polygon`, `multipolygon` | `blob` | | + + + + +## SQL Server Type Mapping + +SQL Server support is currently in a [Beta release](/resources/feature-status). + +SQL Server types are mapped to SQLite types as follows: + +| SQL Server Data Type | PowerSync / SQLite Column Type | Notes | +|----------------------------------------------------|--------------------------------|--------------------------------------------------------| +| `tinyint`, `smallint`, `int`, `bigint` | `integer` | | +| `numeric`, `decimal` | `text` | Numeric string | +| `float`, `real` | `real` | | +| `bit` | `integer` | | +| `money`, `smallmoney` | `text` | Numeric string | +| `xml` | `text` | | +| `char`, `nchar`, `ntext` | `text` | | +| `varchar`, `nvarchar`, `text` | `text` | | +| `uniqueidentifier` | `text` | | +| `timestamp` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | +| `date` | `text` | Format: `YYYY-MM-DD` | +| `time` | `text` | Format: `HH:mm:ss.sss` | +| `datetime`, `datetime2`, `smalldatetime`, `datetimeoffset` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | +| `json` | `text` | Only exists for Azure SQL Database and SQL Server 2025 | +| `geometry`, `geography` | `text` | `text` of JSON object describing the spatial data type | +| `binary`, `varbinary`, `image` | `blob` | * See note below regarding binary types | +| `rowversion`, `timestamp` | `blob` | * See note below regarding binary types | +| User Defined Types: `hiearchyid` | `blob` | * See note below regarding binary types | + + diff --git a/sync/advanced/case-sensitivity.mdx b/sync/advanced/case-sensitivity.mdx index 3591e8f7..7cf4b8e3 100644 --- a/sync/advanced/case-sensitivity.mdx +++ b/sync/advanced/case-sensitivity.mdx @@ -1,43 +1,10 @@ --- title: "Case Sensitivity" -description: "Handle case-sensitive table and column names in PowerSync Sync Streams/Rules, with best practices for lowercase identifiers and quoting strategies." +description: "Handle case-sensitive table and column names in PowerSync sync queries, with best practices for lowercase identifiers and quoting strategies." --- -### Case in Sync Rules +{/* Wrapper page: the content is snippets/sync-shared/case-sensitivity.mdx, which also renders at sync/rules/case-sensitivity.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} -PowerSync converts all table/collection and column/field names to lower-case by default in Sync Rule queries (this is how Postgres also behaves). To preserve the case, surround the names with double quotes, for example: +import CaseSensitivity from '/snippets/sync-shared/case-sensitivity.mdx'; -```sql -SELECT "ID" as id, "Description", "ListID" FROM "TODOs" WHERE "TODOs"."ListID" = bucket.list_id -``` - -When using `SELECT *`, the original case is preserved for the returned columns/fields. - -### Client-Side Case - -On the client side, the case of table and column names in the [client-side schema](/intro/setup-guide#define-your-client-side-schema) must match the case produced by Sync Rules exactly. For the above example, use the following in Dart: - -```dart - Table('TODOs', [ - Column.text('Description'), - Column.text('ListID') - ]) -``` - -SQLite itself is case-insensitive. When querying and modifying the data on the client, any case may be used. For example, the above table may be queried using `SELECT description FROM todos WHERE listid = ?`. - -Operations (`PUT`/`PATCH`/`DELETE`) are stored in the upload queue using the case as defined in the schema above for table and column names, not the case used in queries. - -As another example, in this Sync Rule query: - -```sql -SELECT ID, todo_description as Description FROM todo_items as TODOs -``` - -Each identifier in the example is unquoted and converted to lower case. That means the client-side schema would be: - -```dart -Table('todos', [ - Column.text('description') -]) -``` + diff --git a/sync/advanced/client-id.mdx b/sync/advanced/client-id.mdx index c6086911..5a1683bc 100644 --- a/sync/advanced/client-id.mdx +++ b/sync/advanced/client-id.mdx @@ -3,66 +3,8 @@ title: "Client ID" description: "Understand PowerSync's requirement for a single text-type primary key column called id." --- -For tables where the client will create new rows: +{/* Wrapper page: the content is snippets/sync-shared/client-id.mdx, which also renders at sync/rules/client-id.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} -- Postgres, MySQL and SQL Server: use a UUID for `id`. Use the `uuid()` helper to generate a random UUID (v4) on the client. -- MongoDB: use an `ObjectId` for `_id`. Generate an `ObjectId()` in your app code and store it in the client's `id` column as a string; this will map to MongoDB's `_id`. +import ClientId from '/snippets/sync-shared/client-id.mdx'; -To use a different column/field from the server-side database as the record ID on the client, use a column/field alias in your [Sync Streams](/sync/streams/overview) query (or [Sync Rules](/sync/rules/overview) data query): - -```sql -SELECT client_id as id FROM my_data -``` - - - MongoDB uses `_id` as the name of the ID field in collections. You must use `SELECT _id as id` (and include any other columns you need) in [Sync Streams](/sync/streams/overview) queries and [Sync Rules](/sync/rules/overview) data queries when using MongoDB as the backend source database. When inserting new documents from the client, prefer `ObjectId` values for `_id` (stored in the client's `id` column). - - -Custom transformations can also be used for the ID column. This is useful in certain scenarios for example when dealing with join tables, because PowerSync doesn't currently support composite primary keys. For example: - -```sql --- Concatenate multiple columns into a single id column -SELECT *, item_id || '.' || category_id as id FROM item_categories - --- the source database schema for the above example is CREATE TABLE item_categories(item_id uuid, category_id uuid, PRIMARY KEY(item_id, category_id)); -``` - - - For multiple columns with the same name (e.g. if there was an `id` column in `*`), the last column wins. Prefer writing the `*` before other columns for this reason. - - If you want to upload data to a table with a custom record ID, ensure that `uploadData()` isn't blindly using a field named `id` when handling CRUD operations. See the [Sequential ID mapping tutorial](/client-sdks/advanced/sequential-id-mapping#update-client-to-use-uuids) for an example where the record ID is aliased to `uuid` on the backend. - - -PowerSync does not perform any validation that IDs are unique. Duplicate IDs on a client could occur in any of these scenarios: - -1. A non-unique column is used for the ID. -2. Multiple table partitions are used (Postgres), with the same ID present in different partitions. -3. Multiple data queries returning the same record. This is typically not an issue if the queries return the same values (same transformations used in each query). - -We recommend using a unique index on the fields in the source database to ensure uniqueness — this will prevent (1) at least. - -If the client does sync multiple records with the same ID, only one will be present in the final database. This would typically be the one modified last, but this is subject to change — do not depend on any specific record being picked. - -### Postgres: Strategies for Auto-Incrementing IDs - -With auto-incrementing / sequential IDs (e.g. `sequence` type in Postgres), the issue is that the ID can only be generated on the server, and not on the client while offline. If this _must_ be used, there are some options, depending on the use case. - -#### Option 1: Generate ID when server receives record - -If the client does not use the ID as a reference (foreign key) elsewhere, insert any unique value on the client in the `id` field, then generate a new ID when the server receives it. - -#### Option 2: Pre-create records on the server - -For some use cases, it could work to have the server pre-create a set of e.g. 100 draft records for each user. While offline, the client can populate these records without needing to generate new IDs. This is similar to providing an employee with a paper book of blank invoices — each with an invoice number pre-printed. - -This does mean that a user has a limit on how many records can be populated while offline. - -Care must be taken if a user can populate the same records from different devices while offline — ideally each device must have a unique set of pre-created records. - -#### Option 3: Use an ID mapping - -Use UUIDs on the client, then map them to sequential IDs when performing an update on the server. This allows using a sequential primary key for each record, with a UUID as a secondary ID. - -This mapping must be performed wherever the UUIDs are referenced, including for every foreign key column. - -For more information, have a look at [Sequential ID Mapping](/client-sdks/advanced/sequential-id-mapping). \ No newline at end of file + diff --git a/sync/advanced/compatibility.mdx b/sync/advanced/compatibility.mdx index 444e6e9a..f5d3fc8a 100644 --- a/sync/advanced/compatibility.mdx +++ b/sync/advanced/compatibility.mdx @@ -154,8 +154,8 @@ If no option is given, the default precision depends on the source database: ### `versioned_bucket_ids` -Sync Rules define buckets, which rows to sync are then assigned to. When you run a full defragmentation or -redeploy Sync Rules, the same bucket identifiers are re-used when processing data again. +Streams define buckets, and rows to sync are assigned to those buckets. When you run a full defragmentation or +redeploy your Sync Config, the same bucket identifiers are re-used when processing data again. Because the second iteration uses different checksums for the same bucket ids, clients may sync data twice before realizing that something is off and starting from scratch. @@ -167,7 +167,7 @@ For how bucket identifiers are represented in bucket storage at the persistence ### `fixed_json_extract` -This fixes the `json_extract` functions as well as the `->` and `->>` operators in Sync Rules to behave similar +This fixes the `json_extract` functions as well as the `->` and `->>` operators in stream queries to behave similar to recent SQLite versions: We only split on `.` if the path starts with `$.`. For instance, `'json_extract({"foo.bar": "baz"}', 'foo.bar')` would evaluate to: diff --git a/sync/advanced/multiple-client-versions.mdx b/sync/advanced/multiple-client-versions.mdx index 1c1cb9f3..4562e6b7 100644 --- a/sync/advanced/multiple-client-versions.mdx +++ b/sync/advanced/multiple-client-versions.mdx @@ -3,6 +3,8 @@ title: "Multiple Client Versions" description: "Handle multiple client app versions that require different output schemas from Sync Streams." --- +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/multiple-client-versions.mdx. When you change shared behavior or fix an error here, check whether that page needs the same change. Do not mention Sync Rules on this page. */} + When schema changes are additive, old clients ignore the new tables and columns, and no special handling is required. More drastic changes, such as renaming tables or changing a table's structure, can break older app versions that are still in use. In these cases, define separate versions of the affected [Sync Streams](/sync/streams/overview) so that each client version receives the tables and columns it expects. ## Versioning by Stream Name @@ -33,45 +35,25 @@ Once the older app versions are no longer in use, remove the old stream from you ## Versioning with Connection Parameters -Alternatively, clients can pass their version to the PowerSync Service as a [connection parameter](/sync/streams/parameters#connection-parameters), and stream queries filter on it so each client only receives data for its version. This approach is useful when your streams are auto-subscribed: auto-subscribed streams sync to every client on connect, so clients cannot select a stream version by name. In legacy [Sync Rules](/sync/rules/overview), connection parameters are called [client parameters](/sync/rules/client-parameters). +Alternatively, clients can pass their version to the PowerSync Service as a [connection parameter](/sync/streams/parameters#connection-parameters), and stream queries filter on it so each client only receives data for its version. This approach is useful when your streams are auto-subscribed: auto-subscribed streams sync to every client on connect, so clients cannot select a stream version by name. The example below implements the same `assets` use case, with both stream versions auto-subscribed and filtered by a `schema_version` connection parameter: - - - ```yaml - # Client passes connection params: {"schema_version": } - streams: - user_assets: - auto_subscribe: true - query: SELECT * FROM assets - WHERE user_id = auth.user_id() - AND connection.parameter('schema_version') = '1' - - user_assets_v2: - auto_subscribe: true - query: SELECT * FROM assets AS assets_v2 - WHERE user_id = auth.user_id() - AND connection.parameter('schema_version') = '2' - ``` - - - ```yaml - # Client passes in: "params": {"schema_version": } - user_assets: - parameters: SELECT request.user_id() AS user_id - WHERE request.parameters() ->> 'schema_version' = '1' - data: - - SELECT * FROM assets WHERE user_id = bucket.user_id +```yaml +# Client passes connection params: {"schema_version": } +streams: + user_assets: + auto_subscribe: true + query: SELECT * FROM assets + WHERE user_id = auth.user_id() + AND connection.parameter('schema_version') = '1' - user_assets_v2: - parameters: SELECT request.user_id() AS user_id - WHERE request.parameters() ->> 'schema_version' = '2' - data: - - SELECT * FROM assets AS assets_v2 WHERE user_id = bucket.user_id - ``` - - + user_assets_v2: + auto_subscribe: true + query: SELECT * FROM assets AS assets_v2 + WHERE user_id = auth.user_id() + AND connection.parameter('schema_version') = '2' +``` Handle queries based on parameters set by the client with care. The client can send any value for these parameters, so it's not a good place to do authorization. If the parameter must be authenticated, use parameters from the JWT instead. diff --git a/sync/advanced/partitioned-tables.mdx b/sync/advanced/partitioned-tables.mdx index 251d7b2b..0e78e1e6 100644 --- a/sync/advanced/partitioned-tables.mdx +++ b/sync/advanced/partitioned-tables.mdx @@ -3,55 +3,34 @@ title: "Partitioned Tables (Postgres)" description: "Sync data from Postgres partitioned tables using wildcard table name matching." --- -For partitioned tables in Postgres, each individual partition is replicated and processed using [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)). +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/partitioned-tables.mdx. When you change shared behavior or fix an error here, check whether that page needs the same change. Do not mention Sync Rules on this page. */} + +For partitioned tables in Postgres, each individual partition is replicated and processed using [Sync Streams](/sync/streams/overview). To use the same queries and same output table name for each partition, use `%` for wildcard suffix matching of the table name: - - - ```yaml - streams: - user_todos: - queries: - # Wildcard matches all user partition tables (e.g. users_2024, users_2025) - - SELECT * FROM "users_%" WHERE id = auth.user_id() - # Wildcard matches all todo partition tables (e.g. todos_2024, todos_2025) - - SELECT * FROM "todos_%" AS todos WHERE user_id = auth.user_id() - ``` - - - ```yaml - by_user: - # Use wildcard in a parameter query - parameters: SELECT id AS user_id FROM "users_%" - data: - # Use wildcard in a data query - - SELECT * FROM "todos_%" AS todos WHERE user_id = bucket.user_id - ``` - - +```yaml +streams: + user_todos: + queries: + # Wildcard matches all user partition tables (e.g. users_2024, users_2025) + - SELECT * FROM "users_%" WHERE id = auth.user_id() + # Wildcard matches all todo partition tables (e.g. todos_2024, todos_2025) + - SELECT * FROM "todos_%" AS todos WHERE user_id = auth.user_id() +``` The wildcard character can only be used as the last character in the table name. To match tables across multiple schemas instead, see [Wildcard Schemas](/sync/advanced/schemas-and-connections#wildcard-schemas-postgres). -When using wildcard table names, you can filter on the original table suffix. In Sync Streams, use the `table_suffix()` function, prefixed with the table name or alias from the `FROM` clause (requires PowerSync Service v1.24.0 or later). In legacy Sync Rules, the suffix is available as the special `_table_suffix` column instead: - - - - ```yaml - config: - edition: 3 - - streams: - active_todos: - query: SELECT * FROM "todos_%" AS todos WHERE todos.table_suffix() != 'archived' - ``` - - - ```sql - SELECT * FROM "todos_%" AS todos WHERE _table_suffix != 'archived' - ``` - - +When using wildcard table names, you can filter on the original table suffix with the `table_suffix()` function, prefixed with the table name or alias from the `FROM` clause. This requires PowerSync Service v1.24.0 or later: + +```yaml +config: + edition: 3 + +streams: + active_todos: + query: SELECT * FROM "todos_%" AS todos WHERE todos.table_suffix() != 'archived' +``` When no table alias is provided, the original table name is preserved. diff --git a/sync/advanced/schemas-and-connections.mdx b/sync/advanced/schemas-and-connections.mdx index 7c6633b3..e3653fd9 100644 --- a/sync/advanced/schemas-and-connections.mdx +++ b/sync/advanced/schemas-and-connections.mdx @@ -1,63 +1,10 @@ --- title: "Schemas and Connections" -description: "Configure Postgres schema usage in Sync Streams/Rules queries, including wildcard schemas for schema-per-tenant setups, and connect to high-availability replicas." +description: "Configure Postgres schema usage in sync queries, including wildcard schemas for schema-per-tenant setups, and connect to high-availability replicas." --- -## Schemas (Postgres) +{/* Wrapper page: the content is snippets/sync-shared/schemas-and-connections.mdx, which also renders at sync/rules/schemas-and-connections.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} -When no schema is specified, the Postgres `public` schema is used for every query. A different schema can be specified as a prefix: +import SchemasAndConnections from '/snippets/sync-shared/schemas-and-connections.mdx'; -```sql --- Note: the schema must be in double quotes -SELECT * FROM "other"."assets" -``` - -## Wildcard Schemas (Postgres) - - - Wildcard schemas require [Sync Streams](/sync/streams/overview) and PowerSync Service v1.24.0 or later. They are currently only supported for Postgres connections. - - -Use `%` as a wildcard in the schema name to match tables with the same name across multiple schemas. `"%"` matches every schema, and a prefix such as `"tenant_%"` matches every schema whose name starts with `tenant_`. The wildcard can only be the last character of the schema name. Postgres system schemas (`pg_*` and `information_schema`) are never matched. - -Combine a wildcard schema with the [`schema()` function](/sync/supported-sql#functions), which returns the schema each row was replicated from, to filter rows by schema. This supports schema-per-tenant databases (a single database with one identical schema per tenant): one stream covers every tenant schema, and each client syncs only its own tenant's data, resolved from a JWT claim. - -```yaml -config: - edition: 3 - -streams: - work_orders: - query: SELECT * FROM "%".work_orders WHERE work_orders.schema() = auth.parameter('tenant_schema') -``` - -In this example, rows are grouped into a bucket per schema, and each client syncs only the bucket matching the `tenant_schema` claim in its JWT. Rows from all matched schemas sync into a single client-side table, named after the table in the query (`work_orders` here). - - - Each matched table must be part of the [PowerSync publication](/configuration/source-db/setup#3-create-powersync-publication). Tables that are not in the publication are skipped. - - -## High Availability / Replicated Databases (Postgres) - -When the source Postgres database is replicated, for example with Amazon RDS Multi-AZ deployments, specify a single connection with multiple host endpoints. Each host endpoint will be tried in sequence, with the first available primary connection being used. - -For this, each endpoint must point to the same physical database, with the same replication slots. This is the case when block-level replication is used between the databases, but not when streaming physical or logical replication is used. In those cases, replication slots are unique on each host, and all data would be re-synced in a fail-over event. - -## Multiple Separate Database Connections (Planned) - - - This feature will be available in a future release. See this [item on our roadmap](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections). - - -In the future, it will be possible to configure PowerSync with multiple separate source database connections, where each connection is concurrently replicated. - -You should not add multiple connections to multiple replicas of the same database — this would cause data duplication. Only use this when the data on each connection does not overlap. - -It will be possible for each connection to be configured with a "tag", to distinguish these connections in Sync Rules. The same tag may be used for multiple connections (if the schema is the same in each). - -By default, queries will reference the "default" tag. To use a different connection or connections, assign a different tag, and specify it in the query as a schema prefix. In this case, the schema itself must also be specified. - -```sql --- Note the usage of quotes here -SELECT * FROM "secondconnection.public"."assets" -``` + diff --git a/sync/advanced/sharded-databases.mdx b/sync/advanced/sharded-databases.mdx index afde1abe..6c5e37c4 100644 --- a/sync/advanced/sharded-databases.mdx +++ b/sync/advanced/sharded-databases.mdx @@ -3,45 +3,8 @@ title: "Sharded Databases" description: "Sync data from sharded Postgres databases with per-shard PowerSync connection configuration." --- -In the case of Postgres, PowerSync cannot replicate Postgres [foreign tables](https://www.postgresql.org/docs/current/ddl-foreign-data.html). +{/* Wrapper page: the content is snippets/sync-shared/sharded-databases.mdx, which also renders at sync/rules/sharded-databases.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} -However, PowerSync does have options available to support sharded databases in general. +import ShardedDatabases from '/snippets/sync-shared/sharded-databases.mdx'; - - When using MongoDB, MySQL, or SQL Server as the backend source database, PowerSync does not currently support connecting to sharded clusters. - - -The primary options are: - -1. Use a separate PowerSync Service instance per database. -2. Add a connection for each database in the same PowerSync Service instance ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release). - -Where feasible, using separate PowerSync Service instances would give better performance and give more control over how changes are rolled out, especially around Sync Rule reprocessing. - -Some specific scenarios: - -#### 1\. Different Tables on Different Databases - -This is common when separate "services" use separate databases, but multiple tables across those databases need to be synced to the same users. - -Use a single PowerSync Service instance, with a separate connection for each source database ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release). Use a unique [connection tag](/sync/advanced/schemas-and-connections) for each source database, allowing them to be distinguished in your [Sync Streams](/sync/streams/overview) or [Sync Rules](/sync/rules/overview). - -#### 2a. All Data for a Single Customer Is Contained in a Single Shard - -This is common when sharding per customer account / organization. - -In this case, use a separate PowerSync Service instance for each database. - -#### 2b. Most Customer Data Is in a Single Shard, but Some Data Is in a Shared Database - -If the amount of shared data is small, still use a separate PowerSync Service instance for each database, but also add the shared database connection to each PowerSync Service instance using a separate connection tag ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release). - -#### 2c. Data for a Single Customer Is Split Across Multiple Shards - -This is more complicated than the other cases listed above. Please [reach out to us](/resources/contact-us) if this is your architecture. - -#### 3\. Only Some Tables Are Sharded - -In some cases, most tables would be on a shared server, with only a few large tables being sharded. - -For this case, use a single PowerSync Service instance. Add each shard as a new connection on this instance ([planned](https://roadmap.powersync.com/c/84-support-for-sharding-multiple-database-connections); this capability will be available in a future release) — all with the same connection tag, so that the same [Sync Streams](/sync/streams/overview) or [Sync Rules](/sync/rules/overview) applies to each. + diff --git a/sync/advanced/storage-version-4.mdx b/sync/advanced/storage-version-4.mdx index 2b6b4b9e..ac263241 100644 --- a/sync/advanced/storage-version-4.mdx +++ b/sync/advanced/storage-version-4.mdx @@ -3,6 +3,8 @@ title: "Storage Version 4" description: "Opt in to storage version 4 for faster sync, incremental reprocessing of Sync Streams changes, and S3 object storage." --- +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/storage-version-4.mdx. It is a short summary that links here for details, so keep the anchors opt-in, incremental-reprocessing, s3-object-storage, and self-hosted-s3-setup stable. Mention Sync Rules here only to state that incremental reprocessing requires Sync Streams. */} + Storage version 4 is a new version of the format the PowerSync Service uses to store the data it syncs to clients. It is in [Beta](/resources/feature-status) as of PowerSync Service v1.26.0. Compared to version 2, it provides: @@ -17,11 +19,11 @@ Storage version 4 is compatible with all PowerSync Cloud instances, which alread The PowerSync Cloud and self-hosted columns below apply during the Beta only. Once storage version 4 is generally available, it will become the default for all supported instances. S3 object storage is then also enabled on all PowerSync Cloud instances. For self-hosted deployments, follow the [S3 setup instructions](#self-hosted-s3-setup). -| | Source database | Sync Config | PowerSync Cloud (Beta) | Self-hosted (Beta) | -| --- | --- | --- | --- | --- | -| Storage version 4 | Any | Sync Streams or Sync Rules | Free plan: automatic. Other plans: [opt in](#opt-in). | [Opt in](#opt-in) | -| Incremental reprocessing | MongoDB | Sync Streams | Included with version 4 | Included with version 4 | -| S3 object storage | Any | Sync Streams or Sync Rules | Enabled per instance by PowerSync on request | [Set up S3 object storage](#self-hosted-s3-setup) | +| | Source database | PowerSync Cloud (Beta) | Self-hosted (Beta) | +| --- | --- | --- | --- | +| Storage version 4 | Any | Free plan: automatic. Other plans: [opt in](#opt-in). | [Opt in](#opt-in) | +| Incremental reprocessing | MongoDB | Included with version 4 | Included with version 4 | +| S3 object storage | Any | Enabled per instance by PowerSync on request | [Set up S3 object storage](#self-hosted-s3-setup) | Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/rules/migrate-to-sync-streams). @@ -119,7 +121,7 @@ For self-hosted instances, offloading bucket data to S3 can reduce storage and d Clients connect only to the PowerSync Service and never to the object store, so no client changes are needed. If the object store becomes unreachable, sync is interrupted until it recovers. Clients reconnect and resume automatically. -S3 object storage requires storage version 4 and works with Sync Streams and legacy Sync Rules. It is compatible with all PowerSync Cloud instances. +S3 object storage requires storage version 4. It is compatible with all PowerSync Cloud instances. S3 object storage holds PowerSync's internal sync data. To store files uploaded by your app, use [Attachments](/client-sdks/advanced/attachments). diff --git a/sync/advanced/sync-data-by-time.mdx b/sync/advanced/sync-data-by-time.mdx index ea503e50..b0564046 100644 --- a/sync/advanced/sync-data-by-time.mdx +++ b/sync/advanced/sync-data-by-time.mdx @@ -1,23 +1,18 @@ --- title: "Sync Data by Time with Sync Streams" -description: "Filter and sync data based on time ranges using Sync Streams/Sync Rules, with patterns for recent-only and sliding-window queries." +description: "Filter and sync data based on time ranges using Sync Streams, with patterns for recent-only and sliding-window queries." sidebarTitle: "Sync Data by Time" --- +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/sync-data-by-time.mdx. When you change shared behavior or fix an error here, check whether that page needs the same change. Do not mention Sync Rules on this page. */} + A common need in offline-first apps is syncing data based on time, for example, only syncing issues updated in the last 7 days instead of the entire dataset. You might expect to write something like: ```yaml -# Sync Streams streams: issues_after_start_date: - query: SELECT * FROM issues WHERE updated_at > subscription.parameter('start_at') - -# Sync Rules -bucket_definitions: - issues_after_start_date: - parameters: SELECT request.parameters() ->> 'start_at' as start_at - data: SELECT * FROM issues WHERE updated_at > bucket.start_date + query: SELECT * FROM issues WHERE updated_at > subscription.parameter('start_at') ``` However, this won't work. Here's why. @@ -28,8 +23,6 @@ PowerSync pre-computes and caches which rows belong to which parameters to enabl Additionally, time-based functions like `now()` aren't allowed in parameter expressions because the result changes depending on when the query runs, making pre-computation impossible. -These constraints apply to both Sync Streams and legacy Sync Rules. - This guide covers a few practical workarounds. We are working on a more elegant solution for this problem. When ready, this guide will be updated accordingly. @@ -51,73 +44,38 @@ Update it periodically using a cron job (e.g., with `pg_cron`): UPDATE issues SET updated_this_week = (updated_at > now() - interval '7 days'); ``` - - - ```yaml - config: - edition: 3 - streams: - recent_issues: - auto_subscribe: true - query: SELECT * FROM issues WHERE updated_this_week = true - ``` - - For multiple time ranges, define a stream per range and let the client subscribe to the one it needs: - - ```yaml - config: - edition: 3 - streams: - issues_1week: - query: SELECT * FROM issues WHERE updated_this_week = true - - issues_1month: - query: SELECT * FROM issues WHERE updated_this_month = true - ``` - - The client subscribes to the desired range: - - ```javascript - // Subscribe to one-week range - await db.syncStream('issues_1week').subscribe(); - // Or subscribe to one-month range - await db.syncStream('issues_1month').subscribe(); - ``` - - - ```yaml - bucket_definitions: - recent_issues: - data: - - SELECT * FROM issues WHERE updated_this_week = true - ``` - - For multiple time ranges, add multiple bucket definitions and let the client choose which bucket to sync: - - ```yaml - bucket_definitions: - issues_1week: - parameters: SELECT WHERE request.parameters() ->> 'range' = '1week' - data: - - SELECT * FROM issues WHERE updated_this_week = true - - issues_1month: - parameters: SELECT WHERE request.parameters() ->> 'range' = '1month' - data: - - SELECT * FROM issues WHERE updated_this_month = true - ``` - - The client passes the desired range as a client parameter: - - ```javascript - await db.connect(connector, { - params: { - range: '1week', - }, - }) - ``` - - +Then filter on the column in a stream query: + +```yaml +config: + edition: 3 +streams: + recent_issues: + auto_subscribe: true + query: SELECT * FROM issues WHERE updated_this_week = true +``` + +For multiple time ranges, define a stream per range and let the client subscribe to the one it needs: + +```yaml +config: + edition: 3 +streams: + issues_1week: + query: SELECT * FROM issues WHERE updated_this_week = true + + issues_1month: + query: SELECT * FROM issues WHERE updated_this_month = true +``` + +The client subscribes to the desired range: + +```javascript +// Subscribe to one-week range +await db.syncStream('issues_1week').subscribe(); +// Or subscribe to one-month range +await db.syncStream('issues_1month').subscribe(); +``` This approach works well when you have a small, fixed set of time ranges. However, it requires schema changes and a scheduled job to keep the columns updated. @@ -133,50 +91,25 @@ Instead of pre-defined ranges, create a bucket for each date and let the client Use `substring` to extract the date portion from a timestamp and match it with `=`: -For a complete working example, see the [PowerSync + Supabase: Time-Based Sync demo](https://github.com/powersync-ja/powersync-js/tree/main/demos/react-supabase-time-based-sync). - - - - ```yaml - config: - edition: 3 - streams: - issues_by_date: - query: SELECT * FROM issues WHERE substring(updated_at, 1, 10) = subscription.parameter('date') - ``` - - The client subscribes once per date it wants to sync: - - ```javascript - await db.syncStream('issues_by_date', { date: '2026-01-07' }).subscribe(); - await db.syncStream('issues_by_date', { date: '2026-01-08' }).subscribe(); - await db.syncStream('issues_by_date', { date: '2026-01-09' }).subscribe(); - ``` - - Each subscription can be managed independently — you can subscribe and unsubscribe to individual dates without affecting others. - - - ```yaml - bucket_definitions: - issues_by_update_at: - parameters: SELECT value as date FROM json_each(request.parameters() ->> 'dates') - data: - - SELECT * FROM issues WHERE substring(updated_at, 1, 10) = bucket.date - ``` - - The client passes the dates it wants as client parameters: - - ```javascript - await db.connect(connector, { - params: { - dates: ["2026-01-07", "2026-01-08", "2026-01-09"], - }, - }) - ``` - - - -This gives users full control over which dates to sync, with no schema changes or scheduled jobs required. +```yaml +config: + edition: 3 +streams: + issues_by_date: + query: SELECT * FROM issues WHERE substring(updated_at, 1, 10) = subscription.parameter('date') +``` + +The client subscribes once per date it wants to sync: + +```javascript +await db.syncStream('issues_by_date', { date: '2026-01-07' }).subscribe(); +await db.syncStream('issues_by_date', { date: '2026-01-08' }).subscribe(); +await db.syncStream('issues_by_date', { date: '2026-01-09' }).subscribe(); +``` + +Each subscription can be managed independently — you can subscribe and unsubscribe to individual dates without affecting others. + +This gives users full control over which dates to sync, with no schema changes or scheduled jobs required. For a complete working example, see the [PowerSync + Supabase: Time-Based Sync demo](https://github.com/powersync-ja/powersync-js/tree/main/demos/react-supabase-time-based-sync). The trade-off is granularity. In this example we're using daily buckets. If you need finer precision (hourly), syncing a large range means many buckets, which can degrade sync performance and approach [PowerSync's limit of 1,000 buckets per user](/resources/performance-and-limits#limits). If you use larger buckets (monthly), you lose the ability to filter accurately. @@ -190,69 +123,33 @@ You have to pick a granularity and stick with it. If that's a problem—say, you Combine multiple granularities in a single definition. This lets you use larger buckets (days) for older data and smaller buckets (hours, minutes) for recent data. - - - ```yaml - config: - edition: 3 - streams: - issues_by_partition: - queries: - # By day (e.g., "2026-01-07") - - SELECT * FROM issues WHERE substring(updated_at, 1, 10) = subscription.parameter('partition') - # By hour (e.g., "2026-01-07T14") - - SELECT * FROM issues WHERE substring(updated_at, 1, 13) = subscription.parameter('partition') - # By 10 minutes (e.g., "2026-01-07T14:3") - - SELECT * FROM issues WHERE substring(updated_at, 1, 15) = subscription.parameter('partition') - ``` - - The client subscribes once per partition, mixing granularities as needed: - - ```javascript - await db.syncStream('issues_by_partition', { partition: '2026-01-05' }).subscribe(); - await db.syncStream('issues_by_partition', { partition: '2026-01-06' }).subscribe(); - await db.syncStream('issues_by_partition', { partition: '2026-01-07T10' }).subscribe(); - await db.syncStream('issues_by_partition', { partition: '2026-01-07T11' }).subscribe(); - await db.syncStream('issues_by_partition', { partition: '2026-01-07T12:0' }).subscribe(); - await db.syncStream('issues_by_partition', { partition: '2026-01-07T12:1' }).subscribe(); - await db.syncStream('issues_by_partition', { partition: '2026-01-07T12:2' }).subscribe(); - ``` - - Each query naturally acts as a filter based on the length of the partition value — a day-format partition only matches the day query, an hour-format partition only matches the hour query, and so on. - - - ```yaml - bucket_definitions: - issues_by_time: - parameters: SELECT value as partition FROM json_each(request.parameters() ->> 'partitions') - data: - # By day (e.g., "2026-01-07") - - SELECT * FROM issues WHERE substring(updated_at, 1, 10) = bucket.partition - # By hour (e.g., "2026-01-07T14") - - SELECT * FROM issues WHERE substring(updated_at, 1, 13) = bucket.partition - # By 10 minutes (e.g., "2026-01-07T14:3") - - SELECT * FROM issues WHERE substring(updated_at, 1, 15) = bucket.partition - ``` - - The client then mixes granularities as needed: - - ```javascript - await db.connect(connector, { - params: { - partitions: [ - "2026-01-05", - "2026-01-06", - "2026-01-07T10", - "2026-01-07T11", - "2026-01-07T12:0", - "2026-01-07T12:1", - "2026-01-07T12:2" - ] - }, - }) - ``` - - +```yaml +config: + edition: 3 +streams: + issues_by_partition: + queries: + # By day (e.g., "2026-01-07") + - SELECT * FROM issues WHERE substring(updated_at, 1, 10) = subscription.parameter('partition') + # By hour (e.g., "2026-01-07T14") + - SELECT * FROM issues WHERE substring(updated_at, 1, 13) = subscription.parameter('partition') + # By 10 minutes (e.g., "2026-01-07T14:3") + - SELECT * FROM issues WHERE substring(updated_at, 1, 15) = subscription.parameter('partition') +``` + +The client subscribes once per partition, mixing granularities as needed: + +```javascript +await db.syncStream('issues_by_partition', { partition: '2026-01-05' }).subscribe(); +await db.syncStream('issues_by_partition', { partition: '2026-01-06' }).subscribe(); +await db.syncStream('issues_by_partition', { partition: '2026-01-07T10' }).subscribe(); +await db.syncStream('issues_by_partition', { partition: '2026-01-07T11' }).subscribe(); +await db.syncStream('issues_by_partition', { partition: '2026-01-07T12:0' }).subscribe(); +await db.syncStream('issues_by_partition', { partition: '2026-01-07T12:1' }).subscribe(); +await db.syncStream('issues_by_partition', { partition: '2026-01-07T12:2' }).subscribe(); +``` + +Each query naturally acts as a filter based on the length of the partition value — a day-format partition only matches the day query, an hour-format partition only matches the hour query, and so on. This syncs January 5–6 by day, the morning of January 7 by hour, and the last 30 minutes in 10-minute chunks, without creating hundreds of buckets. diff --git a/sync/grammar/sync-rules/index.mdx b/sync/grammar/sync-rules/index.mdx index 4f7c46d2..1efebf5e 100644 --- a/sync/grammar/sync-rules/index.mdx +++ b/sync/grammar/sync-rules/index.mdx @@ -3,9 +3,13 @@ title: "Grammar Reference (Sync Rules)" description: "Railroad diagram reference for the SQL grammar supported in legacy Sync Rules queries." --- -This page is a formal grammar reference for Sync Rules: it shows the syntax accepted for parameter queries and data queries using railroad diagrams. This page complements the [Supported SQL](/sync/supported-sql) guide, which explains in prose what you can write, with examples and restrictions. + +Sync Rules are deprecated. For the Sync Streams version of this page, see [Grammar Reference](/sync/grammar/sync-streams/index). + -**When to use this page:** If you need to check whether a construct is valid, see how parameter vs data query syntax differs, or you're used to grammar specs, use the diagrams and the "Used by" / "References" links to navigate. For most users just getting started, see [Supported SQL](/sync/supported-sql) and the [Sync Rules](/sync/rules/overview) docs. +This page is a formal grammar reference for Sync Rules: it shows the syntax accepted for parameter queries and data queries using railroad diagrams. This page complements the [Supported SQL](/sync/rules/supported-sql) guide, which explains in prose what you can write, with examples and restrictions. + +**When to use this page:** If you need to check whether a construct is valid, see how parameter vs data query syntax differs, or you're used to grammar specs, use the diagrams and the "Used by" / "References" links to navigate. For most users just getting started, see [Supported SQL](/sync/rules/supported-sql) and the [Sync Rules](/sync/rules/overview) docs. ## ParameterQuery diff --git a/sync/rules/case-sensitivity.mdx b/sync/rules/case-sensitivity.mdx new file mode 100644 index 00000000..13f660cb --- /dev/null +++ b/sync/rules/case-sensitivity.mdx @@ -0,0 +1,15 @@ +--- +title: "Case Sensitivity" +description: "Handle case-sensitive table and column names in PowerSync sync queries, with best practices for lowercase identifiers and quoting strategies." +noindex: true +--- + +{/* Wrapper page: the content is snippets/sync-shared/case-sensitivity.mdx, which also renders at sync/advanced/case-sensitivity.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} + +import CaseSensitivity from '/snippets/sync-shared/case-sensitivity.mdx'; + + +Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. + + + diff --git a/sync/rules/client-id.mdx b/sync/rules/client-id.mdx new file mode 100644 index 00000000..07d440eb --- /dev/null +++ b/sync/rules/client-id.mdx @@ -0,0 +1,15 @@ +--- +title: "Client ID" +description: "Understand PowerSync's requirement for a single text-type primary key column called id." +noindex: true +--- + +{/* Wrapper page: the content is snippets/sync-shared/client-id.mdx, which also renders at sync/advanced/client-id.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} + +import ClientId from '/snippets/sync-shared/client-id.mdx'; + + +Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. + + + diff --git a/sync/rules/client-parameters.mdx b/sync/rules/client-parameters.mdx index cdd33224..fbab9c7b 100644 --- a/sync/rules/client-parameters.mdx +++ b/sync/rules/client-parameters.mdx @@ -3,6 +3,10 @@ title: "Client Parameters" description: "Pass dynamic parameters from the client SDK directly into Sync Rules queries to filter data per user, device, or application context." --- + +Sync Rules are deprecated. For the Sync Streams equivalent, see [Connection Parameters](/sync/streams/parameters#connection-parameters). + + Use client parameters with caution. Please make sure to read the [Security consideration](#security-consideration) section below. diff --git a/sync/rules/data-queries.mdx b/sync/rules/data-queries.mdx index a9892bbe..5341c67d 100644 --- a/sync/rules/data-queries.mdx +++ b/sync/rules/data-queries.mdx @@ -3,6 +3,10 @@ title: "Data Queries" description: "Write Data Queries in Sync Rules to select and filter data for buckets using bucket parameters." --- + +Sync Rules are deprecated. For the Sync Streams equivalent, see [Writing Queries](/sync/streams/queries). + + Data Queries select the data that form part of a [bucket](/architecture/powersync-service#bucket-system), using the bucket [parameters](/sync/rules/overview#parameters). Multiple Data Queries can be specified for a single [bucket definition](/sync/rules/overview#bucket-definition). @@ -19,7 +23,7 @@ Data Queries are used to group data into buckets, so each Data Query must use ev ## Supported SQL -The supported SQL in Data Queries is based on a small subset of the SQL standard syntax. Not all SQL constructs are supported. See [Supported SQL](/sync/supported-sql) for full details. +The supported SQL in Data Queries is based on a small subset of the SQL standard syntax. Not all SQL constructs are supported. See [Supported SQL](/sync/rules/supported-sql) for full details. ## Examples diff --git a/sync/rules/global-buckets.mdx b/sync/rules/global-buckets.mdx index 747727d0..62d30244 100644 --- a/sync/rules/global-buckets.mdx +++ b/sync/rules/global-buckets.mdx @@ -3,6 +3,10 @@ title: "Global Buckets" description: "Set up global buckets in Sync Rules to sync shared reference data to all connected users without per-user filtering or parameter queries." --- + +Sync Rules are deprecated. For the Sync Streams equivalent, see [Global Data](/sync/streams/quickstart#global-data) in the Sync Streams Quickstart. + + Any bucket with no _Parameter Query_ in the bucket definition is automatically a _Global Bucket_. These buckets will be synced to all clients/users. For example, the following Sync Rules sync all `todos` and only unarchived `lists` to all clients/users: diff --git a/sync/rules/many-to-many-join-tables.mdx b/sync/rules/many-to-many-join-tables.mdx index 21a141de..e1de45f5 100644 --- a/sync/rules/many-to-many-join-tables.mdx +++ b/sync/rules/many-to-many-join-tables.mdx @@ -4,11 +4,11 @@ sidebarTitle: "Many-to-Many and Join Tables" description: "Handle many-to-many relationships in Sync Rules using join table strategies." --- -Join tables are often used to implement many-to-many relationships between tables. Join queries are not directly supported in PowerSync Sync Rules, and require some workarounds depending on the use case. This guide contains some recommended strategies. + +Sync Rules are deprecated. Sync Streams support [JOINs](/sync/streams/queries#using-joins) and [nested subqueries](/sync/streams/queries#using-subqueries), which handle most many-to-many relationships directly without the workarounds on this page. See [Many-to-Many with Sync Streams](/sync/streams/examples#many-to-many-relationships). + - -**Using Sync Streams?** Sync Streams support [JOINs](/sync/streams/queries#using-joins) and [nested subqueries](/sync/streams/queries#using-subqueries), which handle most many-to-many relationships directly without the workarounds described here. See [Many-to-Many with Sync Streams](/sync/streams/examples#many-to-many-relationships) for examples. - +Join tables are often used to implement many-to-many relationships between tables. Join queries are not directly supported in PowerSync Sync Rules, and require some workarounds depending on the use case. This guide contains some recommended strategies. **Postgres users:** For Postgres source databases, you can use the [`pg_ivm` extension](https://www.powersync.com/blog/using-pg-ivm-to-enable-joins-in-powersync) to create incrementally maintained materialized views with JOINs that can be referenced directly in Sync Rules. This approach avoids the need to denormalize your schema. diff --git a/sync/rules/migrate-to-sync-streams.mdx b/sync/rules/migrate-to-sync-streams.mdx index 69a2c4f1..e4a56342 100644 --- a/sync/rules/migrate-to-sync-streams.mdx +++ b/sync/rules/migrate-to-sync-streams.mdx @@ -18,7 +18,9 @@ If your Sync Config has a `bucket_definitions:` section, you use Sync Rules and ## Why Migrate? -Beyond matching Sync Rules, Sync Streams add: +{/* TODO: Link to the Sync Rules deprecation announcement once it is published. */} + +Sync Rules are deprecated, and PowerSync is phasing them out in favor of Sync Streams. Beyond matching Sync Rules, Sync Streams add: - **More expressive queries:** Stream queries support JOINs, [CTEs](/sync/streams/ctes), subqueries, and [multiple queries per stream](/sync/streams/queries#multiple-queries-per-stream), with syntax closer to plain SQL. You write one query instead of separate `parameters:` and `data:` blocks. - **On-demand syncing:** Define a stream once, then subscribe from your app one or more times with different parameters. Each subscription has its own lifecycle, so two screens or browser tabs can subscribe to the same stream independently. With Sync Rules, Client Parameters approximate this. You have to aggregate the parameter values yourself across screens and tabs, and remove them when they are no longer needed. diff --git a/sync/rules/multiple-client-versions.mdx b/sync/rules/multiple-client-versions.mdx new file mode 100644 index 00000000..f6dd3115 --- /dev/null +++ b/sync/rules/multiple-client-versions.mdx @@ -0,0 +1,39 @@ +--- +title: "Multiple Client Versions with Sync Rules" +sidebarTitle: "Multiple Client Versions" +description: "Handle multiple client app versions that require different output schemas from legacy Sync Rules." +--- + +{/* Split page: the Sync Streams version of this page is sync/advanced/multiple-client-versions.mdx. Sync Rules are deprecated: keep this page accurate, but do not prioritize additions. When you fix an error here, check whether the Sync Streams page needs the same fix. */} + + +Sync Rules are deprecated. For the Sync Streams version of this page, see [Multiple Client Versions](/sync/advanced/multiple-client-versions). + + +When schema changes are additive, old clients ignore the new tables and columns, and no special handling is required. More drastic changes, such as renaming tables or changing a table's structure, can break older app versions that are still in use. In these cases, define separate versions of the affected bucket definitions so that each client version receives the tables and columns it expects. + +## Versioning with Client Parameters + +Clients pass their version to the PowerSync Service as a [client parameter](/sync/rules/client-parameters), and parameter queries filter on it so that each client only receives the buckets for its version. + +For example, suppose a new app version changes the structure of the `assets` table in its [client-side schema](/intro/setup-guide#define-your-client-side-schema), defining it as `assets_v2`, while older app versions still define `assets`. Define a second bucket definition alongside the existing one, using an alias to map the source `assets` table to the new client-side name, and filter each on a `schema_version` client parameter: + +```yaml +# Client passes in: "params": {"schema_version": } +bucket_definitions: + user_assets: + parameters: SELECT request.user_id() AS user_id + WHERE request.parameters() ->> 'schema_version' = '1' + data: + - SELECT * FROM assets WHERE user_id = bucket.user_id + + user_assets_v2: + parameters: SELECT request.user_id() AS user_id + WHERE request.parameters() ->> 'schema_version' = '2' + data: + - SELECT * FROM assets AS assets_v2 WHERE user_id = bucket.user_id +``` + + + Handle queries based on parameters set by the client with care. The client can send any value for these parameters, so it's not a good place to do authorization. If the parameter must be authenticated, use parameters from the JWT instead. + diff --git a/sync/rules/organize-data-into-buckets.mdx b/sync/rules/organize-data-into-buckets.mdx index 130b0b56..c42b3e45 100644 --- a/sync/rules/organize-data-into-buckets.mdx +++ b/sync/rules/organize-data-into-buckets.mdx @@ -3,6 +3,10 @@ title: "Organize Data Into Buckets" description: "Design Sync Rules to organize data into global and user-filtered buckets." --- + +Sync Rules are deprecated. For the Sync Streams equivalent, see the [Sync Streams Quickstart](/sync/streams/quickstart). + + Designing your Sync Rules is about _organizing data into buckets_, and creating the bucket definitions accordingly. Each [bucket definition](/sync/rules/overview#bucket-definition) defines a set of tables/collections and rows/documents to sync. * If there's some data you want to sync to _all_ your users/clients, you can add bucket definitions for one or more [Global Buckets](/sync/rules/global-buckets). This is the simplest way to get started with PowerSync. @@ -44,7 +48,7 @@ bucket_definitions: - The supported SQL in _Parameter Queries_ and _Data Queries_ is based on a small subset of the SQL standard syntax. Not all SQL constructs are supported. See [Supported SQL](/sync/supported-sql). + The supported SQL in _Parameter Queries_ and _Data Queries_ is based on a small subset of the SQL standard syntax. Not all SQL constructs are supported. See [Supported SQL](/sync/rules/supported-sql). diff --git a/sync/rules/overview.mdx b/sync/rules/overview.mdx index 17921d0e..edeb3c6a 100644 --- a/sync/rules/overview.mdx +++ b/sync/rules/overview.mdx @@ -4,15 +4,17 @@ sidebarTitle: "Overview & Key Concepts" description: "Understand legacy Sync Rules for controlling which data syncs to each client." --- -Sync Rules are PowerSync's original system for partial sync, using YAML bucket definitions. They remain supported for existing projects but are considered legacy. +Sync Rules are PowerSync's original system for partial sync, using YAML bucket definitions. - -**Sync Streams Recommended** +{/* TODO: Link to the Sync Rules deprecation announcement on releases.powersync.com once it is published. */} -[Sync Streams](/sync/streams/overview) are the recommended approach to partial sync for both new and existing projects. They support everything Sync Rules do, plus more expressive queries (including JOIN support), on-demand syncing, and a simpler developer experience (e.g. React hooks that manage subscriptions automatically). + +**Sync Rules are deprecated** -You can migrate in a few clicks. Click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to generate a draft from your current config. See [Migrate to Sync Streams](/sync/rules/migrate-to-sync-streams) for details. - +PowerSync is phasing out Sync Rules in favor of [Sync Streams](/sync/streams/overview), which support everything Sync Rules do and add on-demand syncing, JOINs, CTEs, and subqueries. Nothing changes for your instance today: Sync Rules keep working and stay supported while you migrate. New Sync Config features are added to Sync Streams only. + +To migrate, click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to convert your current config. Migrating does not change what your app syncs. See [Migrate to Sync Streams](/sync/rules/migrate-to-sync-streams). + Sync Rules are defined in a YAML file. For PowerSync Cloud, they are edited and deployed to a specific PowerSync instance in the [PowerSync Dashboard](/tools/powersync-dashboard#project-&-instance-level). For self-hosting setups, they are defined as part of your [instance configuration](/configuration/powersync-service/self-hosted-instances). @@ -52,7 +54,7 @@ The following values can be selected in Parameter Queries: - **Client Parameters** (see below) - **Values From a Table/Collection** (see below) -See [Parameter Queries](/sync/rules/parameter-queries) for more details and examples. Also see [Supported SQL](/sync/supported-sql) for limitations. +See [Parameter Queries](/sync/rules/parameter-queries) for more details and examples. Also see [Supported SQL](/sync/rules/supported-sql) for limitations. ### Authentication Parameters @@ -72,11 +74,11 @@ Clients can specify **Client Parameters** when connecting to PowerSync (i.e. whe ```yaml Example of selecting a Client Parameter in a Parameter Query parameters: SELECT (request.parameters() ->> 'current_project') as current_project ``` -The `->>` operator in the above example extracts a value from a string containing JSON (which is the format provided by ``request.parameters()``). See [Operators and Functions](/sync/supported-sql#operators) +The `->>` operator in the above example extracts a value from a string containing JSON (which is the format provided by ``request.parameters()``). See [Operators and Functions](/sync/rules/supported-sql#operators) A client can pass any value for a Client Parameter. Hence, Client Parameters should always be treated with care, and should [not be used](/sync/rules/client-parameters#security-consideration) for access control purposes. -That being said, Client Parameters can be useful for use cases such as syncing different buckets based on state in the client app, for example only syncing data for the project currently selected, or syncing different buckets based on the client version ([see here](/sync/advanced/multiple-client-versions)). +That being said, Client Parameters can be useful for use cases such as syncing different buckets based on state in the client app, for example only syncing data for the project currently selected, or syncing different buckets based on the client version ([see here](/sync/rules/multiple-client-versions)). See [Client Parameters](/sync/rules/client-parameters) and [Parameter Queries](/sync/rules/parameter-queries) for more details and examples. @@ -103,7 +105,7 @@ data: - SELECT * FROM lists WHERE owner_id = bucket.user_id ``` -See [Data Queries](/sync/rules/data-queries) for more details and examples. Also see [Supported SQL](/sync/supported-sql) for limitations. +See [Data Queries](/sync/rules/data-queries) for more details and examples. Also see [Supported SQL](/sync/rules/supported-sql) for limitations. ### Global Buckets @@ -127,6 +129,3 @@ This architecture is key to the scalability and performance of PowerSync. See th Designing your Sync Rules is basically about _organizing data into buckets_, and creating the bucket definitions accordingly. See [Organize Data Into Buckets](/sync/rules/organize-data-into-buckets). - - - diff --git a/sync/rules/parameter-queries.mdx b/sync/rules/parameter-queries.mdx index 47372f0e..2b532b48 100644 --- a/sync/rules/parameter-queries.mdx +++ b/sync/rules/parameter-queries.mdx @@ -3,6 +3,10 @@ title: "Parameter Queries" description: "Define bucket parameters in Sync Rules using Parameter Queries with JWT claims and client parameters." --- + +Sync Rules are deprecated. For the Sync Streams equivalent, see [Using Parameters](/sync/streams/parameters). + + _Parameter Queries_ allow [parameters](/sync/rules/overview#parameters) to be defined on a [bucket](/sync/rules/overview#bucket-definition) to group data. Each [bucket](/sync/rules/overview#bucket-definition) can have zero or more Parameter Queries. @@ -27,7 +31,7 @@ The following functions allow you to select Authentication Parameters in your Pa | `request.user_id()` | Returns the JWT subject (`sub`). Same as `request.jwt() ->> 'sub'` (see below) | | `request.jwt()` | Returns the entire (signed) JWT payload as a JSON string. If there are other _claims_ in your JWT (in addition to the user ID), you can select them from this JSON string. | -Since `request.jwt()` is a string containing JSON, use the `->>` [operator](/sync/supported-sql#operators) to select values from it: +Since `request.jwt()` is a string containing JSON, use the `->>` [operator](/sync/rules/supported-sql#operators) to select values from it: ```sql request.jwt() ->> 'sub' -- the 'subject' of the JWT - same as `request.user_id() @@ -119,7 +123,7 @@ bucket_definitions: ## Supported SQL -The supported SQL in Parameter Queries is based on a small subset of the SQL standard syntax. Not all SQL constructs are supported. See [Supported SQL](/sync/supported-sql) for full details. +The supported SQL in Parameter Queries is based on a small subset of the SQL standard syntax. Not all SQL constructs are supported. See [Supported SQL](/sync/rules/supported-sql) for full details. ## Usage Examples @@ -203,7 +207,7 @@ For more advanced details on many-to-many relationships and join tables, see [th ### Expanding JSON Array Into Multiple Parameters -Using the `json_each()` [function](/sync/supported-sql#functions) and `->` [operator](/sync/supported-sql#operators), we can expand a parameter that is a JSON array into multiple rows, thereby filtering by multiple parameter values: +Using the `json_each()` [function](/sync/rules/supported-sql#functions) and `->` [operator](/sync/rules/supported-sql#operators), we can expand a parameter that is a JSON array into multiple rows, thereby filtering by multiple parameter values: ```yaml bucket_definitions: diff --git a/sync/rules/partitioned-tables.mdx b/sync/rules/partitioned-tables.mdx new file mode 100644 index 00000000..c0918acf --- /dev/null +++ b/sync/rules/partitioned-tables.mdx @@ -0,0 +1,37 @@ +--- +title: "Partitioned Tables (Postgres) with Sync Rules" +sidebarTitle: "Partitioned Tables (Postgres)" +description: "Sync data from Postgres partitioned tables in legacy Sync Rules using wildcard table name matching." +--- + +{/* Split page: the Sync Streams version of this page is sync/advanced/partitioned-tables.mdx. Sync Rules are deprecated: keep this page accurate, but do not prioritize additions. When you fix an error here, check whether the Sync Streams page needs the same fix. */} + + +Sync Rules are deprecated. For the Sync Streams version of this page, see [Partitioned Tables (Postgres)](/sync/advanced/partitioned-tables). + + +For partitioned tables in Postgres, each individual partition is replicated and processed using [Sync Rules](/sync/rules/overview). + +To use the same queries and same output table name for each partition, use `%` for wildcard suffix matching of the table name: + +```yaml +bucket_definitions: + by_user: + # Use wildcard in a parameter query + parameters: SELECT id AS user_id FROM "users_%" + data: + # Use wildcard in a data query + - SELECT * FROM "todos_%" AS todos WHERE user_id = bucket.user_id +``` + +The wildcard character can only be used as the last character in the table name. To match tables across multiple schemas instead, see [Wildcard Schemas](/sync/advanced/schemas-and-connections#wildcard-schemas-postgres). + +When using wildcard table names, the original table suffix is available as the special `_table_suffix` column, which you can use to filter rows: + +```sql +SELECT * FROM "todos_%" AS todos WHERE _table_suffix != 'archived' +``` + +When no table alias is provided, the original table name is preserved. + +`publish_via_partition_root` on the publication is not supported. The individual partitions must be published. diff --git a/sync/rules/prioritized-sync.mdx b/sync/rules/prioritized-sync.mdx new file mode 100644 index 00000000..52f58ff4 --- /dev/null +++ b/sync/rules/prioritized-sync.mdx @@ -0,0 +1,126 @@ +--- +title: "Prioritized Sync with Sync Rules" +sidebarTitle: "Prioritized Sync" +description: "Assign sync priorities to bucket definitions in legacy Sync Rules so that important data syncs before the rest." +--- + +{/* Split page: the Sync Streams version of this page is sync/streams/prioritized-sync.mdx. Sync Rules are deprecated: keep this page accurate, but do not prioritize additions. When you fix an error here, check whether the Sync Streams page needs the same fix. */} + + +Sync Rules are deprecated. For the Sync Streams version of this page, see [Prioritized Sync](/sync/streams/prioritized-sync). + + +## Overview + +PowerSync supports defining sync priorities, which allows you to control the sync order for different data. This is useful when certain data should be available sooner than others. + +In Sync Rules, you assign priorities to bucket definitions. The priority determines when data in that bucket syncs relative to other buckets. + + +**Availability** + +This feature was introduced in version **1.7.1** of the PowerSync Service, and in the following SDK versions: +- [Flutter v1.12.0](/client-sdks/reference/flutter) +- [React Native v1.18.1](/client-sdks/reference/react-native-and-expo) +- [JavaScript Web v1.14.2](/client-sdks/reference/javascript-web) +- [Kotlin v1.0.0-BETA26](/client-sdks/reference/kotlin) +- [Swift v1.0.0-Beta.8](/client-sdks/reference/swift) +- [.NET v0.0.6-alpha.1](/client-sdks/reference/dotnet) + + +## Why Use Sync Priorities? + +PowerSync's standard sync protocol ensures that: +- The local data view is only updated when a fully consistent checkpoint is available. +- All pending local changes must be uploaded, acknowledged, and synced back before new data is applied. + +While this guarantees consistency, it can lead to delays, especially for large datasets or continuous client-side updates. Sync priorities provide a way to speed up syncing of high-priority data while still maintaining overall integrity. + +## How It Works + +Each bucket is assigned a priority value between 0 and 3, where: + +- 0 is the highest priority and has special behavior (detailed below). +- 3 is the default and lowest priority. +- Lower numbers indicate higher priority. + +Higher-priority data syncs first, and lower-priority data syncs later. If you only use a single priority, there is no difference between priorities 1-3. The difference only comes in when you use multiple different priorities. + +## Syntax and Configuration + +Define priorities using the `priority` YAML key on a bucket definition, or with the `_priority` attribute inside a parameter query: + +```yaml +bucket_definitions: + # Using the `priority` YAML key + user_data: + priority: 1 + parameters: SELECT request.user_id() AS id WHERE ... + data: + # ... + + # Using the `_priority` attribute (useful for multiple parameter queries with different priorities) + project_data: + parameters: SELECT id AS project_id, 2 AS _priority FROM projects WHERE ... + data: + # ... +``` + + +Priorities must be static and cannot depend on row values within a parameter query. + + +## Example: Syncing Lists Before Todos + +Consider a scenario where you want to display lists immediately while loading todos in the background. This approach allows users to view and interact with lists right away without waiting for todos to sync. + +```yaml +bucket_definitions: + user_lists: + priority: 1 # Syncs first + parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() + data: + - SELECT * FROM lists WHERE id = bucket.list_id + + user_todos: + priority: 2 # Syncs after lists + parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() + data: + - SELECT * FROM todos WHERE list_id = bucket.list_id +``` + +The `user_lists` bucket syncs first (priority 1), allowing users to see and interact with their lists immediately. The `user_todos` bucket syncs afterward (priority 2), loading in the background. + +## Behavioral Considerations + +- **Interruption for Higher Priority Data:** Syncing lower-priority data _may_ be interrupted if new data for higher-priority buckets arrives. +- **Local Changes & Consistency:** If local writes fail due to validation or permission issues, they are only reverted after _all_ data has synced. +- **Deleted Data:** Deleted data may only be removed after _all_ priorities have completed syncing. +- **Data Ordering:** Lower-priority data will never appear before higher-priority data. + +## Special Case: Priority 0 + +Priority 0 buckets sync regardless of pending uploads. + +For example, in a collaborative document editing app (e.g., using Yjs), each change is stored as a separate row. Since out-of-order updates don't affect document integrity, Priority 0 can ensure immediate availability of updates. + +Caution: If misused, Priority 0 may cause flickering or inconsistencies, as updates could arrive out of order. + +## Consistency Considerations + +PowerSync's full consistency guarantees only apply once all priorities have completed syncing. + +When higher-priority data is synced, all inserts and updates at that priority level will be consistent. However, deletes are only applied when the full sync completes, so you may still have some stale data at those priority levels. + +Consider the following example: + +Imagine a task management app where users create lists and todos. Some users have millions of todos. To improve first-load speed: + +- Lists are assigned Priority 1, syncing first to allow UI rendering. +- Todos are assigned Priority 2, loading in the background. + +Now, if another user adds new todos, it's possible for the list count (synced at Priority 1) to temporarily not match the actual todos (synced at Priority 2). If real-time accuracy is required, both lists and todos should use the same priority. + +## Client-Side Considerations + +The client SDK APIs for tracking sync status per priority are the same for Sync Rules and Sync Streams: `waitForFirstSync(priority)`, `SyncStatus.priorityStatusEntries()`, and `SyncStatus.statusForPriority(priority)`. See [Client-Side Considerations](/sync/streams/prioritized-sync#client-side-considerations) on the Sync Streams page for details and a Flutter example. diff --git a/sync/rules/schemas-and-connections.mdx b/sync/rules/schemas-and-connections.mdx new file mode 100644 index 00000000..46f34784 --- /dev/null +++ b/sync/rules/schemas-and-connections.mdx @@ -0,0 +1,15 @@ +--- +title: "Schemas and Connections" +description: "Configure Postgres schema usage in sync queries, including wildcard schemas for schema-per-tenant setups, and connect to high-availability replicas." +noindex: true +--- + +{/* Wrapper page: the content is snippets/sync-shared/schemas-and-connections.mdx, which also renders at sync/advanced/schemas-and-connections.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} + +import SchemasAndConnections from '/snippets/sync-shared/schemas-and-connections.mdx'; + + +Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. Configuration examples use Sync Streams syntax. + + + diff --git a/sync/rules/sharded-databases.mdx b/sync/rules/sharded-databases.mdx new file mode 100644 index 00000000..8ec1b0a7 --- /dev/null +++ b/sync/rules/sharded-databases.mdx @@ -0,0 +1,15 @@ +--- +title: "Sharded Databases" +description: "Sync data from sharded Postgres databases with per-shard PowerSync connection configuration." +noindex: true +--- + +{/* Wrapper page: the content is snippets/sync-shared/sharded-databases.mdx, which also renders at sync/advanced/sharded-databases.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} + +import ShardedDatabases from '/snippets/sync-shared/sharded-databases.mdx'; + + +Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. + + + diff --git a/sync/rules/storage-version-4.mdx b/sync/rules/storage-version-4.mdx new file mode 100644 index 00000000..10d461f9 --- /dev/null +++ b/sync/rules/storage-version-4.mdx @@ -0,0 +1,39 @@ +--- +title: "Storage Version 4 with Sync Rules" +sidebarTitle: "Storage Version 4" +description: "What storage version 4 and S3 object storage mean for instances that still use Sync Rules." +--- + +{/* Split page: the Sync Streams version of this page is sync/advanced/storage-version-4.mdx. Sync Rules are deprecated: keep this page accurate, but do not prioritize additions. This page stays short on purpose and links to the Sync Streams page for details, because incremental reprocessing requires Sync Streams. When you fix an error here, check whether the Sync Streams page needs the same fix. */} + + +Sync Rules are deprecated. For the Sync Streams version of this page, see [Storage Version 4](/sync/advanced/storage-version-4). + + +Storage version 4 is a new version of the format the PowerSync Service uses to store the data it syncs to clients. It is in [Beta](/resources/feature-status) as of PowerSync Service v1.26.0. Once it is generally available, it becomes the default for all supported instances. + +With Sync Rules, storage version 4 provides: + +- Faster sync and faster reprocessing after a deployment. +- [S3 object storage](/sync/advanced/storage-version-4#s3-object-storage): larger blocks of synced data move from the storage database to S3. This reduces load on the storage database when many clients sync at once or sync large amounts of data. During the Beta, PowerSync enables it per instance on PowerSync Cloud. Self-hosted instances follow the [S3 setup instructions](/sync/advanced/storage-version-4#self-hosted-s3-setup). + +[Incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing) requires Sync Streams. With Sync Rules, every deployment still reprocesses all data, and clients download all their data again. To use incremental reprocessing, [migrate to Sync Streams](/sync/rules/migrate-to-sync-streams). + +## Opt In + +Free plan instances on PowerSync Cloud are upgraded automatically during the Beta. On other plans, and on self-hosted instances, add `storage_version: 4` to the `config` block of your Sync Rules and deploy them: + +```yaml +config: + storage_version: 4 + +bucket_definitions: + user_lists: + parameters: SELECT request.user_id() AS user_id + data: + - SELECT * FROM lists WHERE owner_id = bucket.user_id +``` + +Self-hosted instances must use MongoDB [bucket storage](/architecture/powersync-service#bucket-storage). Postgres bucket storage is not supported with version 4. + +The first deployment on version 4 reprocesses all data in the background, and clients download their data again once. diff --git a/sync/rules/supported-sql.mdx b/sync/rules/supported-sql.mdx new file mode 100644 index 00000000..ea7ef290 --- /dev/null +++ b/sync/rules/supported-sql.mdx @@ -0,0 +1,159 @@ +--- +title: "Supported SQL with Sync Rules" +sidebarTitle: "Supported SQL" +description: "Reference for SQL syntax, operators, built-in functions, and type casting supported in legacy Sync Rules queries." +--- + +{/* Split page: the Sync Streams version of this page is sync/supported-sql.mdx. Sync Rules are deprecated: keep this page accurate, but do not prioritize additions. When you fix an error here, check whether the Sync Streams page needs the same fix. */} + + +Sync Rules are deprecated. For the Sync Streams version of this page, see [Supported SQL](/sync/supported-sql). + + +This guide explains the SQL supported in [Sync Rules](/sync/rules/overview) parameter queries and data queries: what you can write, with examples and restrictions. + +For the exact syntax the compiler accepts, with railroad diagrams and grammar-rule references, see the [Sync Rules grammar reference](/sync/grammar/sync-rules/index). + + + Some fundamental restrictions on the usage of SQL expressions are: + + 1. They must be deterministic: no random or time-based functions. + 2. No external state can be used. + 3. They must operate on data available within a single row/document. For example, no aggregation functions are allowed. + + For parameter-specific WHERE restrictions, see [Filtering: WHERE Clause](#filtering-where-clause). + + +## Query Syntax + +The supported SQL is based on a small subset of the standard SQL syntax: + +- Simple `SELECT` with column selection +- `WHERE` filtering on parameters (see [Filtering: WHERE Clause](#filtering-where-clause)) +- A limited set of [operators](#operators) and [functions](#functions) + +**Not supported:** subqueries, JOINs, CTEs, aggregation, sorting, or set operations (`GROUP BY`, `ORDER BY`, `LIMIT`, `UNION`, etc.). + +## Filtering: WHERE Clause + +Sync Rules queries support a subset of SQL `WHERE` syntax. Allowed operators and combinations are more restrictive than standard SQL. + +**`=` and `IS NULL`:** Compare a row column to a static value or a bucket parameter: + +```sql +-- Static value +WHERE status = 'active' +WHERE deleted_at IS NULL + +-- Bucket parameter +WHERE owner_id = bucket.user_id +``` + +**`AND`:** Supported in both Parameter Queries and Data Queries. In Parameter Queries, each condition may match a different parameter. However, you cannot combine two `IN` expressions on parameters in the same `AND`; split them into separate Parameter Queries instead. + +```sql +-- Supported: parameter condition + row-value condition +WHERE users.id = request.user_id() + AND users.is_admin = true + +-- Not supported: two IN expressions on parameters in the same AND +-- WHERE bucket.list_id IN lists.allowed_ids +-- AND bucket.org_id IN lists.allowed_org_ids +``` + +**`OR`:** Supported when both sides of the `OR` reference the exact same set of parameters. If the two sides use different parameters, use separate parameter queries instead. + +```sql +-- Supported: both sides reference the same parameter +WHERE lists.owner_id = request.user_id() + OR lists.shared_with = request.user_id() + +-- Not supported: sides reference different parameters +-- WHERE lists.owner_id = request.user_id() +-- OR lists.org_id = bucket.org_id +``` + +**`NOT`:** Supported for simple row-value conditions. Not supported on parameter-matching expressions. + +```sql +-- Supported +WHERE status != 'archived' +WHERE deleted_at IS NOT NULL +WHERE NOT users.is_admin = true + +-- Not supported in parameter queries +-- WHERE NOT users.id = request.user_id() +``` + +## Operators + +Operators can be used in `WHERE` clauses and in `SELECT` expressions. When filtering on parameters (e.g. `request.user_id()`, `bucket.user_id`), some combinations are restricted. See [Filtering: WHERE Clause](#filtering-where-clause). + + + + - **Comparison:** `=`, `!=`, `<`, `>`, `<=`, `>=` — If either side is `null`, the result is `null`. + - **Null:** `IS NULL`, `IS NOT NULL` + + + - **Logical:** `AND`, `OR`, `NOT` — See [Filtering: WHERE Clause](#filtering-where-clause) for restrictions when filtering on parameters. + - **Mathematical:** `+`, `-`, `*`, `/` + + + - `||` — Joins two text values together. + + + - `json -> 'path'` — Returns the value as a JSON string. + - `json ->> 'path'` — Returns the extracted value. + + + - `left IN right` — Returns true if `left` is in the `right` JSON array. In Data Queries, `left` must be a row column and `right` cannot be a bucket parameter. In Parameter Queries, either side may be a parameter. + + + +## Functions + +Functions can be used to transform columns/fields before being synced to a client. They operate on row data or parameters. Type names below (`text`, `integer`, `real`, `blob`, `null`) refer to [SQLite storage classes](https://www.sqlite.org/datatype3.html). + +Most functions are from [SQLite built-in functions](https://www.sqlite.org/lang_corefunc.html) and [SQLite JSON functions](https://www.sqlite.org/json1.html). + + + + - **[upper(text)](https://www.sqlite.org/lang_corefunc.html#upper)** — Convert text to upper case. + - **[lower(text)](https://www.sqlite.org/lang_corefunc.html#lower)** — Convert text to lower case. + - **[substring(text, start, length)](https://www.sqlite.org/lang_corefunc.html#substr)** — Extracts a portion of a string based on specified start index and length. Start index is 1-based. Example: `substring(created_at, 1, 10)` returns the date portion of the timestamp. + - **[instr(string, substring)](https://www.sqlite.org/lang_corefunc.html#instr)** — Finds the first occurrence of the substring within the string and returns the number of prior characters plus 1, or 0 if the substring is not found. Useful for locating a delimiter in compound strings. For example, `substring(value, 1, instr(value, '|') - 1)` extracts the portion before a `|` character. + - **[hex(data)](https://www.sqlite.org/lang_corefunc.html#hex)** — Convert blob or text data to hexadecimal text. + - **base64(data)** — Convert blob or text data to base64 text. + - **[length(data)](https://www.sqlite.org/lang_corefunc.html#length)** — For text, return the number of characters. For blob, return the number of bytes. For null, return null. For integer and real, convert to text and return the number of characters. + + + - `CAST(x AS type)` or `x :: type` — Cast to `text`, `numeric`, `integer`, `real`, or `blob`. See [Type mapping](/sync/types) and [SQLite types](https://www.sqlite.org/datatype3.html). + - **[typeof(data)](https://www.sqlite.org/lang_corefunc.html#typeof)** — Returns `text`, `integer`, `real`, `blob`, or `null`. + + + - **[json_each(data)](https://www.sqlite.org/json1.html#jeach)** — Expands a JSON array or object from a request or token parameter into a set of parameter rows. Example: `SELECT value AS project_id FROM json_each(request.jwt() -> 'project_ids')`. See [Expanding JSON Array Into Multiple Parameters](/sync/rules/parameter-queries#expanding-json-array-into-multiple-parameters). + - **[json_extract(data, path)](https://www.sqlite.org/json1.html#jex)** — Same as `->>` operator, but the path must start with `$.` + - **[json_array_length(data)](https://www.sqlite.org/json1.html#jarraylen)** — Given a JSON array (as text), returns the length of the array. If data is null, returns null. If the value is not a JSON array, returns 0. + - **[json_valid(data)](https://www.sqlite.org/json1.html#jvalid)** — Returns 1 if the data can be parsed as JSON, 0 otherwise. + - **json_keys(data)** — Returns the set of keys of a JSON object as a JSON array. Example: `SELECT * FROM items WHERE bucket.user_id IN json_keys(permissions_json)`. + + + - **[ifnull(x, y)](https://www.sqlite.org/lang_corefunc.html#ifnull)** — Returns x if non-null, otherwise returns y. + + + - **[iif(x, y, z)](https://www.sqlite.org/lang_corefunc.html#iif)** — Returns y if x is true, otherwise returns z. + + + - **[unixepoch(time-value, [modifier])](https://www.sqlite.org/lang_datefunc.html)** — Returns a time-value as Unix timestamp. If modifier is "subsec", the result is a floating point number, with milliseconds included in the fraction. The time-value argument is required. This function cannot be used to get the current time. + - **[datetime(time-value, [modifier])](https://www.sqlite.org/lang_datefunc.html)** — Returns a time-value as a date and time string, in the format YYYY-MM-DD HH:MM:SS. If the specifier is "subsec", milliseconds are also included. If the modifier is "unixepoch", the argument is interpreted as a Unix timestamp. Both modifiers can be included: `datetime(timestamp, 'unixepoch', 'subsec')`. The time-value argument is required. This function cannot be used to get the current time. + - **[uuid_blob(id)](https://sqlite.org/src/file/ext/misc/uuid.c)** — Convert a UUID string to bytes. + + + - **[ST_AsGeoJSON(geometry)](/client-sdks/advanced/gis-data-postgis)** — Convert [PostGIS](/client-sdks/advanced/gis-data-postgis) (in Postgres) geometry from WKB to GeoJSON. Combine with JSON operators to extract specific fields. + - **[ST_AsText(geometry)](/client-sdks/advanced/gis-data-postgis)** — Convert [PostGIS](/client-sdks/advanced/gis-data-postgis) (in Postgres) geometry from WKB to Well-Known Text (WKT). + - **[ST_X(point)](/client-sdks/advanced/gis-data-postgis)** — Get the X coordinate of a [PostGIS](/client-sdks/advanced/gis-data-postgis) point (in Postgres). + - **[ST_Y(point)](/client-sdks/advanced/gis-data-postgis)** — Get the Y coordinate of a [PostGIS](/client-sdks/advanced/gis-data-postgis) point (in Postgres). + + + +If you need an operator or function not listed, [contact us](/resources/contact-us) so we can consider adding it. diff --git a/sync/rules/sync-data-by-time.mdx b/sync/rules/sync-data-by-time.mdx new file mode 100644 index 00000000..d8b32a7e --- /dev/null +++ b/sync/rules/sync-data-by-time.mdx @@ -0,0 +1,174 @@ +--- +title: "Sync Data by Time with Sync Rules" +sidebarTitle: "Sync Data by Time" +description: "Filter and sync data based on time ranges using legacy Sync Rules, with patterns for recent-only and sliding-window queries." +--- + +{/* Split page: the Sync Streams version of this page is sync/advanced/sync-data-by-time.mdx. Sync Rules are deprecated: keep this page accurate, but do not prioritize additions. When you fix an error here, check whether the Sync Streams page needs the same fix. */} + + +Sync Rules are deprecated. For the Sync Streams version of this page, see [Sync Data by Time](/sync/advanced/sync-data-by-time). + + +A common need is syncing data based on time, for example, only syncing issues updated in the last 7 days instead of the entire dataset. You might expect to write something like: + +```yaml +bucket_definitions: + issues_after_start_date: + parameters: SELECT request.parameters() ->> 'start_at' AS start_at + data: + - SELECT * FROM issues WHERE updated_at > bucket.start_at +``` + +However, this does not work. Here's why. + +## The Problem + +PowerSync pre-computes and caches which rows belong to which bucket parameters to enable efficient streaming. This means parameter-based filtering is limited to equality checks (`=`, `IN`, `IS NULL`). Range operators like `>`, `<`, `>=`, or `<=` are not supported on parameters. + +Additionally, time-based functions like `now()` are not allowed in parameter expressions because the result changes depending on when the query runs, making pre-computation impossible. + +This guide covers a few practical workarounds. + +## Workarounds + +### 1: Pre-Defined Time Ranges + +Add a boolean column to your table that indicates whether a row falls within a specific time range. Keep this column updated in your source database using a scheduled job. + +For example, add an `updated_this_week` column: + +```sql +ALTER TABLE issues ADD COLUMN updated_this_week BOOLEAN DEFAULT false; +``` + +Update it periodically using a cron job (e.g., with `pg_cron`): + +```sql +UPDATE issues SET updated_this_week = (updated_at > now() - interval '7 days'); +``` + +Then filter on the column in a data query: + +```yaml +bucket_definitions: + recent_issues: + data: + - SELECT * FROM issues WHERE updated_this_week = true +``` + +For multiple time ranges, add multiple bucket definitions and let the client choose which bucket to sync: + +```yaml +bucket_definitions: + issues_1week: + parameters: SELECT WHERE request.parameters() ->> 'range' = '1week' + data: + - SELECT * FROM issues WHERE updated_this_week = true + + issues_1month: + parameters: SELECT WHERE request.parameters() ->> 'range' = '1month' + data: + - SELECT * FROM issues WHERE updated_this_month = true +``` + +The client passes the desired range as a [client parameter](/sync/rules/client-parameters): + +```javascript +await db.connect(connector, { + params: { + range: '1week', + }, +}) +``` + +This approach works well when you have a small, fixed set of time ranges. However, it requires schema changes and a scheduled job to keep the columns updated, and it is limited to pre-defined time ranges. + +If you need more flexibility, such as letting users pick arbitrary date ranges, see Workaround 2 below. + +### 2: Buckets Per Date + +Instead of pre-defined ranges, create a bucket for each date and let the client specify which dates to sync. + +Use `substring` to extract the date portion from a timestamp and match it with `=`: + +```yaml +bucket_definitions: + issues_by_update_at: + parameters: SELECT value AS date FROM json_each(request.parameters() ->> 'dates') + data: + - SELECT * FROM issues WHERE substring(updated_at, 1, 10) = bucket.date +``` + +The client passes the dates it wants as client parameters: + +```javascript +await db.connect(connector, { + params: { + dates: ["2026-01-07", "2026-01-08", "2026-01-09"], + }, +}) +``` + +This gives users full control over which dates to sync, with no schema changes or scheduled jobs required. + +The trade-off is granularity. In this example we're using daily buckets. If you need finer precision (hourly), syncing a large range means many buckets, which can degrade sync performance and approach [PowerSync's limit of 1,000 buckets per user](/resources/performance-and-limits#limits). If you use larger buckets (monthly), you lose the ability to filter accurately. + + +You must commit to a single granularity. Daily buckets mean too many buckets for long ranges. Monthly buckets lose precision for recent data. + + +If that is a problem, for example when you want hourly precision for recent data but do not want hundreds of buckets when syncing a full month, see Workaround 3 below. + +### 3: Multiple Granularities + +Combine multiple granularities in a single bucket definition. This lets you use larger buckets (days) for older data and smaller buckets (hours, minutes) for recent data. + +```yaml +bucket_definitions: + issues_by_time: + parameters: SELECT value AS partition FROM json_each(request.parameters() ->> 'partitions') + data: + # By day (e.g., "2026-01-07") + - SELECT * FROM issues WHERE substring(updated_at, 1, 10) = bucket.partition + # By hour (e.g., "2026-01-07T14") + - SELECT * FROM issues WHERE substring(updated_at, 1, 13) = bucket.partition + # By 10 minutes (e.g., "2026-01-07T14:3") + - SELECT * FROM issues WHERE substring(updated_at, 1, 15) = bucket.partition +``` + +The client then mixes granularities as needed: + +```javascript +await db.connect(connector, { + params: { + partitions: [ + "2026-01-05", + "2026-01-06", + "2026-01-07T10", + "2026-01-07T11", + "2026-01-07T12:0", + "2026-01-07T12:1", + "2026-01-07T12:2" + ] + }, +}) +``` + +Each data query acts as a filter based on the length of the partition value: a day-format partition only matches the day query, an hour-format partition only matches the hour query, and so on. + +This syncs January 5–6 by day, the morning of January 7 by hour, and the last 30 minutes in 10-minute chunks, without creating hundreds of buckets. + +The trade-off is complexity. The client must decide which granularity to use for each time segment, and each row belongs to multiple buckets, which increases replication overhead. + + +When using multiple time granularities (e.g., monthly, daily, hourly), rows move between buckets as time passes. Since each granularity creates a different bucket ID, the client must re-download the row from the new bucket even if it already has the data. This re-download overhead can nullify the benefits of granular filtering. For this reason, in some cases it may be better to sync entire months, avoiding the re-sync overhead, even if you sync more data initially. + + +## Conclusion + +Time-based sync is a common need, but PowerSync doesn't support range operators or time-based functions on parameters directly. To recap the workarounds: + +- **Pre-defined time ranges:** Simplest option. Use when you have a fixed set of time ranges and don't mind schema changes. +- **Buckets per date:** More flexible. Use when you need arbitrary date ranges but can live with a single granularity. +- **Multiple granularities:** Most flexible. Use when you need precision for recent data without syncing hundreds of buckets. Be mindful of the re-sync overhead. diff --git a/sync/rules/types.mdx b/sync/rules/types.mdx new file mode 100644 index 00000000..dc722ed1 --- /dev/null +++ b/sync/rules/types.mdx @@ -0,0 +1,16 @@ +--- +title: "Types" +sidebarTitle: "Type Mapping" +description: "How Postgres, MongoDB, MySQL, SQL Server and Convex types map to PowerSync's SQLite-based sync column definitions." +noindex: true +--- + +{/* Wrapper page: the content is snippets/sync-shared/types.mdx, which also renders at sync/types.mdx in the Sync Streams section. Edit the snippet, not this file. The deprecation callout stays here, outside the snippet. */} + +import TypeMapping from '/snippets/sync-shared/types.mdx'; + + +Sync Rules are deprecated. This page applies to both Sync Streams and Sync Rules unless a section says otherwise. + + + diff --git a/sync/streams/bucket-count.mdx b/sync/streams/bucket-count.mdx index 42a563bc..a7ded805 100644 --- a/sync/streams/bucket-count.mdx +++ b/sync/streams/bucket-count.mdx @@ -318,10 +318,6 @@ This is why a checkpoint log can read `buckets: 7 | param_results: 6`. One globa The parameter limit can stop a sync while the bucket count still looks safe. A user can fail with far fewer than 1,000 buckets, because their parameter lookups returned more than 1,000 rows. Always check both numbers. - -In legacy [Sync Rules](/sync/rules/overview), these two limits were effectively one number, because each parameter-query result became one bucket. In Sync Streams they can diverge. - - ### Total Buckets vs Buckets Per User The 1,000 limit applies to each individual user, not to your whole instance. Your PowerSync Service can track millions of buckets in total, as long as each user syncs fewer than the limit. A large total bucket count is not a problem on its own. diff --git a/sync/streams/client-usage.mdx b/sync/streams/client-usage.mdx index 0765ba9c..bd347783 100644 --- a/sync/streams/client-usage.mdx +++ b/sync/streams/client-usage.mdx @@ -532,7 +532,7 @@ When different components subscribe to the same stream with the same parameters ## Connection Parameters -Connection parameters are a more advanced feature for values that apply to all streams in a session. They're the Sync Streams equivalent of [Client Parameters](/sync/rules/client-parameters) in legacy Sync Rules. +Connection parameters are a more advanced feature for values that apply to all streams in a session. For most use cases, **subscription parameters** (passed when subscribing) are more flexible and recommended. Use connection parameters only when you need a single global value across all streams, like an environment flag. diff --git a/sync/streams/examples.mdx b/sync/streams/examples.mdx index 172d3a2c..991e91c7 100644 --- a/sync/streams/examples.mdx +++ b/sync/streams/examples.mdx @@ -233,7 +233,7 @@ streams: Clients auto-subscribe to all three streams when they connect. Each query joins through `board_subscriptions` to find relevant data: posts in the user's boards, comments on those posts, and other users sharing those boards. -Unlike with legacy [Sync Rules](/sync/rules/many-to-many-join-tables), you don't need to denormalize your schema or maintain array columns to express these relationships. At scale, denormalizing the scope key onto child tables can still be the best way to control how many buckets each user syncs. See [Bucket Count and Limits](/sync/streams/bucket-count). +You don't need to denormalize your schema or maintain array columns to express these relationships. At scale, denormalizing the scope key onto child tables can still be the best way to control how many buckets each user syncs. See [Bucket Count and Limits](/sync/streams/bucket-count). ## Use Case Examples diff --git a/sync/streams/overview.mdx b/sync/streams/overview.mdx index 5fbe3ba2..4f3d808e 100644 --- a/sync/streams/overview.mdx +++ b/sync/streams/overview.mdx @@ -8,12 +8,6 @@ With Sync Streams, you write SQL-like queries to define streams of data, and you For example, you might define a stream that syncs only the current user's to-do items, another for shared projects they have access to, and another for reference data that everyone needs. Your app subscribes to these streams on demand, and only that data syncs to the device. Apps that need all relevant data available upfront can set `auto_subscribe: true` so streams sync automatically when clients connect. - -**Are you still using Sync Rules?** Sync Streams support everything Sync Rules do, plus more expressive queries (including JOIN support), on-demand syncing, and a simpler developer experience (e.g. React hooks that manage subscriptions automatically). - -You can migrate in a few clicks. Click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to generate a draft from your current config. See [Migrate to Sync Streams](/sync/rules/migrate-to-sync-streams) for details. - - ## How It Works Each PowerSync Service instance has a deployed Sync Streams configuration: a YAML file that defines the streams that exist. Each stream has a name and a SQL-like query that selects the tables and columns to sync, filters rows by static conditions or by parameters, and can rename or transform columns. The Service uses this configuration in two places: when it replicates data from your source database into buckets, and when it streams those buckets to clients. diff --git a/sync/streams/parameters.mdx b/sync/streams/parameters.mdx index fe3e90d4..aa88f765 100644 --- a/sync/streams/parameters.mdx +++ b/sync/streams/parameters.mdx @@ -54,7 +54,7 @@ streams: ## Connection Parameters -Specified "globally" at the connection level, before any streams are subscribed. These are the equivalent of [Client Parameters](/sync/rules/client-parameters) in Sync Rules. Use them when you need a value that applies across all streams for the session. +Specified "globally" at the connection level, before any streams are subscribed. Use them when you need a value that applies across all streams for the session. ```yaml streams: diff --git a/sync/streams/prioritized-sync.mdx b/sync/streams/prioritized-sync.mdx index 064c1dd5..6b9a9650 100644 --- a/sync/streams/prioritized-sync.mdx +++ b/sync/streams/prioritized-sync.mdx @@ -3,11 +3,13 @@ title: "Prioritized Sync" description: "Prioritize which tables sync first so users can start working immediately while remaining data continues loading in the background." --- +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/prioritized-sync.mdx. When you change shared behavior or fix an error here, check whether that page needs the same change. Do not mention Sync Rules on this page. */} + ## Overview PowerSync supports defining sync priorities, which allows you to control the sync order for different data. This is particularly useful when certain data should be available sooner than others. -In Sync Streams, priorities are assigned to streams and PowerSync manages the underlying buckets internally. (In legacy Sync Rules, priorities were assigned to buckets explicitly.) +Priorities are assigned to streams, and PowerSync manages the underlying buckets internally. ## Why Use Sync Priorities? @@ -19,7 +21,7 @@ While this guarantees consistency, it can lead to delays, especially for large d ## How It Works -Each bucket is assigned a priority value between 0 and 3, where: +Each stream is assigned a priority value between 0 and 3, where: - 0 is the highest priority and has special behavior (detailed below). - 3 is the default and lowest priority. @@ -27,57 +29,9 @@ Each bucket is assigned a priority value between 0 and 3, where: Higher-priority data syncs first, and lower-priority data syncs later. If you only use a single priority, there is no difference between priorities 1-3. The difference only comes in when you use multiple different priorities. - - -In Sync Streams, you assign priorities directly to streams. PowerSync manages buckets internally, so you don't need to think about bucket structure. Each stream with a given priority will have its data synced at that priority level. - -```yaml -streams: - lists: - auto_subscribe: true - query: SELECT * FROM lists WHERE owner_id = auth.user_id() - priority: 1 # Syncs first - - todos: - auto_subscribe: true - query: SELECT * FROM todos WHERE list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) - priority: 2 # Syncs after lists -``` - -Clients can also override the priority when subscribing: - -```js -// Override the stream's default priority for this subscription -const sub = await db.syncStream('todos', { list_id: 'abc' }).subscribe({ priority: 1 }); -``` - -When different components subscribe to the same stream with the same parameters but different priorities, PowerSync uses the highest priority for syncing. That higher priority is kept until the subscription ends (or its TTL expires). Subscriptions with different parameters are independent and do not conflict. - - -In Sync Rules, you assign priorities to bucket definitions. The priority determines when data in that bucket syncs relative to other buckets. - -```yaml -bucket_definitions: - user_lists: - priority: 1 # Syncs first - parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() - data: - - SELECT * FROM lists WHERE id = bucket.list_id - - user_todos: - priority: 2 # Syncs after lists - parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() - data: - - SELECT * FROM todos WHERE list_id = bucket.list_id -``` - - - ## Syntax and Configuration - - -In Sync Streams, set the `priority` option on the stream definition: +Set the `priority` option on the stream definition: ```yaml streams: @@ -91,38 +45,24 @@ streams: query: SELECT * FROM background_table WHERE user_id = auth.user_id() priority: 2 ``` - - -In Sync Rules, priorities can be defined using the `priority` YAML key on bucket definitions, or with the `_priority` attribute inside parameter queries: -```yaml -bucket_definitions: - # Using the `priority` YAML key - user_data: - priority: 1 - parameters: SELECT request.user_id() AS id WHERE ... - data: - # ... - - # Using the `_priority` attribute (useful for multiple parameter queries with different priorities) - project_data: - parameters: SELECT id AS project_id, 2 AS _priority FROM projects WHERE ... - data: - # ... +Clients can also override the priority when subscribing: + +```js +// Override the stream's default priority for this subscription +const sub = await db.syncStream('todos', { list_id: 'abc' }).subscribe({ priority: 1 }); ``` - - + +When different components subscribe to the same stream with the same parameters but different priorities, PowerSync uses the highest priority for syncing. That higher priority is kept until the subscription ends (or its TTL expires). Subscriptions with different parameters are independent and do not conflict. -Priorities must be static and cannot depend on row values within a parameter query. +Priorities are static values that you set in the stream definition or when subscribing. They cannot depend on row values. ## Example: Syncing Lists Before Todos Consider a scenario where you want to display lists immediately while loading todos in the background. This approach allows users to view and interact with lists right away without waiting for todos to sync. - - ```yaml config: edition: 3 @@ -142,38 +82,17 @@ streams: ``` The `lists` stream syncs first (priority 1), allowing users to see and interact with their lists immediately. The `todos` stream syncs afterward (priority 2), loading in the background. - - -```yaml -bucket_definitions: - user_lists: - priority: 1 # Syncs first - parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() - data: - - SELECT * FROM lists WHERE id = bucket.list_id - - user_todos: - priority: 2 # Syncs after lists - parameters: SELECT id AS list_id FROM lists WHERE user_id = request.user_id() - data: - - SELECT * FROM todos WHERE list_id = bucket.list_id -``` - -The `user_lists` bucket syncs first (priority 1), allowing users to see and interact with their lists immediately. The `user_todos` bucket syncs afterward (priority 2), loading in the background. - - - ## Behavioral Considerations -- **Interruption for Higher Priority Data**: Syncing lower-priority data _may_ be interrupted if new data for higher-priority streams/buckets arrives. +- **Interruption for Higher Priority Data**: Syncing lower-priority data _may_ be interrupted if new data for higher-priority streams arrives. - **Local Changes & Consistency**: If local writes fail due to validation or permission issues, they are only reverted after _all_ data has synced. - **Deleted Data**: Deleted data may only be removed after _all_ priorities have completed syncing. Future updates may improve this behavior. - **Data Ordering**: Lower-priority data will never appear before higher-priority data. ## Special Case: Priority 0 -Priority 0 buckets sync regardless of pending uploads. +Priority 0 streams sync regardless of pending uploads. For example, in a collaborative document editing app (e.g., using Yjs), each change is stored as a separate row. Since out-of-order updates don’t affect document integrity, Priority 0 can ensure immediate availability of updates. diff --git a/sync/streams/quickstart.mdx b/sync/streams/quickstart.mdx index c8ae8cd6..36a84d05 100644 --- a/sync/streams/quickstart.mdx +++ b/sync/streams/quickstart.mdx @@ -129,7 +129,6 @@ const sub = await db.syncStream('list_todos', { list_id: 'abc123' }).subscribe() Set `auto_subscribe: true` to sync data automatically when clients connect. This is useful for: - Reference data that all users need, or that is needed in many screens in the app. - User data that should always be available offline. -- Keeping the "sync everything upfront" behavior of legacy [Sync Rules](/sync/rules/overview) when migrating to Sync Streams. ```yaml config: diff --git a/sync/supported-sql.mdx b/sync/supported-sql.mdx index 9287c428..b246d733 100644 --- a/sync/supported-sql.mdx +++ b/sync/supported-sql.mdx @@ -1,11 +1,13 @@ --- title: "Supported SQL" -description: "Reference for SQL syntax, operators, built-in functions, and type casting supported in Sync Streams/Sync Rules queries." +description: "Reference for SQL syntax, operators, built-in functions, and type casting supported in Sync Streams queries." --- -This guide explains the SQL supported in [Sync Streams](/sync/streams/overview) and [Sync Rules (legacy)](/sync/rules/overview): what you can write, with examples and restrictions. +{/* Split page: the Sync Rules (Legacy) version of this page is sync/rules/supported-sql.mdx. When you change shared behavior or fix an error here, check whether that page needs the same change. Do not mention Sync Rules on this page. */} -For the exact syntax the compiler accepts — railroad diagrams and grammar-rule references — see the [Sync Streams](/sync/grammar/sync-streams/index) or [Sync Rules](/sync/grammar/sync-rules/index) grammar reference. +This guide explains the SQL supported in [Sync Streams](/sync/streams/overview): what you can write, with examples and restrictions. + +For the exact syntax the compiler accepts, with railroad diagrams and grammar-rule references, see the [Sync Streams grammar reference](/sync/grammar/sync-streams/index). Some fundamental restrictions on the usage of SQL expressions are: @@ -19,36 +21,22 @@ For the exact syntax the compiler accepts — railroad diagrams and grammar-rule ## Query Syntax -The supported SQL is based on a subset of the standard SQL syntax. Sync Streams support more SQL features than the legacy Sync Rules. - - - - - `SELECT` with column selection and [`WHERE` filtering](#filtering-where-clause) - - [Subqueries](/sync/streams/queries#using-subqueries) with `IN (SELECT ...)` and nested subqueries - - [`INNER JOIN`](#join-syntax) (selected columns must come from a single table) - - [Common Table Expressions (CTEs)](#cte-and-with-syntax) via the `with:` block - - Multiple queries per stream via `queries:` - - Table-valued functions such as `json_each()` for [expanding arrays](/sync/streams/parameters#expanding-json-arrays) - - `BETWEEN` and `CASE` expressions - - A limited set of [operators](#operators) and [functions](#functions) - - **Not supported**: aggregation, sorting, or set operations (`GROUP BY`, `ORDER BY`, `LIMIT`, `UNION`, etc.). See [Writing Queries](/sync/streams/queries) for details. - - - - Simple `SELECT` with column selection - - `WHERE` filtering on parameters (see [Filtering: WHERE Clause](#filtering-where-clause)) - - A limited set of [operators](#operators) and [functions](#functions) - - **Not supported**: subqueries, JOINs, CTEs, aggregation, sorting, or set operations (`GROUP BY`, `ORDER BY`, `LIMIT`, `UNION`, etc.). - - +The supported SQL is based on a subset of the standard SQL syntax: -## Filtering: WHERE Clause +- `SELECT` with column selection and [`WHERE` filtering](#filtering-where-clause) +- [Subqueries](/sync/streams/queries#using-subqueries) with `IN (SELECT ...)` and nested subqueries +- [`INNER JOIN`](#join-syntax) (selected columns must come from a single table) +- [Common Table Expressions (CTEs)](#cte-and-with-syntax) via the `with:` block +- Multiple queries per stream via `queries:` +- Table-valued functions such as `json_each()` for [expanding arrays](/sync/streams/parameters#expanding-json-arrays) +- `BETWEEN` and `CASE` expressions +- A limited set of [operators](#operators) and [functions](#functions) -Sync queries support a subset of SQL `WHERE` syntax. Allowed operators and combinations differ between Sync Streams and Sync Rules, and are more restrictive than standard SQL. +**Not supported**: aggregation, sorting, or set operations (`GROUP BY`, `ORDER BY`, `LIMIT`, `UNION`, etc.). See [Writing Queries](/sync/streams/queries) for details. - - +## Filtering: WHERE Clause + +Sync Streams queries support a subset of SQL `WHERE` syntax. Allowed operators and combinations are more restrictive than standard SQL. **`=` and `IS NULL`** — Compare a row column to a static value, a parameter, or another column: @@ -109,59 +97,6 @@ WHERE category NOT IN ROW('draft', 'hidden') -- WHERE id NOT IN subscription.parameter('excluded_ids') ``` - - - -**`=` and `IS NULL`** — Compare a row column to a static value or a bucket parameter: - -```sql --- Static value -WHERE status = 'active' -WHERE deleted_at IS NULL - --- Bucket parameter -WHERE owner_id = bucket.user_id -``` - -**`AND`** — Supported in both Parameter Queries and Data Queries. In Parameter Queries, each condition may match a different parameter. However, you cannot combine two `IN` expressions on parameters in the same `AND`; split them into separate Parameter Queries instead. - -```sql --- Supported: parameter condition + row-value condition -WHERE users.id = request.user_id() - AND users.is_admin = true - --- Not supported: two IN expressions on parameters in the same AND --- WHERE bucket.list_id IN lists.allowed_ids --- AND bucket.org_id IN lists.allowed_org_ids -``` - -**`OR`** — Supported when both sides of the `OR` reference the exact same set of parameters. If the two sides use different parameters, use separate parameter queries instead. - -```sql --- Supported: both sides reference the same parameter -WHERE lists.owner_id = request.user_id() - OR lists.shared_with = request.user_id() - --- Not supported: sides reference different parameters --- WHERE lists.owner_id = request.user_id() --- OR lists.org_id = bucket.org_id -``` - -**`NOT`** — Supported for simple row-value conditions. Not supported on parameter-matching expressions. - -```sql --- Supported -WHERE status != 'archived' -WHERE deleted_at IS NOT NULL -WHERE NOT users.is_admin = true - --- Not supported in parameter queries --- WHERE NOT users.id = request.user_id() -``` - - - - ## Operators Operators can be used in `WHERE` clauses and in `SELECT` expressions. When filtering on parameters (e.g. `auth.user_id()`, `subscription.parameter('id')`), some combinations are restricted — see [Filtering: WHERE Clause](#filtering-where-clause). @@ -183,15 +118,12 @@ Operators can be used in `WHERE` clauses and in `SELECT` expressions. When filte - `json ->> 'path'` — Returns the extracted value. - - **Sync Streams:** `left IN right` — `left` can be a row column and `right` a parameter array (e.g. `id IN subscription.parameter('ids')`), or `left` a parameter and `right` a row JSON array column. Also supports subqueries: `id IN (SELECT ...)`. - - **Sync Rules:** Returns true if `left` is in the `right` JSON array. In Data Queries, `left` must be a row column and `right` cannot be a bucket parameter. In Parameter Queries, either side may be a parameter. + - `left IN right` — `left` can be a row column and `right` a parameter array (e.g. `id IN subscription.parameter('ids')`), or `left` a parameter and `right` a row JSON array column. Also supports subqueries: `id IN (SELECT ...)`. - `x BETWEEN a AND b`, `x NOT BETWEEN a AND b` — True if `x` is in the inclusive range `[a, b]`. Usable in `WHERE` or as a `SELECT` expression. If any operand is `null`, the result is `null`. Example: `WHERE price BETWEEN 10 AND 100` - - Supported in Sync Streams only. Not available in Sync Rules. - ` && ` — True if the JSON array in `left` and the set `right` share at least one value. Use when the row stores an array (e.g. a `tagged_users` column). `left` must be a row column (JSON array); `right` must be a subquery or parameter array. @@ -199,8 +131,6 @@ Operators can be used in `WHERE` clauses and in `SELECT` expressions. When filte Example: `WHERE tagged_users && (SELECT id FROM org_members WHERE org_id = auth.parameter('org_id'))` Use `IN` when the row has a single value to check against a set; use `&&` when the row has an array and you want to match any element. - - Supported in Sync Streams only. Not available in Sync Rules. @@ -225,13 +155,11 @@ Most functions are from [SQLite built-in functions](https://www.sqlite.org/lang_ - **[typeof(data)](https://www.sqlite.org/lang_corefunc.html#typeof)** — Returns `text`, `integer`, `real`, `blob`, or `null`. - - **[json_each(data)](https://www.sqlite.org/json1.html#jeach)** — Expands a JSON array into rows. - - **Sync Streams:** Works with auth and connection parameters (e.g. `JOIN json_each(auth.parameter('ids')) AS t` or `WHERE id IN (SELECT value FROM json_each(auth.parameter('ids')))`). Can also be used with columns from joined tables in some cases (e.g. `SELECT * FROM lists WHERE id IN (SELECT lists.value FROM access_control a, json_each(a.allowed_lists) as lists WHERE a.user = auth.user_id())`). See [Expanding JSON arrays](/sync/streams/parameters#expanding-json-arrays). - - **Sync Rules:** Expands a JSON array or object from a request or token parameter into a set of parameter rows. Example: `SELECT value AS project_id FROM json_each(request.jwt() -> 'project_ids')`. + - **[json_each(data)](https://www.sqlite.org/json1.html#jeach)** — Expands a JSON array into rows. Works with auth and connection parameters (e.g. `JOIN json_each(auth.parameter('ids')) AS t` or `WHERE id IN (SELECT value FROM json_each(auth.parameter('ids')))`). Can also be used with columns from joined tables in some cases (e.g. `SELECT * FROM lists WHERE id IN (SELECT lists.value FROM access_control a, json_each(a.allowed_lists) as lists WHERE a.user = auth.user_id())`). See [Expanding JSON arrays](/sync/streams/parameters#expanding-json-arrays). - **[json_extract(data, path)](https://www.sqlite.org/json1.html#jex)** — Same as `->>` operator, but the path must start with `$.` - **[json_array_length(data)](https://www.sqlite.org/json1.html#jarraylen)** — Given a JSON array (as text), returns the length of the array. If data is null, returns null. If the value is not a JSON array, returns 0. - **[json_valid(data)](https://www.sqlite.org/json1.html#jvalid)** — Returns 1 if the data can be parsed as JSON, 0 otherwise. - - **json_keys(data)** — Returns the set of keys of a JSON object as a JSON array. Example: `SELECT * FROM items WHERE bucket.user_id IN json_keys(permissions_json)`. + - **json_keys(data)** — Returns the set of keys of a JSON object as a JSON array. Example: `SELECT id, json_keys(permissions_json) AS permission_keys FROM items`. - **[ifnull(x, y)](https://www.sqlite.org/lang_corefunc.html#ifnull)** — Returns x if non-null, otherwise returns y. @@ -251,7 +179,7 @@ Most functions are from [SQLite built-in functions](https://www.sqlite.org/lang_ - **table_name()** — Returns the name of the table the row was replicated from. This is the source table's name, not the alias or the output table name. - **table_suffix()** — Returns the part of the table name matched by the trailing `%` of a [wildcard table name](/sync/advanced/partitioned-tables). For example, with `FROM "todos_%" AS todos`, `todos.table_suffix()` returns `2024` for rows from the `todos_2024` table. On tables without a wildcard name, the result is always empty and the compiler reports a warning. - Supported in Sync Streams only, with PowerSync Service v1.24.0 or later. + Requires PowerSync Service v1.24.0 or later. - **[ST_AsGeoJSON(geometry)](/client-sdks/advanced/gis-data-postgis)** — Convert [PostGIS](/client-sdks/advanced/gis-data-postgis) (in Postgres) geometry from WKB to GeoJSON. Combine with JSON operators to extract specific fields. @@ -265,8 +193,6 @@ If you need an operator or function not listed, [contact us](/resources/contact- ## JOIN Syntax -Supported in Sync Streams only. Not available in Sync Rules. - Sync Streams support a subset of join syntax. The following rules define what is valid: - **Only inner joins:** Use `JOIN` or `INNER JOIN`. `LEFT`, `RIGHT`, and `OUTER` joins are not supported. @@ -295,8 +221,6 @@ For how to use JOINs in your stream queries (when to use them, patterns, and exa ## CTE and WITH Syntax -Supported in Sync Streams only. Not available in Sync Rules. - Common Table Expressions (CTEs) can be defined in a `with:` block **inside a stream** (stream-level, scoped to that stream) or at the **top level** of the Sync Config (global, shared across all streams). Each CTE is a name and a single `SELECT` query. The following rules apply: - **Stream-level CTEs take precedence over global CTEs.** If a stream defines a CTE with the same name as a global CTE, the stream-level definition is used within that stream. @@ -341,8 +265,6 @@ For how to use CTEs, see [Common Table Expressions (CTEs)](/sync/streams/ctes). ## CASE Expressions -Supported in Sync Streams only. Not available in Sync Rules. - `CASE` is allowed anywhere an expression is allowed — in `SELECT` columns or `WHERE` clauses. **Searched CASE** — Each `WHEN` is an independent boolean condition: diff --git a/sync/types.mdx b/sync/types.mdx index 0e69f3ce..eae7205d 100644 --- a/sync/types.mdx +++ b/sync/types.mdx @@ -4,165 +4,8 @@ sidebarTitle: "Type Mapping" description: "How Postgres, MongoDB, MySQL, SQL Server and Convex types map to PowerSync's SQLite-based sync column definitions." --- -import BinaryType from '/snippets/binary-type.mdx'; +{/* Wrapper page: the content is snippets/sync-shared/types.mdx, which also renders at sync/rules/types.mdx in the Sync Rules (Legacy) section. Edit the snippet, not this file. */} -The supported client-side SQLite types are: +import TypeMapping from '/snippets/sync-shared/types.mdx'; -1. `null` -2. `integer`: a 64-bit signed integer -3. `real`: a 64-bit floating point number -4. `text`: A UTF-8 text string -5. `blob`: Binary data - - -## Postgres Type Mapping - -Postgres types are mapped to SQLite types as follows: - -| Postgres Data Type | PowerSync / SQLite Column Type | Notes | -|--------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `text`, `varchar` | `text` | | -| `int2`, `int4`, `int8` | `integer` | | -| `numeric` / `decimal` | `text` | These types have arbitrary precision in Postgres, so can only be represented accurately as text in SQLite | -| `bool` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | -| `float4`, `float8` | `real` | | -| `enum` | `text` | | -| `uuid` | `text` | | -| `timestamptz` | `text` | Format: `YYYY-MM-DD hh:mm:ss.sssZ`. This is compatible with ISO8601 and SQLite's functions. Precision matches the precision used in Postgres. `-infinity` becomes `0000-01-01 00:00:00Z` and `infinity` becomes `9999-12-31 23:59:59Z`. | -| `timestamp` | `text` | Format: `YYYY-MM-DD hh:mm:ss.sss`. In most cases, timestamptz should be used instead. `-infinity` becomes `0000-01-01 00:00:00` and `infinity` becomes `9999-12-31 23:59:59`. | -| `date`, `time` | `text` | | -| `json`, `jsonb` | `text` | `json` and `jsonb` values are treated as `text` values in their serialized representation. [JSON functions and operators](/sync/supported-sql#operators) operate directly on these `text` values. | -| `interval` | `text` | | -| `macaddr` | `text` | | -| `inet` | `text` | | -| `bytea` | `blob` | Cannot sync directly to client — convert to hex or base64 first. See [Operators & Functions](/sync/supported-sql). | -| `geometry` (PostGIS) | `text` | Hex string of the binary data. Use the [ST functions](/sync/supported-sql#functions) to convert to other formats | -| Arrays | `text` | JSON array. | -| `DOMAIN` types | `text` / depends | Depending on [compatibility options](/sync/advanced/compatibility#custom_postgres_types), inner type or raw wire representation (legacy). | -| Custom types | `text` | Depending on [compatibility options](/sync/advanced/compatibility#custom_postgres_types), JSON object or raw wire representation (legacy). | -| (Multi-)ranges | `text` | Depending on [compatibility options](/sync/advanced/compatibility#custom_postgres_types), JSON object (array for multi-ranges) or raw wire representation (legacy). | - - - - -## Convex Type Mapping - - - The Convex replicator is currently released as an [experimental feature](/resources/feature-status). APIs and - behavior may change, and we can't yet guarantee continued support or long-term stability. - - -Convex values are mapped to SQLite types as follows: - -| Convex Type | TS/JS Type | PowerSync / SQLite Column Type | Notes | -| ----------- | ---------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `Id` | `string` | `text` | Convex document IDs are exposed as `_id` and can be synced as `text`. For synced client tables, use client-side ID mapping with a stable UUID column as `id` instead of relying on Convex-generated `_id` values. | -| `Null` | `null` | `null` | | -| `Int64` | `base-10 string` | `text` | Cast to `INTEGER` in Sync Streams when you want to sync the value as a SQLite integer. | -| `Float64` | `number` | `real` | | -| `Boolean` | `boolean` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | -| `String` | `string` | `text` | | -| `Bytes` | `base64 string` | `text` | Decode from base64 in your app if you need binary data on the client. | -| `Array` | `Array` | `text` | Converted to a JSON string. | -| `Object` | `Object` | `text` | Converted to a JSON string. | -| `Record` | `Record` | `text` | Converted to a JSON string. | - -- Convex documents are converted to a flat list of columns, one column per top-level field. -- Nested objects and arrays are converted to JSON, and [JSON functions and operators](/sync/supported-sql#operators) can be used to query them in Sync Streams or on the client-side SQLite database. -- Cast Convex `Int64` fields to `INTEGER` in Sync Streams when you want SQLite integer values on the client, for example `CAST(an_int64_column AS INTEGER) AS an_int64_column`. - - -## MongoDB Type Mapping - -MongoDB types are mapped to SQLite types as follows: - -| BSON Type | PowerSync / SQLite Column Type | Notes | -|--------------------|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------| -| `String` | `text` | | -| `Int`, `Long` | `integer` | | -| `Double` | `real` | | -| `Decimal128` | `text` | | -| `Object` | `text` | Converted to a JSON string | -| `Array` | `text` | Converted to a JSON string | -| `ObjectId` | `text` | Lower-case hex string | -| `UUID` | `text` | Lower-case hex string | -| `Boolean` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | -| `Date` | `text` | Format: `YYYY-MM-DD hh:mm:ss.sssZ` | -| `Null` | `null` | | -| `Binary` | `blob` | Cannot sync directly to client — convert to hex or base64 first. See [Operators & Functions](/sync/supported-sql). | -| Regular Expression | `text` | JSON text in the format `{"pattern":"...","options":"..."}` | -| `Timestamp` | `integer` | Converted to a 64-bit integer | -| `Undefined` | `null` | | -| `DBPointer` | `text` | JSON text in the format `{"collection":"...","oid":"...","db":"...","fields":...}` | -| `JavaScript` | `text` | JSON text in the format `{"code": "...", "scope": ...}` | -| `Symbol` | `text` | | -| `MinKey`, `MaxKey` | `null` | | - -* Data is converted to a flat list of columns, one column per top-level field in the MongoDB document. -* Special BSON types are converted to plain SQLite alternatives. For example, `ObjectId`, `Date`, `UUID` are all converted to a plain `TEXT` column. -* Nested objects and arrays are converted to JSON, and [JSON functions and operators](/sync/supported-sql#operators) can be used to query them (in the Sync Streams / Sync Rules and/or on the client-side SQLite statements). -* Binary data nested in objects or arrays is not supported. - - - - -## MySQL Type Mapping - -MySQL support is currently in a [Beta release](/resources/feature-status). - -MySQL types are mapped to SQLite types as follows: - -| MySQL Data Type | PowerSync / SQLite Column Type | Notes | -|----------------------------------------------------|--------------------------------|-----------------------------------------------------------------------------------| -| `tinyint`, `smallint`, `mediumint`, `bigint`, `integer`, `int` | `integer` | | -| `numeric`, `decimal` | `text` | | -| `bool`, `boolean` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | -| `float`, `double`, `real` | `real` | | -| `enum` | `text` | | -| `set` | `text` | Converted to JSON array | -| `char`, `varchar` | `text` | | -| `tinytext`, `text`, `mediumtext`, `longtext` | `text` | | -| `timestamp` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | -| `date` | `text` | Format: `YYYY-MM-DD` | -| `time`, `datetime` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | -| `year` | `text` | | -| `json` | `text` | There is no dedicated JSON type in SQLite — JSON functions operate directly on text values. | -| `bit` | `blob` | * See note below regarding syncing binary types | -| `binary`, `varbinary` | `blob` | | -| `image` | `blob` | | -| `geometry`, `geometrycollection` | `blob` | | -| `point`, `multipoint` | `blob` | | -| `linestring`, `multilinestring` | `blob` | | -| `polygon`, `multipolygon` | `blob` | | - - - - -## SQL Server Type Mapping - -SQL Server support is currently in a [Beta release](/resources/feature-status). - -SQL Server types are mapped to SQLite types as follows: - -| SQL Server Data Type | PowerSync / SQLite Column Type | Notes | -|----------------------------------------------------|--------------------------------|--------------------------------------------------------| -| `tinyint`, `smallint`, `int`, `bigint` | `integer` | | -| `numeric`, `decimal` | `text` | Numeric string | -| `float`, `real` | `real` | | -| `bit` | `integer` | | -| `money`, `smallmoney` | `text` | Numeric string | -| `xml` | `text` | | -| `char`, `nchar`, `ntext` | `text` | | -| `varchar`, `nvarchar`, `text` | `text` | | -| `uniqueidentifier` | `text` | | -| `timestamp` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | -| `date` | `text` | Format: `YYYY-MM-DD` | -| `time` | `text` | Format: `HH:mm:ss.sss` | -| `datetime`, `datetime2`, `smalldatetime`, `datetimeoffset` | `text` | ISO 8601 format: `YYYY-MM-DDTHH:mm:ss.sssZ` | -| `json` | `text` | Only exists for Azure SQL Database and SQL Server 2025 | -| `geometry`, `geography` | `text` | `text` of JSON object describing the spatial data type | -| `binary`, `varbinary`, `image` | `blob` | * See note below regarding binary types | -| `rowversion`, `timestamp` | `blob` | * See note below regarding binary types | -| User Defined Types: `hiearchyid` | `blob` | * See note below regarding binary types | - - +