2 schema

Cargo.toml

@@ -10,8 +10,9 @@ repository = "https://github.com/luxas/cedar-sql"
 [dependencies]
 cedar-policy = { version = "4.12.0", features = ["tpe"] }
 cedar-policy-core = { version = "4.12.0", features = ["tpe"] }
-indexmap = "2"
+indexmap = { version = "2", features = ["serde"] }
 postgres = { version = "0.19", features = ["with-serde_json-1"] }
+serde = { version = "1", features = ["derive"] }
 serde_json = "1"
 smol_str = "0.3"
 thiserror = "2"

README.md

@@ -744,6 +744,8 @@ Deviations from the text above that were decided during implementation:
 - Sets and records are stored as canonical JSONB (sets deduplicated and sorted, records and entity references
   wrapped) rather than `[]<type>` arrays, so that JSONB equality is Cedar equality for every nested shape.
 - The hierarchy table may hold the transitive closure instead of the direct edges (the recursive CTE tolerates both).
+- The `@sql_entity_id_column` target may be the table's (sole) primary key instead of being declared unique.
+- Foreign keys are `DEFERRABLE INITIALLY DEFERRED`, so rows load in any order within a transaction.
 
 The crate expects a checkout of [`cedar-woodpecker`] (a fork of `cedar`) next to this repository, as `cedar-spec`
 does. Tests need a Postgres: set `CEDAR_SQL_PG_URL`, or leave it unset to have an embedded Postgres downloaded and

docs/followups/2-schema.md

@@ -0,0 +1,36 @@
+# Follow-ups for branch 2 — schema (review fixes)
+
+## PR description
+
+Idea 1 of the README: a Cedar schema with its `@sql_*` annotations becomes a `DatabaseConfiguration`, renders
+as Postgres DDL, and Cedar entities load as rows (`docs/plans/2-schema.md`). Tests run the README's schemas'
+DDL on Postgres and read loaded rows back.
+
+## What this branch contains
+
+`SQLIdentifier` and literal quoting; the configuration model and its builder from the validator schema and
+the annotations; DDL with deferred foreign keys; the loader with the canonical JSON encoding of compound values.
+
+## Review findings (fixed here)
+
+- Constraint-name shortening looped forever on long names with multibyte characters; now `ident::shortened`
+  cuts at a char boundary and appends a hash (shared with the query compiler's generated names).
+- With `@sql_primary_key` naming another column, the `__entity_id` column was not unique, so foreign keys to it
+  failed; it is now `UNIQUE` whenever it is not the primary key.
+- The loader stored entity references of the wrong type, and strings in reference columns, silently; `literal`
+  now checks the referenced entity type (`ForeignKey::entity_type`) for attributes and tag values.
+- Attribute annotations behind a qualified or cross-namespace common type name were dropped; `resolve_record`
+  resolves `Ns::Type` in the named namespace.
+- `@sql_entity_id_column` naming an undefined column now adds a `TEXT NOT NULL UNIQUE` column, as the README says.
+- Conflicting `@sql_entity_id_column`/`@sql_primary_key` and `@sql_custom_config` values are errors; record keys
+  are NUL-checked; generated interface columns are `NOT NULL`.
+
+## Divergences from the README
+
+Recorded in the README's "Status and phases": canonical JSONB for sets and records, the closure in the
+hierarchy table, the primary key accepted as the entity id column, deferred foreign keys.
+
+## Suggested follow-ups
+
+- Arrays as a fast path for flat scalar sets (Plan 8 or later).
+- A `CHECK`-style validation query for the canonical form of loaded JSON.

docs/plans/2-schema.md

@@ -0,0 +1,61 @@
+# Plan 2 — schema: the database configuration, DDL and the entity loader
+
+## Goal
+
+Idea 1 of the README end to end: a Cedar schema, with its `@sql_*` annotations, becomes a
+`DatabaseConfiguration` (tables, columns, primary keys, tags tables, the hierarchy table and the
+`__entity_id`/`__entity_type` interface columns), the configuration renders as Postgres DDL, and Cedar
+entities load as rows of those tables — so that later phases have data to query.
+
+## Design
+
+- **Identifiers** (`src/ident.rs`): `SQLIdentifier` (non-empty, at most 63 bytes, no NUL; `Display` is the
+  double-quoted form) and `quoted_literal` (single quotes doubled; NUL rejected, since Postgres `text` cannot
+  hold it).
+- **Model** (`src/config.rs`): `SQLType {Text, BigInt, Bool, Jsonb, Set, Custom}`, `ColumnConfiguration`
+  (type, nullable, unique, custom attributes, foreign key, generated expression), `TableConfiguration`
+  (entity type, primary keys, columns in definition order, the tags table, the entity id column, the
+  attribute-to-column map), `DatabaseConfiguration` (tables in entity-type order, the hierarchy table and the
+  interface column names, `emit_foreign_keys`, `hierarchy_closed`). `SQLType::for_cedar_type` maps `Bool`,
+  `Long`, `String`, a single entity type (a `Text` id with a foreign key) and, per the decision recorded in the
+  README, `Set` and `Record` to `Jsonb`; extension, union, unspecified and never types are `Unsupported`, as are
+  entity types with additional attributes.
+- **The builder**: entity types sorted by name; the table is `@sql_table` or the fully-qualified type name;
+  one column per attribute (`@sql_column`, `@sql_unique`, `@sql_custom_attrs`), nullable iff optional; the
+  custom columns, primary keys and entity id column of `@sql_custom_config` (JSON, `entity_type` must be
+  `null`); the tags table is `@sql_tags_table` or `<table>_tags`. Conflicts are errors naming the annotation
+  that resolves them: two attributes on one column, a custom column on an attribute column, two entity types on
+  one table, a tags table on an entity table. The interface columns and the hierarchy table take the default
+  name unless any table's column (or any table) uses it, then the first free numeric suffix from 2 — the
+  README's `__entity_id2`/`cedar_entity_hierarchy2` rule. `@sql_entity_id_column` must name a non-nullable
+  text column that is unique or the primary key; the interface id column is then generated from it, otherwise
+  it is the physical primary key. The type column is always generated from the type name.
+- **DDL** (`src/ddl.rs`, `src/dialect.rs`): `create_tables` renders every entity table, tags table
+  (`entity_id, tag, value`, key `(entity_id, tag)`) and the hierarchy table, then the foreign keys as
+  `ALTER TABLE … DEFERRABLE INITIALLY DEFERRED` so table order and reference cycles do not matter and rows load
+  in any order; constraint names over 63 bytes are shortened with a hash. `drop_tables` is the inverse. The
+  `Dialect` trait (Postgres only for now) owns the type names, the generated-column clause and the literal forms.
+- **Loader** (`src/load.rs`): `entities_to_sql` renders one `INSERT` per entity, per ancestor (the hierarchy
+  table gets the transitive closure Cedar entities already carry) and per tag; action entities are returned
+  separately for partial evaluation. `canonical_json` is the JSON encoding of compound values: sets as arrays
+  (deduplicated by Cedar's own set semantics), records as `{"r": {…}}`, entity references as
+  `{"e": {"t": type, "i": id}}`; equality of these is defined order-insensitively by later phases, so no
+  element ordering is relied on. Undeclared attributes, missing required attributes, tags on tagless types,
+  unknown types and NUL characters are errors; residual (unknown) attribute values are `Unsupported`.
+
+## Files
+
+`Cargo.toml`, `src/{ident,config,annotations,ddl,dialect,load,error}.rs`, `tests/{config,ddl_golden,load_pg}.rs`,
+`tests/schemas/*.cedarschema`, `tests/golden/*.sql`, `tests/entities/kitchen_sink.json`, `docs/plans/2-schema.md`.
+
+## Verification
+
+`cargo fmt --all --check`, `cargo clippy --all-targets --all-features -- -D warnings`,
+`cargo test --all-features` with `CEDAR_SQL_PG_URL` set: the README's two schemas and a kitchen-sink schema
+against golden DDL (`UPDATE_GOLDEN=1` regenerates) that also executes on Postgres, the configuration rules and
+error cases, and a load round trip that reads the rows, hierarchy and tags back and checks the deferred
+foreign keys with `SET CONSTRAINTS ALL IMMEDIATE`.
+
+## History
+
+New; the second `cedar-sql` branch.

src/annotations.rs

@@ -1,7 +1,228 @@
 //! Reading the `@sql_*` schema annotations.
 //!
-//! Plan 2 adds the entity-type annotations (`@sql_table`, `@sql_tags_table`,
-//! `@sql_primary_key`, `@sql_custom_config`, `@sql_entity_id_column`) and the
-//! attribute annotations (`@sql_column`, `@sql_unique`, `@sql_custom_attrs`),
-//! read from the schema fragment since annotations do not survive into the
-//! validator schema.
+//! Annotations live on the schema fragment and do not survive into the
+//! validator schema, so the schema text is parsed again as a fragment (the
+//! pattern of `cedar-policy-symcc`'s `@semantics` collector). Entity types
+//! carry `@sql_table`, `@sql_tags_table`, `@sql_primary_key`,
+//! `@sql_entity_id_column` and `@sql_custom_config`; attributes carry
+//! `@sql_column`, `@sql_unique` and `@sql_custom_attrs`. Attribute annotations
+//! are read from the entity type's record shape, or from the common type the
+//! shape names.
+
+use std::collections::HashMap;
+use std::str::FromStr;
+
+use cedar_policy_core::ast::EntityType;
+use cedar_policy_core::est::Annotations;
+use cedar_policy_core::extensions::Extensions;
+use cedar_policy_core::validator::RawName;
+use cedar_policy_core::validator::json_schema::{
+    EntityTypeKind, Fragment, NamespaceDefinition, RecordType, Type, TypeVariant,
+};
+use smol_str::SmolStr;
+
+use crate::config::CustomConfig;
+use crate::ident::SQLIdentifier;
+use crate::{Error, Result};
+
+/// `@sql_table("users")`: the table name.
+pub const TABLE: &str = "sql_table";
+/// `@sql_tags_table("users_tags")`: the tags table name.
+pub const TAGS_TABLE: &str = "sql_tags_table";
+/// `@sql_primary_key("id")`: the primary key column.
+pub const PRIMARY_KEY: &str = "sql_primary_key";
+/// `@sql_entity_id_column("id")`: the column holding the entity id.
+pub const ENTITY_ID_COLUMN: &str = "sql_entity_id_column";
+/// `@sql_custom_config("{ … }")`: custom columns and table modifiers as JSON.
+pub const CUSTOM_CONFIG: &str = "sql_custom_config";
+/// `@sql_column("first_name")`: the column name of an attribute.
+pub const COLUMN: &str = "sql_column";
+/// `@sql_unique("true")`: whether the attribute's column is unique.
+pub const UNIQUE: &str = "sql_unique";
+/// `@sql_custom_attrs("DEFAULT TRUE")`: raw SQL appended to the column.
+pub const CUSTOM_ATTRS: &str = "sql_custom_attrs";
+
+/// The `@sql_*` annotations of a schema.
+#[derive(Clone, Debug, Default, PartialEq)]
+pub struct SqlAnnotations {
+    /// By fully-qualified entity type.
+    pub entity_types: HashMap<EntityType, EntityTypeAnnotations>,
+}
+
+/// The annotations of one entity type and its attributes.
+#[derive(Clone, Debug, Default, PartialEq)]
+pub struct EntityTypeAnnotations {
+    /// `@sql_table`.
+    pub table: Option<SQLIdentifier>,
+    /// `@sql_tags_table`.
+    pub tags_table: Option<SQLIdentifier>,
+    /// `@sql_primary_key`.
+    pub primary_key: Option<SQLIdentifier>,
+    /// `@sql_entity_id_column`.
+    pub entity_id_column: Option<SQLIdentifier>,
+    /// `@sql_custom_config`.
+    pub custom_config: Option<CustomConfig>,
+    /// The attributes' annotations, by attribute name.
+    pub attributes: HashMap<SmolStr, AttributeAnnotations>,
+}
+
+/// The annotations of one attribute.
+#[derive(Clone, Debug, Default, PartialEq)]
+pub struct AttributeAnnotations {
+    /// `@sql_column`.
+    pub column: Option<SQLIdentifier>,
+    /// `@sql_unique`.
+    pub unique: bool,
+    /// `@sql_custom_attrs`.
+    pub custom_attrs: Option<String>,
+}
+
+/// Collects the annotations of a schema in the Cedar schema syntax.
+pub fn from_cedarschema_str(src: &str) -> Result<SqlAnnotations> {
+    let (fragment, _warnings) =
+        Fragment::<RawName>::from_cedarschema_str(src, Extensions::all_available())
+            .map_err(|e| Error::Schema(e.to_string()))?;
+    collect(&fragment)
+}
+
+/// Collects the annotations of a schema in the JSON syntax.
+pub fn from_json_str(src: &str) -> Result<SqlAnnotations> {
+    let fragment =
+        Fragment::<RawName>::from_json_str(src).map_err(|e| Error::Schema(e.to_string()))?;
+    collect(&fragment)
+}
+
+/// Collects the annotations of a parsed fragment.
+pub fn collect(fragment: &Fragment<RawName>) -> Result<SqlAnnotations> {
+    let mut out = SqlAnnotations::default();
+    for (namespace, def) in &fragment.0 {
+        for (id, entity_type) in &def.entity_types {
+            let name = match namespace {
+                Some(ns) => format!("{ns}::{id}"),
+                None => id.to_string(),
+            };
+            let ety = EntityType::from_str(&name).map_err(|e| {
+                Error::Schema(format!("the entity type name {name} does not parse: {e}"))
+            })?;
+            let on = format!("entity type {ety}");
+            let a = &entity_type.annotations;
+            let mut ann = EntityTypeAnnotations {
+                table: identifier(a, TABLE, &on)?,
+                tags_table: identifier(a, TAGS_TABLE, &on)?,
+                primary_key: identifier(a, PRIMARY_KEY, &on)?,
+                entity_id_column: identifier(a, ENTITY_ID_COLUMN, &on)?,
+                custom_config: None,
+                attributes: HashMap::new(),
+            };
+            if let Some(json) = value(a, CUSTOM_CONFIG) {
+                ann.custom_config =
+                    Some(serde_json::from_str(json).map_err(|e| Error::Annotation {
+                        annotation: CUSTOM_CONFIG,
+                        on: on.clone(),
+                        message: e.to_string(),
+                    })?);
+            }
+            if let EntityTypeKind::Standard(standard) = &entity_type.kind
+                && let Some(record) = resolve_record(&standard.shape.0, def, fragment)
+            {
+                for (attr, attr_ty) in &record.attributes {
+                    let on = format!("attribute {attr} of {ety}");
+                    let a = &attr_ty.annotations;
+                    let attr_ann = AttributeAnnotations {
+                        column: identifier(a, COLUMN, &on)?,
+                        unique: boolean(a, UNIQUE, &on)?,
+                        custom_attrs: value(a, CUSTOM_ATTRS).map(str::to_owned),
+                    };
+                    if attr_ann != AttributeAnnotations::default() {
+                        ann.attributes.insert(attr.clone(), attr_ann);
+                    }
+                }
+            }
+            if ann != EntityTypeAnnotations::default() {
+                out.entity_types.insert(ety, ann);
+            }
+        }
+    }
+    Ok(out)
+}
+
+/// The record type an entity shape denotes: the record itself, or the record
+/// behind a common type name. An unqualified name is looked up in the
+/// declaring namespace, then in the empty one; a qualified name in the
+/// namespace it names.
+fn resolve_record<'f>(
+    ty: &'f Type<RawName>,
+    def: &'f NamespaceDefinition<RawName>,
+    fragment: &'f Fragment<RawName>,
+) -> Option<&'f RecordType<RawName>> {
+    match ty {
+        Type::Type {
+            ty: TypeVariant::Record(record),
+            ..
+        } => Some(record),
+        Type::CommonTypeRef { type_name, .. }
+        | Type::Type {
+            ty: TypeVariant::EntityOrCommon { type_name },
+            ..
+        } => {
+            let full = type_name.to_string();
+            let (namespaces, id): (Vec<Option<&NamespaceDefinition<RawName>>>, &str) =
+                match full.rsplit_once("::") {
+                    Some((ns, id)) => {
+                        let named = fragment
+                            .0
+                            .iter()
+                            .find(|(k, _)| k.as_ref().is_some_and(|k| k.to_string() == ns))
+                            .map(|(_, d)| d);
+                        (vec![named], id)
+                    }
+                    None => (vec![Some(def), fragment.0.get(&None)], full.as_str()),
+                };
+            namespaces
+                .into_iter()
+                .flatten()
+                .flat_map(|d| d.common_types.iter())
+                .find(|(k, _)| k.to_string() == id)
+                .and_then(|(_, common)| resolve_record(&common.ty, def, fragment))
+        }
+        Type::Type { .. } => None,
+    }
+}
+
+/// The annotation's value; a bare `@key` (no value) counts as the empty string.
+fn value<'a>(annotations: &'a Annotations, key: &str) -> Option<&'a str> {
+    annotations
+        .0
+        .iter()
+        .find(|(id, _)| id.as_ref() == key)
+        .map(|(_, a)| a.as_ref().map_or("", |a| a.val.as_str()))
+}
+
+fn identifier(
+    annotations: &Annotations,
+    key: &'static str,
+    on: &str,
+) -> Result<Option<SQLIdentifier>> {
+    value(annotations, key)
+        .map(|v| {
+            SQLIdentifier::new(v).map_err(|e| Error::Annotation {
+                annotation: key,
+                on: on.to_owned(),
+                message: e.to_string(),
+            })
+        })
+        .transpose()
+}
+
+fn boolean(annotations: &Annotations, key: &'static str, on: &str) -> Result<bool> {
+    match value(annotations, key) {
+        None => Ok(false),
+        Some("true") => Ok(true),
+        Some("false") => Ok(false),
+        Some(other) => Err(Error::Annotation {
+            annotation: key,
+            on: on.to_owned(),
+            message: format!("expected \"true\" or \"false\", got {other:?}"),
+        }),
+    }
+}