Repository navigation
Conversation
…s it references A mapping document names an ontology and a semantic model by iri, but validate.py checked only its own shape, so a mapping whose references name the wrong document, whose concepts are not in the ontology, or whose expressions name fields the model does not declare still passed. Read each iri as a URI reference and follow its path from the mapping file's own directory, as ontology.md says. Check that the document found is named as the reference says, that every concept the mapping names is declared there or built in, and that every dataset.field an expression names is a declared field of the model. A dataset that declares no fields list is not checked, since the core spec does not require one. A reference with no iri, with a scheme, or with a relative path and no location to resolve against is left to a catalog and reported as a warning, so documents that only a catalog can resolve still pass. Generated-by: Claude Code
…ncept A referent mapping names a relationship of the concept it maps, and a link mapping names one of the concept mapping's concept, but the cross-checks read only the concept and expression of each node, so a relationship the ontology does not declare passed. Resolve each relationship on its concept or on a concept it extends, the way validate_ontology resolves identify_by. Nested referent mappings resolve on the concept that plays the last role of the enclosing relationship, and a relationship with no such role cannot carry them. A link mapping's children map the objects in the last position of their relationship's links unless their object mapping names a concept, and a link mapping's level has to match the arity of the relationship it names, as ontology.md describes. A name written as Concept.name names the concept being mapped or one it extends, as in the spec examples. Generated-by: Claude Code
|
Pushed a second commit with the check JB suggested on the list: every relationship a referent or link mapping names has to resolve on its concept, directly or through extends, the way validate_ontology already resolves identify_by. A nested referent mapping resolves on the concept that plays the last role of the enclosing relationship, and a relationship with no role at all cannot carry one (a relationship with more than one role still resolves on its last). A link mapping's child maps the last role of its own relationship unless its object mapping names a concept, and the level of a link mapping has to match the arity of the relationship it names. The flights.mapping.yaml still passes. With |
| r"|/\*.*?\*/", # block comment | ||
| re.DOTALL, | ||
| ) | ||
| _FIELD_REFERENCE = re.compile(r"(?<![\w.])([A-Za-z_]\w*)\.([A-Za-z_]\w*)(?![\w.]|\s*\()") |
There was a problem hiding this comment.
This treats any name.name outside a quote or comment as a dataset.field reference, so ordinary SQL fails validation even when every field exists:
CAST(orders.order_id AS pg_catalog.int4)andorders.order_id::pg_catalog.int4report "declares no dataset 'pg_catalog'"transform(orders.ids, x -> x.y)reports "declares no dataset 'x'"orders.payload:customer.id(a Snowflake path) reports "declares no dataset 'customer'"'it\'s a.b' || orders.order_id(a backslash escape) reports "declares no dataset 'a'"
sqlglot is already a dependency of this script, so could this use the parser instead of a regex?
sqlglot.parse_one(expression).find_all(exp.Column) returns only orders.order_id / orders.ids for the cast and lambda cases. It gives each column's table and name along with their quoting, so quoted identifiers can be resolved too (they are skipped today).
Mapping expressions declare no dialect, so an expression that doesn't parse with the default dialect (the Snowflake path and backslash cases don't) could skip this check with a ValidationWarning rather than guess. The same goes for when sqlglot is not installed, as validate_sql already does.
| expression = node.get("expression") | ||
| if fields_by_dataset is not None and isinstance(expression, str): | ||
| for dataset, field in field_references(expression): | ||
| if dataset not in fields_by_dataset: |
There was a problem hiding this comment.
core-spec/expression_language.md says regular (unquoted) identifiers are case-insensitive, and that matching is done on normalized identifiers: unquoted ones upper-cased, quoted ones taken as written. Both the dataset and the field comparison here are exact, so ORDERS.ORDER_ID against dataset orders / field order_id is reported as an error and the file fails validation, even though it names the same field.
Could both sides be normalized before comparing? That means treating the model's name values as regular identifiers, and upper-casing the expression's parts unless they are quoted. Following the spec's comparison table, "ORDERS".order_id would then match and "orders".order_id would not.
| "[Reference] concept_mappings -> 0 -> object_mappings -> 1: expression names " | ||
| "'orders.order_nr' but dataset 'orders' declares no field 'order_nr'", | ||
| "[Reference] concept_mappings -> 0 -> object_mappings -> 2: expression names " | ||
| "'ORDERS.order_id' but the semantic model declares no dataset 'ORDERS'", |
There was a problem hiding this comment.
This pins down the case-sentsitive behavior. Per expression_language.md, ORDERS.order_id names dataset orders, so I would expect no error for it.
Could it move to the expressions that pass, maybe with a case like ORDERS.ORDER_NR that is still reported as a missing field?
| if ontology is not None: | ||
| if isinstance(ontology.get("ontology"), list): | ||
| components = [c for c in ontology["ontology"] if isinstance(c, dict)] | ||
| concepts = {c.get("concept") for c in components} | BUILT_IN_CONCEPTS |
There was a problem hiding this comment.
The referenced ontology is not schema-checked before it is used, so values of any type end up in this set, and also in the fields_by_dataset keys and sets below. A referenced ontology with concept: [Order], or a modal with a dataset name: [orders] or a field name: {a: 1}, raises TypeError: unhashable type. validate_file then reports an internal error with a traceback instead of a diagnostic about the referenced file.
Rather than adding isinstance guard here, could each referenced document go through validate_schema right after resolve_document_reference loads it? That would be ontology/ontology.json and core-spec/ossie-schema.json, using the existing local registry. If a document fails, its half of the checks would be skipped, with an error that names the referenced file. Helpers like as_list already assume "shapes that already failed the schema", and that is not true for these documents.
For example, roles given as a mapping is read as zero roles today and produces a misleading arity error.
| f"[Reference] Warning: {label} '{name}' has no iri, so only a catalog can " | ||
| f"resolve it: {skipped}" | ||
| )] | ||
| parts = urlsplit(iri) |
There was a problem hiding this comment.
urlsplit raises ValueError for a malformed bracketed host (http://[oops/o.yaml or //[::1/o.yaml gives "Invalid IPv6 URL"), and path.read_text below raises ValueError for an embedded NUL.
Neither is caught, so a typo in an iri shows up as an internal validator error instead of a [Reference] diagnostic. Something like:
try:
parts = urlsplit(iri)
except ValueError as e:
return None, [f"[Reference] {label} iri '{iri}' is not a valid URI reference: {e}"]plus a separate except ValueError as e: next to the OSError and around read_text. It can't share that clause, because ValueError has no strerror.
2 participants
Summary
A mapping document names its ontology and semantic model through
ontology_refandsemantic_model_ref, butvalidate.py --schema ontology/mapping.jsonchecked only the mapping's own shape. On current main, a copy ofexamples/flights.mapping.yamlwithsemantic_model_ref.namechanged toFlights model, the first concept changed toRunwyandRUNWAY.designatorchanged toRUNWAY.designaterstill reportsValidation PASSED.The validator now follows each relative
irifrom the mapping file's own directory, as ontology.md specifies, and checks that the document found carries thenamethe reference says, that every concept the mapping names is declared in that ontology or is a built-in concept, and that everydataset.fieldan expression names is a declared field of that semantic model. The copy above now fails with a[Reference]error for each edit, two forRUNWAY.designatersince the example writes it twice, and the unmodified example passes with nothing reported.A reference the validator does not follow is left to a catalog and reported as a warning, not an error: no
iri, aniriwith a scheme, or a relative one on a document validated without a location to resolve against. Theiriis read as a URI reference, so a percent-encoded path, a query or a fragment resolve as expected. A path that does not read as a YAML document is an error. Field references are the two-part identifiers in the expression text outside string literals, quoted identifiers and comments, compared exactly as the core reference checks do. A dataset that declares nofieldslist is not checked, since the core spec does not require one.This is the validate.py follow-up from the dev@ thread "Decouple Ontologies, Semantic Models, and Their Mappings Through External References", which Shao handed over on 2026-10-15. Concepts reuse
BUILT_IN_CONCEPTSfrom #489. The key coverage warning raised on that thread is not included, for the reason Shao gave there.Verification
validation/tests/: 227 passed, 26 of them new, also in random order. The validation workflow run locally against current main (9531429).examples/flights.mapping.yamlpasses with no warnings. The three-edit copy above reports four errors, as described. The other examples are unaffected.validate.pyor its tests: no conflicts, and no test changes outcome except the one feat(core-spec): support dataset-scoped metrics #591 already has on its own.Related Issues
dev@ thread: https://lists.apache.org/thread/9op4mr1qfb6o0c229shdbnzp44jp4wxd
Checklist
Tests
pytest/ CI green)Compliance