Feat/custom extensions object data

[MikeCarlo]

Summary

Changes custom_extensions[].data from a JSON-encoded string to a structured object with vendor-defined keys, implementing Proposal A from discussion #481.

This is an intentional hard break in the unreleased 0.2.0.dev0 spec. There is no deprecation period, since the project is still in early incubation.

Motivation

Today data is an opaque JSON string:

custom_extensions:
  - vendor_name: SNOWFLAKE
    data: '{"warehouse": "ANALYTICS_WH", "some_unmapped_flag": true}'

Problems with this:

• Values are not first-class YAML/JSON, so JSON Schema can't see them.
• Nested structure is hidden behind JSON.parse.
• Every consumer has to special-case-parse each vendor's string.

Converters increasingly use extensions to carry unmapped or partially mapped properties, so this friction keeps growing.

Change

data is now an object:

custom_extensions:
  - vendor_name: SNOWFLAKE
    data:
      warehouse: ANALYTICS_WH
      some_unmapped_flag: true

• No reserved keys. Keys are entirely vendor-defined.
• Any JSON value is allowed as a value: strings, numbers, booleans, null, arrays, nested objects.
• data is still required. {} is valid.
• A raw string is still possible by putting it under any key the vendor chooses, e.g. data: { text: '{"…"}' }. A bare string is no longer valid.

Files changed

File	Change
core-spec/ossie-schema.json	$defs/CustomExtension.data is now "type": "object"
core-spec/spec.md	Custom Extensions section: new field table, guidance on keys and values, all examples rewritten as native YAML maps (including model, dataset and complete-example snippets)
core-spec/spec.yaml	data: string becomes data: object (5 occurrences)
examples/tpcds_semantic_model.yaml	Salesforce and DBT extensions converted to native maps
python/src/ossie/models.py	OssieCustomExtension.data: str becomes dict[str, Any]
docs/index.md	FAQ and glossary wording updated
python/tests/*, validation/tests/test_validate.py	Tests for object data (nested values, nulls, {}, YAML/JSON round-trip) and for rejecting strings, arrays, numbers, booleans and null

Validation

Command	Result
uv run validation/test_validate.py	✅ OK
uv run --with pytest --with pyyaml --with jsonschema --with sqlglot -m pytest validation/tests/	✅ 118 passed
uv run --with pytest --with jsonschema --with ./python -m pytest python/tests/	✅ 114 passed
cd python && uv run pytest	✅ 114 passed
uv run scripts/generate-spec-types.py --check	✅ in sync
uv run validation/validate.py examples/tpcds_semantic_model.yaml	✅ passed
uv run validation/validate.py examples/flights.semantic_model.yaml	✅ passed
uv run validation/validate.py examples/flights.yaml --schema ontology/ontology.json	✅ passed
uv run validation/validate.py examples/flights.ontology.yaml --schema ontology/ontology.json	✅ passed
uv run validation/validate.py examples/flights.mapping.yaml --schema ontology/mapping.json	✅ passed

Checklist

Specification

• Spec changes are included in core-spec/ and follow the existing structure
• Spec changes have been discussed on the mailing list or in a linked issue
• Breaking changes to the spec are clearly called out in the summary

Ontology (No change needed)

Converters (Out of scope)

Validation

• Validation rules in validation/ are updated if the spec changed
• New validation cases are covered by tests

Documentation

• docs/ is updated to reflect any user-facing changes
• New features or behaviors are documented with examples where appropriate
• CONTRIBUTING.md is updated if the contribution process changed

Examples

• examples/ are added or updated for any new spec constructs or converter support

Tests

• All existing tests pass (pytest / CI green) - core tests
• New functionality is covered by tests

Compliance

• No third-party dependencies are added without PMC/IPMC approval

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 2

Open (2)

• The table uses || prefixes, which typically renders as an extra empty column in standard Markdown… · New
• model_config = ConfigDict(frozen=True) no longer effectively makes OssieCustomExtension… · New

What changed in this PR

Updates the Ossie spec and Python model to represent custom_extensions[].data as a structured object (vendor-defined map) instead of a JSON-encoded string, aligning validation and examples with the new shape.

Changes:

• Update JSON Schema / YAML spec to require custom_extensions[].data as an object
• Update Python model (OssieCustomExtension) and tests to accept objects and reject non-objects
• Rewrite documentation and examples to use native YAML mappings for extension data

File	Description
core-spec/​ossie-schema.json	Changes CustomExtension.data schema type to object and updates description
core-spec/​spec.yaml	Updates the spec type annotations from string → object for custom_extensions[].data
core-spec/​spec.md	Updates the Custom Extensions documentation and examples to use YAML objects
docs/​index.md	Updates FAQ/glossary wording to match object-based extension data
examples/​tpcds_semantic_model.yaml	Migrates example extension data from JSON strings to native YAML objects
python/​src/​ossie/​models.py	Updates OssieCustomExtension.data from str → dict[str, Any]
python/​tests/​test_validation.py	Updates/extends validation tests for object-based data
python/​tests/​test_serialization.py	Adds round-trip tests for nested custom extension data
python/​tests/​test_models.py	Updates fixtures/examples using custom_extensions.data
validation/​tests/​test_validate.py	Adds schema-level validation tests for object vs non-object extension data

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ekut]

Ran the converter suites against this branch merged onto current main (2491533), since the converter workflows on this PR are still waiting for approval. Spec, schema, example and the Python package agree with each other, and the core checks listed in the description pass. What breaks on merge is the converters, which still treat data as a string:

• 7 of the 14 Python converter suites fail (172 tests); all 14 are green on main. In six of them the converter writes data with json.dumps(...) and the schema or the model now rejects it; Sigma builds OssieCustomExtension(data=json.dumps(...)) on the in-repo package, so Sigma imports raise wherever they write an extension. The seventh, the ontology converter, fails the other way: its own CustomExtension has data: str, so it can no longer load examples/tpcds_semantic_model.yaml as updated here.
• A green suite doesn't mean the converter handles the new form. Given its own vendor's extension with data as an object, Cube, Omni, Microsoft, Databricks and Sigma raise TypeError from json.loads, ThoughtSpot raises "expected a JSON string", and Honeydew, NVIDIA and OrionBelt catch the error and drop the extension silently. For OrionBelt, round-tripping its TPC-DS OBML fixture with object data drops all 36 comment entries (the string path keeps them) and renames the column customer_full_name to its SQL expression, with no warning. GoodData already accepts both forms (_get_gooddata_extension).
• converters/README.md, section "Custom Extensions", still tells converter authors that data is a JSON string to parse. ThoughtSpot's README.md and docs/vendor-payload.md say it is never a nested object.

How would you like to sequence this: the converter changes in this PR, or readers first accept both forms the way GoodData does and then the schema flips? The order also interacts with #523: once the converters move to apache/ossie-converters, their CI no longer runs here, and these failures show up at the first submodule bump there instead. Open #378 adds new json.dumps writes for DBT, so it needs the same change whichever lands first.

Per-converter results

Converter	Suite, main → this PR	Writes data as	Reading object data
cube	864 passed → 124 failed	JSON string	TypeError
databricks (python)	103 passed → 103 passed	JSON string	TypeError
dbt	176 passed → 176 passed	—	—
gooddata	94 passed → 94 passed	JSON string	works
honeydew	151 passed → 151 passed	JSON string	dropped silently
microsoft	410 passed → 1 failed	JSON string	TypeError
nvidia	89 passed → 2 failed	JSON string	dropped silently
omni	105 passed → 105 passed	JSON string	TypeError
ontology	170 passed → 1 failed	own model, data: str	fails to load
orionbelt	269 passed → 7 failed	JSON string	dropped silently
sigma	103 passed → 34 failed	JSON string, rejected by the model	TypeError
snowflake	186 passed → 186 passed	—	drops all extensions
thoughtspot	977 passed → 3 failed	JSON string	ConversionError
wisdom	37 passed → 37 passed	—	reports, doesn't parse

Repro: cd converters/<name> && uv sync && uv run pytest (needs uv >= 0.9). Every failure is data rejected as a string by the schema or the model, except ontology, which rejects the object. The Java converters (Databricks Java, Polaris, Salesforce) were not run; Polaris models data as String.

[MikeCarlo]

@ekut We discussed in the last two OSSIE meetings that because we are currently in incubation that breaking changes occur now. Thus, the move to the new Converter Repo makes the most sense to me. Enforce the new standard in the test in the new converter repo.

Tagging @jbonofre about the communication in our meeting notes about breaking changes occurring now.

Here is how I would see the next steps moving:

1. Update the spec with this breaking change, bump the spec with a major version
2. Move the converters to the new repo
3. Validate and update converters to adhere to the new spec as part of testing validations on the new repo

This enables a clean cut between the two bodies of work required.

This PR is in response to this discussion:
#481