Repository navigation
docs(core-spec): Add Ossie specification layers overview - #452
justint-db wants to merge 10 commits into
Conversation
willpugh
left a comment
willpugh
left a comment
There was a problem hiding this comment.
I think this looks like a pretty good first draft.
Tell me when you are ready to move it out of draft mode. I don't see anything blocking in there.
justint-db
changed the title
Introduces layered_query_interface.md, a short overview of the four-layer query model (Expression Language, Relational Interface, Wide Table, Ontology) for readers navigating the individual layer specs. Clarifies that Layer 1 is an interchange / model-definition language, that query languages are a per-layer concern, and that Layer 2 specifically extends Layer 1 expressions with relational SQL constructs plus MEASURE(). References both the original layered-interface proposal (Justin Talbot) and the four-layer refinement (Will Pugh), and credits both as contributors. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…ce only Co-authored-by: Isaac <no-reply@databricks.com>
…ce overview Expands the Background section with links to the five dev@ossie.apache.org threads that trace how the layered model emerged and converged, from Will Pugh's original foundational semantics proposal through the August debate over separating normative from opinionated behaviors, to the September convergence on the hybrid layered model. Co-authored-by: Isaac <no-reply@databricks.com>
…de Table Query Interface - Retitle document to "Ossie Specification Layers" - Add summary table showing the role of each layer (model definition, query interface x2, ontology workstream) - Rename Layer 2 to "Relational Query Interface" and Layer 3 to "Wide Table Query Interface" to make clear both are query interfaces - Clarify Layer 1 is a model definition concern, not a query concern - Tighten Layer 4 description as a separate workstream Co-authored-by: Isaac <no-reply@databricks.com>
- Rename layered_query_interface.md → spec_layers.md - Rename Layer 1 from "Expression Language" to "Model Definition" - Expand Layer 1 to cover both the model format (spec.md) and the expression language (expression_language.md), framing them together as the authoring/interchange layer Co-authored-by: Isaac <no-reply@databricks.com>
…Layer 2 equivalence Address review feedback: update the status line, state up front that this is an informational overview pointing to the normative layer specs, and clarify that Layer 2 is the semantic target for Layer 3 rather than a required execution path. Co-authored-by: Isaac <no-reply@databricks.com>
justint-db
changed the title
812327e to
cf79792
Compare
…tion, and Layer 3 drop-down Address remaining discussion feedback on the spec layers overview: - Layer 2 adds a measure type and in-query measure definitions, not only MEASURE(). - Clarify that Layer 1 expressions have SQL meaning over a single relation, and that the query interfaces define evaluation across a model's multiple datasets. - State that a Layer 3 consumer can write a query in Layer 2 instead. Co-authored-by: Isaac <no-reply@databricks.com>
| CTEs, ordering, etc. To this it adds one extension, `MEASURE()`, which evaluates a named | ||
| measure at its declared grain. | ||
|
|
||
| Layer 2 establishes the guarantee: **measures are evaluated without duplication regardless of |
There was a problem hiding this comment.
Layer 3 resolves which source tables to join.
With that, would joining explicitly to other models be a supported scenario in Layer 2? If it is, shouldn't we define the expected semantics for the cases like below
- A physical table that is part of a model, is joined directly.
- Two separate models are explicitly joined.
There was a problem hiding this comment.
Layer 2 is designed to sit below the Layer 3 multi-table interface. Layer 2 objects behave like tables with measures, while Layer 3 provides a higher-level interface over a graph of datasets. Layer 3 has additional semantics (e.g., dimension domains can depend on which measures are selected) that make interoperability with SQL joins more involved.
To your questions:
- Joining a physical table: Supported. A regular SQL table can be joined with a Layer 2 dataset, just as it can with any other table.
- Joining two models: Layer 2 supports explicitly joining datasets from different models. However, joining two Layer 3 model-level query interfaces is outside its scope. Layer 2 operates on datasets and other table-shaped objects, not entire models.
It's reasonable to define how Layer 3 objects, including their graph-aware semantics and dynamic dimension domains, interact with broader SQL. But I'd expect that to be addressed in the Layer 3 proposal rather than Layer 2.
| how they are queried.** Fan-out and chasm-trap errors cannot occur when measures are queried | ||
| through this interface. The model's datasets are exposed as SQL relations; consumers query them | ||
| with ordinary SQL, using `MEASURE()` in place of raw aggregates. | ||
|
|
There was a problem hiding this comment.
Shouldn't the BI tools go explicitly through Layer 3? Isn't that simpler, and wouldn't that result in avoiding potentially incorrect queries?
There was a problem hiding this comment.
Generally no. Layer 3 interfaces tend to be very hard for SQL-based BI tools to work with. The reason is that Layer 3 is not relational. So while the results of individual queries against Layer 3 may look reasonable in isolation (and are often completely usable by a human), they do not compose well together.
Perhaps a typical example is a Top N filter, e.g. "Show me the # of unique customers over time in the top 10 states by total revenue".
SQL requires expressing this question as a join between two subqueries that compute the # of unique customers by date and total revenue by state separately. Layer 3 decides which joins to use for those two subqueries independently. If it picks different joins or join types (which is common), Layer 3 does not guarantee that the set of states considered by one subquery align with the set of states considered by the other, so the join of the two subqueries can be incorrect. It could, for example, return no states, even when there are 10 valid states to return.
There was a problem hiding this comment.
Experimentally, we've found that BI tools have a much lower barrier to entry connecting to Layer 2 while preserving their existing feature sets.
We've started collecting BI-generated SQL examples in the repository. BI tools generate fairly complex SQL (e.g. subqueries, joins, window functions) and generally assume that the objects they're querying behave like regular tables.
This creates two challenges when connecting directly to Layer 3:
- SQL compatibility: BI-generated SQL may not run because operations such as joins and COUNT(*) aren't defined over Layer 3 objects.
- Semantic compatibility: BI tools make assumptions about table-like behavior (e.g. fixed dimension domains) when optimizing queries or implementing features like LoD calculations, Top N filters, and relationships. Those assumptions may not hold for Layer 3 objects, potentially producing incorrect results.
For example, these Tableau experiments illustrate Tableau implementing complex features on top of Layer 2 without product changes, using its existing support for passthrough aggregates.
BI tools also have their own semantics for things like filter propagation and dimension domains in multi-table analyses. Supporting all of these behaviors through Layer 3 would likely require additional extensions and customization.
I agree that Layer 3's higher-level semantics and validity checks are valuable, especially for clients that want the semantic model to handle joins and query construction. But for BI tools that already implement these features, Layer 2 provides a more natural integration point: they can preserve their existing behavior while reusing measures defined in Ossie datasets.
Follow the updates to the Relational Query Interface spec (apache#354): the evaluation function is AGG(), and "grain-safe" becomes "duplication-free". Co-authored-by: Isaac <no-reply@databricks.com>
4 participants
Summary
Adds
core-spec/spec_layers.md, a short overview document that orients readers to Ossie's four specification layers before they dive into any individual layer spec.spec.md) and expression language (expression_language.md) — the interchange and authoring layerMEASURE()for grain-safe aggregation (open PR: #354)Key points the document establishes:
The document also includes a Background section with links to the prior proposals (Justin Talbot, Will Pugh) and the mailing list threads that trace how the layered model emerged and converged.
References
AI disclosure
Per the ASF Generative Tooling Guidance, this contribution was prepared with AI assistance. All specification decisions and design choices are mine. I have reviewed and verified every change.