feat(databricks): label imported expressions with their most portable dialect

[Haoranli503]

Summary

Addresses the dev@ thread on dialect round trips (https://lists.apache.org/thread/dnl13g1nlh7l2o6gnw9qjp7wqmrh0tds): importing a Metric View used to label every expression DATABRICKS, so a model taken Apache Ossie -> Metric View -> Apache Ossie came back Databricks-only even when it was plain ANSI or Ossie SQL.

Import now labels each expression with the most portable dialect it fits: OSSIE_SQL_2026 when it uses only the Apache Ossie expression language, else ANSI_SQL when it uses only standard SQL, else DATABRICKS.

Before and after, for the hello world model (one sales dataset, SUM(sales.amount)) written in ANSI_SQL or OSSIE_SQL_2026:

# before: Ossie -> MV -> Ossie
- dialect: DATABRICKS
  expression: SUM(amount)
# after
- dialect: OSSIE_SQL_2026
  expression: SUM(amount)

The check (ExpressionDialect) is lexical, not a parser, and deliberately conservative. Anything it does not recognize stays DATABRICKS, which is always correct for an expression read from a Metric View. A construct counts as portable only if it is allowlisted and, where the spec defines its behavior, Databricks evaluates it the same way. An expression stays DATABRICKS when:

• Databricks reads it differently from portable SQL. Examples: "x" (a string in Databricks), 'a''b' (two concatenated literals under a legacy setting), backslash escapes, TIMESTAMP (session time zone), DATEDIFF (counts whole elapsed units), a VARCHAR, FLOAT, integer, or bare DECIMAL CAST, a window (OVER, where NULLs sort first by default), or one-argument LOG, COUNT(a, b), CEIL(x, scale), and the << / >> shifts.
• It names something other than a dataset column: a struct field, source. in a dimension, a parameter, or a dimension that may shadow a column (bare or as dataset.name). In a measure, any name outside an aggregate call counts as a reference to another measure (cost / row_count). A view with an unnamed dimension other than * or source.* (such as customer.*) keeps everything DATABRICKS, since its expanded names may shadow columns.
• It is a measure with a window or partition, or any measure of a view with a filter. These settings change the value and live only in the DATABRICKS stash.

Export already prefers DATABRICKS, then ANSI_SQL, then OSSIE_SQL_2026, so MV -> Apache Ossie -> MV stays lossless. Behavior the spec leaves open (integer division, NULL handling in CONCAT/GREATEST, rounding of ties, the result type of DATE_TRUNC/DATEADD, date unit names) is listed in the README as not checked. This is a first, crude pass meant to cover the basic cases; it can be extended as the spec settles.

Related Issues

Checklist

Specification

• Spec changes are included in core-spec/ and follow the existing structure
• Spec changes have been discussed on the mailing list or in a linked issue
• Breaking changes to the spec are clearly called out in the summary

Ontology

• Ontology changes in ontology/ are consistent with spec changes
• New or modified terms are defined and documented

Converters

• Converter logic in converters/ is updated to reflect spec or ontology changes
• New converters include tests under the converter's test directory
• If adding a new converter, .github/labeler.yml is updated with the new path

Validation

• Validation rules in validation/ are updated if the spec changed
• New validation cases are covered by tests

Documentation

• docs/ is updated to reflect any user-facing changes
• New features or behaviors are documented with examples where appropriate
• CONTRIBUTING.md is updated if the contribution process changed

Examples

• examples/ are added or updated for any new spec constructs or converter support

Tests

• All existing tests pass (pytest / CI green)
• New functionality is covered by tests

Compliance

• ASF license headers are present on all new source files
• No third-party dependencies are added without PMC/IPMC approval
• If third-party source code is vendored/copied (not just declared as a dependency), NOTICE and/or LICENSE have been updated per ASF policy

This pull request and its description were written by Isaac.

[kayemkim]

I round-tripped the hello world model from Chris's thread through this branch. SUM(amount) comes back OSSIE_SQL_2026 whether it went in as ANSI_SQL or OSSIE_SQL_2026, where main returned DATABRICKS for both. The README rules held on the cases I tried: a measure over a derived dimension (SUM(net) with net: amount - 1) and every measure under a view filter stay DATABRICKS, CHAR_LENGTH lands on ANSI_SQL, and TRUNC, CURRENT_TIME and CAST(x AS INT) stay DATABRICKS as documented.

One design question. dialects is a list, and many expressions here fit both OSSIE_SQL_2026 and ANSI_SQL. Emitting every label that fits, with the same text, instead of only the most portable one would let a model that went in as ANSI_SQL come back with that label intact, which is what the thread's fidelity check counts, and a consumer that only knows ANSI_SQL could still read it. Export order would not change. The cost is a duplicated expression text per field, so I can see the argument either way.

One note for later: SUM(amount) FILTER (WHERE amount > 0) is labeled ANSI_SQL, which is right against the spec as it stands. Once #382 lands, FILTER would belong in the Ossie set too.

[jbonofre]

LGTM, thanks for this!

The dialect classifier reads cleanly, and the demotion of view filter, window and partition to DATABRICKS is handled well. The new labels also still round-trip through the OssieConverterCommon fallback.

One small thing that could be fixed: || is in SYMBOLS and falls into the default -> true branch of symbol(), so it is never marked as non-Ossie. An expression like a || b can end up labeled OSSIE_SQL_2026, but core-spec/expression_language.md doesn't list || as an operator. A spec-following consumer could reject or misparse it.

Suggested fix: handle || in symbol() like % and !=. Clear ossie and keep ansi, so it gets the ANSI_SQL label. A test covering a a || b would be good too.

[Haoranli503]

Thanks @kayemkim, great point on duplicating expression that can fit multiple labels , I'll discuss with Chris a bit on weather we'll do it. And for the SUM(amount) FILTER (WHERE amount > 0), I noted it as a follow up and will apply once #382 lands.

[Haoranli503]

Thanks, @jbonofre. I noticed something interesting in the String Manipulation (REQUIRED) section of the spec: || is listed as a supported operation in the source .md file, but it doesn’t render correctly. I suspect this may be related to how it’s escaped. Is || intentionally unsupported, or is this just a rendering issue?

[christianeu-db]

Thanks, @jbonofre. I noticed something interesting in the String Manipulation (REQUIRED) section of the spec: || is listed as a supported operation in the source .md file, but it doesn’t render correctly. I suspect this may be related to how it’s escaped. Is || intentionally unsupported, or is this just a rendering issue?

That's a good find - that looks like a rendering issue (there's been a few of those we've had to fix). I'm happy to file a PR with you as co-author

[christianeu-db]

Thanks @kayemkim, great point on duplicating expression that can fit multiple labels , I'll discuss with Chris a bit on weather we'll do it. And for the SUM(amount) FILTER (WHERE amount > 0), I noted it as a follow up and will apply once #382 lands.

@Haoranli503 @kayemkim

I'd initially bias towards most portable. IIRC, the ANSI dialect comes from early OSI before the expression language had even been fleshed out. For now, Ossie SQL is generally a subset of ANSI. The distinction will become more interesting if Ossie supports functions / constructs outside of ANSI. Then, you might need to export both.

@kayemkim if you have the time, I'd be curious if you'd like to join in on the parent thread to help seed some of the initial measurement test cases / methodology.