test(compliance): add portability measurement fixtures M0-M3 (#576)

[MrBlack1995]

Summary

The fixture half of #576 (measurements for model portability across converters). A small portable ladder where each rung adds exactly one capability on top of the one below, so that a difference in a converter's round-trip localizes to a single construct.

Rung	Capability added	Construct
M0	baseline	one dataset, one field, one SUM metric (three variants)
M1	a scalar field, no aggregation	amount * 1.1 calculated column
M2	the core aggregate set	SUM, COUNT, COUNT(DISTINCT), AVG, MIN, MAX
M3	a filtered aggregate	SUM(CASE WHEN ... THEN x ELSE 0 END)

M0 carries three variants: ANSI_SQL qualified (primary), the same model tagged OSSIE_SQL_2026 to isolate tag survival, and the bare-reference form. This matches the three hello-world variants @kayemkim already drove through the nine converters on the #576 thread, so the baseline rung is a committed version of a run that already exists.

Design rules

• Portable dialect only. Every fixture is ANSI_SQL or OSSIE_SQL_2026. A vendor-tagged input would only exercise one converter, which defeats a cross-converter measurement.
• Qualified reference form is primary. Where a metric references a dataset field, the qualified dataset.field form is primary and the bare form is a sibling variant, pending the name-resolution decisions in docs(core-spec): Clarify name resolution for fields, keys, relationships, and metrics #486 and feat: support dataset-scoped metrics #343.
• Conformance is a floor, not the measurement. Scoring is left to a grader that diffs the returned document against the input element by element; that belongs with the interoperability harness in Add cross-converter interoperability harness #348, not here. manifest.yaml lists the fixtures, the element each rung exercises, and the per-element outcome vocabulary.

Scope

Rungs M4-M8 (relationships and cross-dataset metrics, metric-to-metric composition, date and time scalars, window and offset functions) are held pending #354, #591 and #587, and are not in this set.

Placement

The files sit under compliance/portability/ as a starting point. If they should live with the interoperability harness from #348 or elsewhere once that lands, they can move: the fixtures are self-contained and the manifest paths are relative. Happy to relocate per reviewer preference.

Verification

• All 8 fixtures pass validation/validate.py on current main (9531429) with sqlglot 29.0.1.

Related

• Issue Add measurements for model portability across converters #576 (portability measurements)
• Harness Add cross-converter interoperability harness #348, name resolution docs(core-spec): Clarify name resolution for fields, keys, relationships, and metrics #486 / feat: support dataset-scoped metrics #343

Checklist

Compliance

• ASF license headers are present on all new files
• No third-party dependencies are added

This pull request and its description were written by Isaac.

[kayemkim]

I ran all eight fixtures through the same nine converters as the hello-world run, on current main (64a1ae4). 66 documents came back (thoughtspot refused six exports) and every one of them passes validate.py, including the two below that changed meaning.

converter	M0 ansi	M0 ossie	M0 bare	M1 field	M2 qual	M2 bare	M3 qual	M3 bare
cube	kept	kept, tag kept	requalified	kept	kept	requalified	kept	requalified
dbt	kept	relabeled ANSI	requalified	kept	COUNT rewritten	requalified, COUNT rewritten	SUM(SUM(...))	SUM(SUM(...))
omni	kept	dropped, warned	requalified	kept	kept	requalified	kept	kept bare
sigma	SIGMA + quoted ANSI	dropped, warned	dropped as cross-dataset	SIGMA + quoted	SIGMA + quoted	dropped x6	SIGMA + quoted	dropped
wisdom	kept	relabeled ANSI	kept, warned	kept	kept	kept	kept	kept
orionbelt	quoted	relabeled, quoted	+1 helper metric	field renamed to its expression	quoted	+6 helper metrics	quoted	+1 helper metric
honeydew	kept	relabeled, warned	kept	kept	kept, reordered	kept, reordered	kept	kept
microsoft	DAX only	DAX only	DAX only	BLANK()	DAX only	DAX only	BLANK()	BLANK()
thoughtspot	THOUGHTSPOT only	rejected	rejected	rejected	THOUGHTSPOT only	rejected	rejected	rejected

One thing on the fixtures themselves. The three M0 metrics carry no datatype and every metric from M1 up carries one, so M1 changes two things against M0, and thoughtspot's TS-MODEL-METRIC-DATATYPE-UNWRITABLE warning shows up on M1 for a reason that has nothing to do with the scalar field. Giving M0 the same datatype (or taking it off the others) keeps each rung to one construct.

M1 does its job on orionbelt. The calculated field comes back as name: amount * 1.1 with amount_with_tax only in obml_column_name and the synonyms: the export writes the field's expression into the OBML column code, and the import takes code as the Ossie field name. I could not find an issue or PR for it, so I will open one. For the grader it is also a hint that elements need an identity beyond their name, otherwise this scores as one DROPPED_SILENT plus one ADDED.

M3 on dbt is the SUM(SUM(CASE WHEN ...)) that #437 is fixing, with the same before and after text as this fixture, so the rung reproduces an open fix rather than a new defect. What it exposes in manifest.yaml is a missing row: the tag is kept and the text changed, which none of the eight outcomes names. Most such rewrites are benign (cube, dbt and omni requalify SUM(amount) to SUM(sales.amount), dbt turns COUNT(sales.order_id) into SUM(CASE WHEN sales.order_id IS NOT NULL THEN 1 ELSE 0 END) the way MetricFlow itself does, sigma and orionbelt quote identifiers, honeydew reorders metrics), so a REWRITTEN outcome only works if the grader compares normalized expressions, say sqlglot trees after qualification and unquoting, and reports the dbt case as a change in meaning rather than in spelling.

The microsoft column needs a row too. The M1 field and the M3 metric go out as BLANK() with the original expression and dialect stored in annotations, with a warning, and the import does not read those annotations back. A DAX dialect is present, so the current vocabulary scores it SURVIVED_VENDOR_ONLY while the content is gone. Either a PLACEHOLDER outcome, or counting it under DROPPED_DIAGNOSED since the export does warn.

Two rows to read with the legend in mind: thoughtspot exits non-zero for the whole document when one element will not translate (M1 is the amount * 1.1 field), so REJECTED there is a document-level outcome, and sigma drops every bare-form metric as cross-dataset on a one-dataset model, same as on the hello-world run.

On the path, and on who produces this table next time. This one came out of the same hand-driven script as the hello-world run, which is fine for calibrating the vocabulary and not fine as the measurement, since nobody else can rerun it. The manifest points scoring at the harness in #348, but that PR has had no review or commit since September 2 and its description is still the template. Unless Isaac is picking it back up, I would put the grader I signed up for next to these fixtures: a thin runner that drives each converter the way converters/cube/tools/interop_matrix.py already does, and a grader that takes the input document and the returned document and emits the per-element outcomes from manifest.yaml, stdlib only, so this table becomes one command. compliance/ on main is a one-line README from the expression language PR and reads as the place for spec compliance tests, while these fixtures measure converters, which are on their way to apache/ossie-converters (#523). So compliance/portability/ is right for now, with the runner beside it, and the whole directory moves with the converters when that lands, unless JB would rather have it wait for the converters repository.

[MrBlack1995]

@kayemkim this is the measurement the fixtures were for, thank you for driving all eight through the nine converters. Sixty-six documents back and every one passing validate.py is the grounding the vocabulary needed, and the two rows that changed meaning are exactly the cases worth naming precisely. I have pushed the changes below to this branch.

Fixture fix (the one-construct point). You are right that M0 carrying no metric datatype while M1 up carried one made M1 change two things and pulled in thoughtspot's datatype warning for a reason unrelated to the scalar field. The three M0 metrics now carry datatype: Decimal, so the metric datatype is constant from M0 and each rung adds exactly one construct. The three M0 fixtures still validate on current main.

Two outcomes the run exposed, now in manifest.yaml.

• REWRITTEN: a portable tag kept with the expression text changed, which none of the eight named. As you say this only means anything with a normalized comparison, so the grader compares sqlglot trees after stripping table qualifiers and quoting: a requalify (SUM(amount) to SUM(sales.amount)), an unquote, or a reorder auto-clears as cosmetic, while anything it cannot prove equal is flagged for review. The dbt SUM(SUM(CASE ...)) lands in review, which is what surfaces the fix(dbt): preserve aggregate expression semantics #437 case. The dbt COUNT(x) to SUM(CASE WHEN x IS NOT NULL THEN 1 ELSE 0 END) rewrite lands in review too, since a tree comparison cannot prove that semantic identity; I would rather it ask for a human nod than assert equivalence it cannot show.
• PLACEHOLDER: the microsoft case, where the field and metric go out as BLANK() with the original stashed in an annotation the import does not read back. A DAX dialect is present so the old vocabulary scored it SURVIVED_VENDOR_ONLY while the content is gone; PLACEHOLDER names it as a diagnosed loss, distinct from a real vendor-only survival.

Element identity. Agreed, and thank you for opening the orionbelt issue. The grader now pairs a dropped input element with an added output element that carries an equivalent expression, so the M1 field coming back with its own expression as its name scores as one element whose identity was not preserved, rather than one DROPPED_SILENT plus one ADDED.

Document-level outcomes. The grader records REJECTED at document scope, for the thoughtspot case where the whole document exits non-zero when any one element will not translate. Noted too that sigma drops every bare-form metric as cross-dataset even on a one-dataset model.

On who produces the table next time, and the path. Agreed it cannot stay a hand-driven script if it is to be the measurement. I drafted the grader half you describe while this thread was quiet and have pushed it to compliance/portability/grader/: standard library, takes the input document and the returned document and emits the per-element outcomes from manifest.yaml, with sqlglot (already a converter and validate.py dependency) used when present for the normalized comparison above. Thirteen tests, one per outcome plus an identity check that grades the real fixtures all-PORTABLE. If that is useful, I would be glad to pair on the thin runner, modelled on converters/cube/tools/interop_matrix.py, so your table becomes one command; and equally glad for you to take the grader as a starting point and rework it.

Agreed compliance/portability/ is right for now, and that the whole directory should move with the converters when #523 lands. Happy to defer to @jbonofre on whether it waits for the converters repository or moves after.