diff --git a/converters/databricks/java/README.md b/converters/databricks/java/README.md index d9c00ead4..905847b68 100644 --- a/converters/databricks/java/README.md +++ b/converters/databricks/java/README.md @@ -46,7 +46,11 @@ ignored with a notice (the first one wins) rather than rejected. On **import** ( Apache Ossie), Metric-View-only features (`filter`, `parameters`, `materialization`, per-column `format`, measure `window` / `partition`) are instead **preserved** in `custom_extensions[DATABRICKS]`, so -`MV -> Apache Ossie -> MV` is lossless. Any input that breaks a [requirement](#requirements) +`MV -> Apache Ossie -> MV` is lossless. Each imported expression is labeled with the most +portable dialect it fits: `OSSIE_SQL_2026` when it uses only the Apache Ossie expression +language, else `ANSI_SQL` when it uses only standard SQL, else `DATABRICKS` (see +[Dialect labels on import](#dialect-labels-on-import)). Any input that breaks a +[requirement](#requirements) **raises a `ConversionException`** -- the converter never silently drops a field or produces an invalid result. @@ -115,13 +119,37 @@ Each row maps in both directions; the **Notes** flag where a behavior is specifi | `relationship.from`/`to` direction | join `cardinality` | Export: source on the many (`from`) side -> `many_to_one`; on the one (`to`) side -> `one_to_many`. | | `dataset.primary_key` / `unique_keys` | join `rely.at_most_one_match` | Both directions: export sets `at_most_one_match` when a key covers the join columns; import recovers a `unique_keys` from it. | | `dataset.fields[]` | `dimensions[]` | Export: fields flatten into one list and a joined column is qualified by its full join path (`customer.c_name`; `customer.region.r_name` when nested). | -| `field.expression.dialects[]` | `expr` | Export: prefer the `DATABRICKS` dialect, then `ANSI_SQL`, then `OSSIE_SQL_2026` (Apache Ossie's portable, ANSI-SQL-compatible dialect); other alternatives are ignored when a supported one is present, and a field with no supported dialect is dropped with a notice. | +| `field.expression.dialects[]` | `expr` | Export: prefer the `DATABRICKS` dialect, then `ANSI_SQL`, then `OSSIE_SQL_2026` (Apache Ossie's portable, ANSI-SQL-compatible dialect); other alternatives are ignored when a supported one is present, and a field with no supported dialect is dropped with a notice. Import: one entry, labeled with the most portable dialect it fits (see [Dialect labels on import](#dialect-labels-on-import)). | | `metrics[]` | `measures[]` | Fact columns are referenced bare (`SUM(amount)`). A joined column is addressed by dataset name in Apache Ossie and by its full join path in the Metric View, so export expands `SUM(region.population)` to `SUM(customer.region.population)` and import maps it back. | | `field.label` | dimension `display_name` | A measure's `display_name` has no `label` on the Apache Ossie metric shape, so it rides in the stash instead (see the `custom_extensions` row). | | `field` / `metric` `description` | `comment` | | | `ai_context.synonyms` | `synonyms` | Only `synonyms`: every other member of an `ai_context` object is dropped with a notice. | | `custom_extensions[DATABRICKS]` | `filter`, `parameters`, `materialization`, per-column `format`, measure `window` / `partition` / `display_name` | Import stashes Metric-View-only features here; export restores them -- keeping `MV -> Apache Ossie -> MV` lossless. | +### Dialect labels on import + +Import labels each expression with the most portable dialect it fits, so a model that went +`Apache Ossie -> Metric View -> Apache Ossie` keeps a portable expression portable instead of +coming back as `DATABRICKS`. The check is lexical and conservative: anything it does not +recognize stays `DATABRICKS`, which is always correct for an expression read from a Metric View. +An expression also stays `DATABRICKS` when: + +- it is a measure with a `window` or `partition`, or any measure of a view with a `filter`: those + settings change the value and ride only in the `DATABRICKS` stash; +- it is a measure that names anything outside an aggregate call (`cost / row_count`), which the + Metric View resolves to another measure rather than a column; +- it names a parameter, or a dimension other than a plain same-named column, bare or as + `dataset.name`, since the Metric View may resolve that name to something other than the column; +- it belongs to a view with an unnamed dimension other than a fact wildcard (`*`, `source.*`), + such as `customer.*`, whose expanded names may shadow a column; +- it qualifies a name with anything but a dataset (a struct field, or `source.` in a dimension); +- it uses a construct whose Databricks behavior differs from the portable one, such as a window + (`OVER`), `DATEDIFF`, a `VARCHAR`, `FLOAT`, or integer `CAST`, or a double-quoted name. + +The label does not account for behavior the Apache Ossie specification leaves open, such as +integer division, `NULL` handling in `CONCAT` and `GREATEST`, rounding of ties, the result type of +`DATE_TRUNC` and `DATEADD`, and the unit names a date function accepts. + ## Requirements Conversion throws a `ConversionException` (rather than guessing or emitting something invalid) when @@ -161,6 +189,7 @@ src/main/java/org/apache/ossie/converter/databricks/ OssieConverterCommon.java shared constants, YAML I/O, map accessors, the stash codec OssieToMetricView.java export: Apache Ossie -> Metric View v1.1 MetricViewToOssie.java import: Metric View v1.1 -> Apache Ossie + ExpressionDialect.java import: the most portable dialect label for an expression OssieDatabricksConverter.java command-line entry point (export / import) ``` diff --git a/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/ExpressionDialect.java b/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/ExpressionDialect.java new file mode 100644 index 000000000..f62186c2a --- /dev/null +++ b/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/ExpressionDialect.java @@ -0,0 +1,571 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ossie.converter.databricks; + +import static org.apache.ossie.converter.databricks.OssieConverterCommon.DIALECT_ANSI; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.DIALECT_DATABRICKS; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.DIALECT_OSSIE_SQL; + +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.Deque; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +/** + * Labels a Databricks SQL expression with the most portable Apache Ossie dialect it can carry on + * import: {@code OSSIE_SQL_2026} when it uses only the Ossie expression language + * (core-spec/expression_language.md), else {@code ANSI_SQL} when it uses only standard SQL, else + * {@code DATABRICKS}. + * + *

A lexical check against allowlists, not a parser, and deliberately conservative: a construct + * counts as portable only if it is listed here and, where the Ossie spec defines its behavior, + * Databricks evaluates it that way. Anything else stays {@code DATABRICKS}, which is always + * correct because the expression came from a Metric View. The traps it guards against include + * Databricks reading {@code "x"} as a string where portable SQL reads an identifier, a doubled + * quote in {@code 'a''b'} that a legacy Databricks setting reads as two concatenated literals, + * backslash escapes in strings, {@code TIMESTAMP} carrying the session time zone, and same-named + * functions with other Databricks meanings. It does not check behavior the spec leaves open, such + * as integer division, NULL handling in CONCAT or GREATEST, rounding of ties, or the timestamp + * type DATE_TRUNC and DATEADD return. + */ +final class ExpressionDialect { + + // Functions in the Ossie expression language whose Databricks meaning matches. Left out on + // purpose: TRUNC/TRUNCATE (date truncation in Databricks), DATEDIFF (counts whole elapsed units + // in Databricks, unit boundaries crossed elsewhere), REGEXP_* (regex flavors differ), TO_CHAR + // and the format-string TO_DATE (format tokens differ), TO_TIMESTAMP (session-time-zone result), + // and window functions (OVER is not portable: Databricks sorts NULLs first by default, and a + // window over a measure depends on the query's grain). + private static final Set OSSIE_FUNCTIONS = Set.of( + "SUM", "COUNT", "AVG", "MIN", "MAX", "STDDEV", "STDDEV_POP", "STDDEV_SAMP", "VARIANCE", + "VAR_POP", "VAR_SAMP", "MEDIAN", "PERCENTILE_CONT", "PERCENTILE_DISC", + "APPROX_COUNT_DISTINCT", "APPROX_PERCENTILE", "CURRENT_DATE", "CURRENT_TIMESTAMP", "YEAR", + "QUARTER", "MONTH", "DAY", "DAYOFYEAR", "HOUR", "MINUTE", "SECOND", "EXTRACT", "DATE_PART", + "DATE_TRUNC", "DATEADD", "TO_DATE", "CONCAT", "LENGTH", "LOWER", "UPPER", "TRIM", "LTRIM", + "RTRIM", "LEFT", "RIGHT", "SUBSTRING", "REPLACE", "SPLIT_PART", "POSITION", "CHARINDEX", + "CONTAINS", "STARTSWITH", "ENDSWITH", "ABS", "ROUND", "FLOOR", "CEIL", "CEILING", "MOD", + "SIGN", "POWER", "SQRT", "EXP", "LN", "LOG", "LOG10", "SIN", "COS", "TAN", "ASIN", "ACOS", + "ATAN", "ATAN2", "RADIANS", "DEGREES", "PI", "GREATEST", "LEAST", "IF", "IFF", "NULLIF", + "COALESCE", "IFNULL", "NVL", "NVL2", "ZEROIFNULL", "NULLIFZERO", "CAST", "TRY_CAST"); + + // Standard SQL functions whose Databricks meaning matches. Those outside OSSIE_FUNCTIONS + // (CORR, COVAR_*, EVERY, the *_LENGTH family) make an expression ANSI_SQL rather than Ossie. + private static final Set ANSI_FUNCTIONS = Set.of( + "SUM", "COUNT", "AVG", "MIN", "MAX", "STDDEV_POP", "STDDEV_SAMP", "VAR_POP", "VAR_SAMP", + "CORR", "COVAR_POP", "COVAR_SAMP", "EVERY", "PERCENTILE_CONT", "PERCENTILE_DISC", + "CURRENT_DATE", "CURRENT_TIMESTAMP", "EXTRACT", "CAST", "COALESCE", "NULLIF", "UPPER", + "LOWER", "TRIM", "SUBSTRING", "POSITION", "CHAR_LENGTH", "CHARACTER_LENGTH", "OCTET_LENGTH", + "ABS", "MOD", "LN", "EXP", "POWER", "SQRT", "FLOOR", "CEIL", "CEILING"); + + // Aggregate calls (and the FILTER / WITHIN GROUP clauses that qualify one): in a measure, a bare + // name inside one is a column, while outside one it refers to another measure. + private static final Set AGGREGATES = Set.of( + "SUM", "COUNT", "AVG", "MIN", "MAX", "STDDEV", "STDDEV_POP", "STDDEV_SAMP", "VARIANCE", + "VAR_POP", "VAR_SAMP", "MEDIAN", "PERCENTILE_CONT", "PERCENTILE_DISC", + "APPROX_COUNT_DISTINCT", "APPROX_PERCENTILE", "CORR", "COVAR_POP", "COVAR_SAMP", "EVERY", + "FILTER", "GROUP"); + + // Argument counts the portable form allows, for functions whose other Databricks forms mean + // something else (one-argument LOG = LN, COUNT(a, b) counting rows where all are non-null, + // CEIL(x, scale), and so on). + private static final Map ARITY = Map.ofEntries( + Map.entry("COUNT", new int[] {1, 1}), + Map.entry("CEIL", new int[] {1, 1}), + Map.entry("CEILING", new int[] {1, 1}), + Map.entry("FLOOR", new int[] {1, 1}), + Map.entry("DATEADD", new int[] {3, 3}), + Map.entry("LOG", new int[] {2, 2}), + Map.entry("TO_DATE", new int[] {1, 1}), + Map.entry("APPROX_COUNT_DISTINCT", new int[] {1, 1}), + Map.entry("APPROX_PERCENTILE", new int[] {2, 2}), + Map.entry("TRIM", new int[] {1, 1}), + Map.entry("LTRIM", new int[] {1, 1}), + Map.entry("RTRIM", new int[] {1, 1}), + Map.entry("REPLACE", new int[] {3, 3}), + Map.entry("CHARINDEX", new int[] {2, 2})); + + // CAST target types whose Databricks meaning matches. Left out: VARCHAR/CHAR (a plain STRING in + // Databricks, without length semantics), FLOAT/REAL (32-bit in Databricks), the integer types + // (Databricks truncates a fractional value where others round), TIME, TIMESTAMP (session time + // zone), and DECIMAL/NUMERIC without (p, s) (DECIMAL(10, 0) in Databricks). + private static final Set OSSIE_TYPES = Set.of( + "STRING", "DECIMAL", "NUMERIC", "DOUBLE", "BOOLEAN", "DATE", "TIMESTAMP_NTZ"); + private static final Set ANSI_TYPES = Set.of("DECIMAL", "NUMERIC", "BOOLEAN", "DATE"); + private static final Set PARAMETERIZED_TYPES = Set.of("DECIMAL", "NUMERIC"); + + // Keywords valid in both dialects wherever they appear (the grammar itself is not checked). + private static final Set KEYWORDS = Set.of( + "CASE", "WHEN", "THEN", "ELSE", "END", "AND", "OR", "NOT", "IN", "BETWEEN", "LIKE", "IS", + "NULL", "TRUE", "FALSE", "DISTINCT", "ORDER", "BY", "ASC", "DESC", "WITHIN"); + + // Keywords that may directly precede an opening parenthesis. + private static final Set KEYWORDS_BEFORE_PAREN = Set.of( + "AND", "OR", "NOT", "WHEN", "THEN", "ELSE", "CASE", "IN", "LIKE", "ILIKE", "BETWEEN", + "DISTINCT", "BY", "FILTER", "GROUP"); + + // Keywords only valid in a specific position, checked in Scan.word(). + private static final Set CONTEXT_KEYWORDS = Set.of( + "AS", "FROM", "FOR", "ESCAPE", "WHERE", "NULLS", "BOTH", "LEADING", "TRAILING"); + + // Keywords that are themselves a complete operand. + private static final Set VALUE_KEYWORDS = Set.of("NULL", "TRUE", "FALSE"); + + // Typed-literal prefixes: DATE '2024-01-15'. TIMESTAMP is left out (session time zone). + private static final Set LITERAL_PREFIXES = Set.of("DATE", "TIMESTAMP_NTZ"); + + // Words that mean an expression is not a plain portable one (subqueries, intervals, windows, + // session functions, Databricks-only operators and constructors). + private static final Set REJECTED_WORDS = Set.of( + "SELECT", "WITH", "UNION", "EXCEPT", "INTERSECT", "HAVING", "JOIN", "LATERAL", "INTERVAL", + "EXISTS", "ARRAY", "MAP", "STRUCT", "RLIKE", "REGEXP", "DIV", "ANY", "SOME", "ALL", + "QUALIFY", "LIMIT", "WINDOW", "VALUES", "TABLE", "OVER", "FILTER", "GROUP", "CURRENT_USER", + "USER", "SESSION_USER", "CURRENT_TIME", "LOCALTIME", "GROUPING__ID"); + + private static final List SYMBOLS = List.of( + "<>", "<=", ">=", "!=", "||", "<", ">", "=", "+", "-", "*", "/", "%", "(", ")", ",", "."); + // Databricks-only operators (shifts included) and comments, checked before SYMBOLS can match a + // prefix of them. + private static final List REJECTED_SYMBOLS = List.of( + "<=>", "==", "->", "=>", "<<", ">>", "--", "/*"); + private static final Pattern NUMBER = + Pattern.compile("(?:\\d+(?:\\.\\d*)?|\\.\\d+)(?:[eE][+-]?\\d+)?"); + + private ExpressionDialect() {} + + /** + * The most portable dialect {@code expr} can carry: OSSIE_SQL_2026, ANSI_SQL, or DATABRICKS. + * + * @param datasets lower-cased dataset names of the Apache Ossie model; {@code a.b} is portable + * only when {@code a} is one of them (not {@code source.}, a join alias, or a struct) + * @param opaqueNames lower-cased names whose meaning comes from the Metric View rather than a + * column (parameters, dimensions shadowing a column); a reference to one, bare or as + * {@code dataset.name}, is not portable + * @param measure whether {@code expr} is a measure, where a name outside an aggregate call + * refers to another measure rather than a column, so it is not portable + */ + static String classify( + String expr, Set datasets, Set opaqueNames, boolean measure) { + List tokens = expr == null ? null : tokenize(expr); + if (tokens == null || tokens.isEmpty()) { + return DIALECT_DATABRICKS; + } + Scan scan = new Scan(tokens, datasets, opaqueNames, measure); + if (!scan.run()) { + return DIALECT_DATABRICKS; + } + return scan.ossie ? DIALECT_OSSIE_SQL : scan.ansi ? DIALECT_ANSI : DIALECT_DATABRICKS; + } + + private enum Kind { IDENT, NUMBER, STRING, SYMBOL } + + private record Token(Kind kind, String text) { + boolean is(String symbol) { + return kind == Kind.SYMBOL && text.equals(symbol); + } + + boolean isWord(String word) { + return kind == Kind.IDENT && text.equalsIgnoreCase(word); + } + + String upper() { + return text.toUpperCase(Locale.ROOT); + } + } + + /** Splits {@code s} into tokens, or returns null on any construct this check does not handle. */ + private static List tokenize(String s) { + List out = new ArrayList<>(); + int n = s.length(); + int i = 0; + while (i < n) { + char c = s.charAt(i); + if (Character.isWhitespace(c)) { + i++; + } else if (c == '\'') { + int end = s.indexOf('\'', i + 1); + if (end < 0) { + return null; + } + String body = s.substring(i + 1, end); + // A backslash escape reads differently, and so can a doubled quote: 'a''b' is one + // literal with a quote in standard SQL, but two concatenated literals under a legacy + // Databricks setting. + if (body.indexOf('\\') >= 0 || (end + 1 < n && s.charAt(end + 1) == '\'')) { + return null; + } + out.add(new Token(Kind.STRING, body)); + i = end + 1; + } else if (Character.isLetter(c) || c == '_') { + int j = i + 1; + while (j < n && isIdentPart(s.charAt(j))) { + j++; + } + out.add(new Token(Kind.IDENT, s.substring(i, j))); + i = j; + } else if (Character.isDigit(c) + || (c == '.' && i + 1 < n && Character.isDigit(s.charAt(i + 1)))) { + Matcher m = NUMBER.matcher(s).region(i, n); + if (!m.lookingAt()) { + return null; + } + int j = m.end(); + // A trailing letter is a Databricks typed-number suffix (1L, 2BD) or a malformed number. + if (j < n && isIdentPart(s.charAt(j))) { + return null; + } + out.add(new Token(Kind.NUMBER, s.substring(i, j))); + i = j; + } else { + String sym = symbolAt(s, i); + if (sym == null) { + return null; + } + out.add(new Token(Kind.SYMBOL, sym)); + i += sym.length(); + } + } + return out; + } + + private static boolean isIdentPart(char c) { + return Character.isLetterOrDigit(c) || c == '_'; + } + + private static String symbolAt(String s, int i) { + for (String rejected : REJECTED_SYMBOLS) { + if (s.startsWith(rejected, i)) { + return null; + } + } + for (String sym : SYMBOLS) { + if (s.startsWith(sym, i)) { + return sym; + } + } + return null; // backticks, double quotes, ':', '[', '!', '|', ... + } + + private static boolean isKeyword(String u) { + return KEYWORDS.contains(u) || KEYWORDS_BEFORE_PAREN.contains(u) + || CONTEXT_KEYWORDS.contains(u); + } + + /** One pass over the tokens, narrowing which portable dialects the expression still fits. */ + private static final class Scan { + private final List tokens; + private final Set datasets; + private final Set opaqueNames; + private final boolean measure; + private final Deque frames = new ArrayDeque<>(); + private String pendingOpener = ""; + boolean ossie = true; + boolean ansi = true; + + /** A parenthesized group: the word that opened it, and its top-level argument shape. */ + private static final class Frame { + final String opener; + final boolean aggregate; + int commas; + boolean empty = true; + boolean keywordForm; // EXTRACT ... FROM, POSITION ... IN, FILTER (WHERE ...), and so on + + Frame(String opener) { + this.opener = opener; + this.aggregate = AGGREGATES.contains(opener); + } + } + + Scan(List tokens, Set datasets, Set opaqueNames, boolean measure) { + this.tokens = tokens; + this.datasets = datasets; + this.opaqueNames = opaqueNames; + this.measure = measure; + } + + boolean run() { + for (int i = 0; i < tokens.size(); i++) { + Token t = tokens.get(i); + Token prev = i > 0 ? tokens.get(i - 1) : null; + Token next = i + 1 < tokens.size() ? tokens.get(i + 1) : null; + if (prev != null && endsOperand(prev) && startsOperand(t) && !isTypedLiteral(prev, t)) { + return false; // two operands side by side: 'a' 'b', or an alias as in SUM(x) total + } + Frame top = frames.peek(); + if (top != null && !t.is(")")) { + top.empty = false; + } + boolean ok = switch (t.kind) { + case STRING, NUMBER -> true; + case SYMBOL -> symbol(t, prev, next, top); + case IDENT -> ident(i, t, prev, next, top); + }; + if (!ok) { + return false; + } + } + return frames.isEmpty(); + } + + private static boolean endsOperand(Token t) { + if (t.kind == Kind.IDENT) { + String u = t.upper(); + return !isKeyword(u) || VALUE_KEYWORDS.contains(u) || u.equals("END"); + } + return t.kind == Kind.NUMBER || t.kind == Kind.STRING || t.is(")"); + } + + private static boolean startsOperand(Token t) { + if (t.kind == Kind.IDENT) { + String u = t.upper(); + return !isKeyword(u) || VALUE_KEYWORDS.contains(u); + } + return t.kind == Kind.NUMBER || t.kind == Kind.STRING; + } + + private static boolean isTypedLiteral(Token prev, Token t) { + return t.kind == Kind.STRING && prev.kind == Kind.IDENT + && LITERAL_PREFIXES.contains(prev.upper()); + } + + private boolean symbol(Token t, Token prev, Token next, Frame top) { + switch (t.text) { + case "(": + frames.push(new Frame(pendingOpener)); + pendingOpener = ""; + return true; + case ")": + return !frames.isEmpty() && close(frames.pop()); + case ",": + if (top == null) { + return false; + } + top.commas++; + return true; + case ".": + // Only between two identifiers; `t.*` and a dot after a call are Databricks-only here. + return prev != null && prev.kind == Kind.IDENT && next != null && next.kind == Kind.IDENT; + case "%": + case "!=": + ansi = false; // listed by the Ossie language but not standard SQL + return true; + default: + return true; + } + } + + private boolean close(Frame f) { + if (f.keywordForm && f.commas > 0) { + return false; // a keyword form takes no extra arguments: TRIM(BOTH FROM s, 'x') + } + switch (f.opener) { + case "POSITION": + return f.keywordForm; // POSITION(sub IN str), not POSITION(sub, str) + case "FILTER": + return f.keywordForm; // FILTER (WHERE ...) + case "SUBSTRING": + if (!f.keywordForm) { + ansi = false; // the comma form is the Ossie language; standard SQL uses FROM/FOR + } + break; + default: + break; + } + int[] arity = ARITY.get(f.opener); + if (arity == null || f.keywordForm) { + return true; + } + int args = f.empty ? 0 : f.commas + 1; + return args >= arity[0] && args <= arity[1]; + } + + private boolean ident(int i, Token t, Token prev, Token next, Frame top) { + if (prev != null && prev.is(".")) { + return true; // a later part of a qualified name, already checked at its first part + } + String u = t.upper(); + if (next != null && next.is(".")) { + return qualifiedName(i, t); + } + if (top != null && isCast(top) && prev != null && prev.isWord("AS")) { + return castType(u, next); + } + if (next != null && next.is("(")) { + return call(u, prev); + } + if (next != null && next.kind == Kind.STRING && !isKeyword(u)) { + return typedLiteral(u); + } + return word(t, u, prev, next, top); + } + + /** {@code dataset.field} is portable; any other head, deeper paths, or calls are not. */ + private boolean qualifiedName(int i, Token head) { + if (!datasets.contains(head.text.toLowerCase(Locale.ROOT)) || !isColumnPosition()) { + return false; + } + boolean twoParts = i + 2 < tokens.size() && tokens.get(i + 2).kind == Kind.IDENT; + boolean deeper = i + 3 < tokens.size() && tokens.get(i + 3).is("."); + boolean call = i + 3 < tokens.size() && tokens.get(i + 3).is("("); + // `source.col` reads the raw column in Databricks, but `dataset.col` names the field when + // a dimension of that name shadows it. + return twoParts && !deeper && !call + && !opaqueNames.contains(tokens.get(i + 2).text.toLowerCase(Locale.ROOT)); + } + + /** False for a name in a measure outside an aggregate call, which refers to a measure. */ + private boolean isColumnPosition() { + if (!measure) { + return true; + } + for (Frame f : frames) { + if (f.aggregate) { + return true; + } + } + return false; + } + + private boolean call(String u, Token prev) { + boolean inOssie = OSSIE_FUNCTIONS.contains(u); + boolean inAnsi = ANSI_FUNCTIONS.contains(u); + if (inOssie || inAnsi) { + ossie &= inOssie; + // Standard SQL writes CURRENT_DATE and CURRENT_TIMESTAMP without parentheses. + ansi &= inAnsi && !u.startsWith("CURRENT_"); + pendingOpener = u; + return true; + } + if (!KEYWORDS_BEFORE_PAREN.contains(u)) { + return false; // a function neither dialect lists, MEASURE(...) and window OVER included + } + if (u.equals("FILTER")) { + if (prev == null || !prev.is(")")) { + return false; // FILTER only qualifies an aggregate call: SUM(x) FILTER (WHERE ...) + } + ossie = false; + } else if (u.equals("ILIKE")) { + ansi = false; + } else if (u.equals("GROUP") && (prev == null || !prev.isWord("WITHIN"))) { + return false; + } + pendingOpener = u; + return true; + } + + private boolean typedLiteral(String u) { + if (!LITERAL_PREFIXES.contains(u)) { + return false; // TIMESTAMP '...' (session time zone), X'..', and other prefixes + } + ansi &= u.equals("DATE"); + return true; + } + + private boolean castType(String u, Token next) { + boolean params = next != null && next.is("("); + if (PARAMETERIZED_TYPES.contains(u) != params) { + return false; // DECIMAL needs (p, s); other types take none + } + if (params) { + pendingOpener = "TYPE"; + } + boolean inOssie = OSSIE_TYPES.contains(u); + boolean inAnsi = ANSI_TYPES.contains(u); + ossie &= inOssie; + ansi &= inAnsi; + return inOssie || inAnsi; + } + + private boolean word(Token t, String u, Token prev, Token next, Frame top) { + String opener = top == null ? "" : top.opener; + switch (u) { + case "AS": + return isCast(top); + case "FROM": + if (opener.equals("EXTRACT")) { + top.keywordForm = true; + return true; + } + if (prev != null && prev.isWord("DISTINCT")) { + return true; // IS [NOT] DISTINCT FROM + } + return standardKeywordForm(top, "SUBSTRING", "TRIM"); + case "FOR": + return standardKeywordForm(top, "SUBSTRING"); + case "BOTH": + case "LEADING": + case "TRAILING": + return standardKeywordForm(top, "TRIM"); + case "WHERE": + if (opener.equals("FILTER") && prev != null && prev.is("(")) { + top.keywordForm = true; + return true; + } + return false; + case "IN": + if (opener.equals("POSITION")) { + top.keywordForm = true; + } + return true; + case "ILIKE": + ansi = false; + return true; + case "ESCAPE": + ossie = false; + return true; + case "NULLS": + if (next != null && (next.isWord("FIRST") || next.isWord("LAST"))) { + ossie = false; + return true; + } + return false; + default: + if (REJECTED_WORDS.contains(u)) { + return false; + } + // Any other identifier is a column, field, or date-part reference, unless its meaning + // comes from the Metric View itself. + return isKeyword(u) + || (isColumnPosition() && !opaqueNames.contains(t.text.toLowerCase(Locale.ROOT))); + } + } + + /** FROM / FOR / BOTH inside SUBSTRING or TRIM: standard SQL, not the Ossie language. */ + private boolean standardKeywordForm(Frame top, String... openers) { + if (top == null) { + return false; + } + for (String opener : openers) { + if (top.opener.equals(opener)) { + top.keywordForm = true; + ossie = false; + return true; + } + } + return false; + } + + private static boolean isCast(Frame f) { + return f != null && (f.opener.equals("CAST") || f.opener.equals("TRY_CAST")); + } + } +} diff --git a/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java b/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java index c1a6a3c36..fc1e3205b 100644 --- a/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java +++ b/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java @@ -82,6 +82,10 @@ final class MetricViewToOssie { private static final Pattern NON_EQUI_RE = Pattern.compile("[<>!]=|<>|[<>]"); private static final Pattern AND_SPLIT_RE = Pattern.compile("\\s+AND\\s+", Pattern.CASE_INSENSITIVE); private static final Pattern EQ_CLAUSE_RE = Pattern.compile("^\\s*(.+?)\\s*=\\s*(.+?)\\s*$"); + // A fact wildcard expands to dimensions named after, and equal to, the fact's own columns. + private static final Pattern FACT_WILDCARD_RE = Pattern.compile( + "^\\s*(?:source\\s*\\.\\s*)?\\*(?:\\s+EXCEPT\\s*\\(.*\\))?\\s*$", + Pattern.CASE_INSENSITIVE); private MetricViewToOssie() {} @@ -191,6 +195,12 @@ private static Map convertView( fieldsByDataset.put((String) d.get("name"), new ArrayList<>()); } List dims = !dimList.isEmpty() ? dimList : fieldList; + Set datasetNames = new HashSet<>(); + for (Map d : datasets) { + datasetNames.add(((String) d.get("name")).toLowerCase(Locale.ROOT)); + } + DialectScope scope = + new DialectScope(datasetNames, viewDefinedNames(view, dims), hasHiddenNames(dims)); for (Object dimObj : dims) { Map dim = asMap(dimObj); if (isWildcard(dim)) { @@ -198,7 +208,7 @@ private static Map convertView( + "' has no Apache Ossie field representation; skipped"); continue; } - Object[] converted = convertDimension(dim, aliasToDataset, factName); + Object[] converted = convertDimension(dim, aliasToDataset, factName, scope); fieldsByDataset.get((String) converted[0]).add(converted[1]); } for (Map d : datasets) { @@ -211,6 +221,8 @@ private static Map convertView( List metrics = new ArrayList<>(); // Fixed once the join tree is walked, so compile it once for all measures. Pattern aliasHead = qualifierChainPattern(aliasToDataset.keySet()); + // A view filter changes every measure's value, and it rides only in the DATABRICKS stash. + boolean filtered = view.containsKey("filter"); for (Object mObj : asList(get(view, "measures"))) { Map m = asMap(mObj); if (isWildcard(m)) { @@ -218,7 +230,7 @@ private static Map convertView( + "' has no Apache Ossie metric representation; skipped"); continue; } - metrics.add(convertMeasure(m, aliasHead, aliasToDataset)); + metrics.add(convertMeasure(m, aliasHead, aliasToDataset, scope, filtered)); } Map model = new LinkedHashMap<>(); @@ -445,8 +457,8 @@ private static String[] splitAlias(String operand) { return new String[] {null, operand}; } - private static Object[] convertDimension( - Map dim, Map aliasToDataset, String factName) { + private static Object[] convertDimension(Map dim, + Map aliasToDataset, String factName, DialectScope scope) { String name = requireStr(dim, "name", "dimension"); String expr = requireStr(dim, "expr", "dimension '" + name + "'"); String[] resolved = resolveColumn(expr, aliasToDataset, factName); @@ -455,7 +467,7 @@ private static Object[] convertDimension( Map field = new LinkedHashMap<>(); field.put("name", name); - field.put("expression", dialectExpr(ossieExpr)); + field.put("expression", dialectExpr(ossieExpr, scope.dialectOf(ossieExpr, name, false))); if (truthy(get(dim, "comment"))) { field.put("description", get(dim, "comment")); } @@ -514,14 +526,20 @@ private static String[] resolveColumn( * and it is what {@code resolveColumn} already does for a dimension -- measures used to keep the * raw path, so a nested one survived only by accident. */ - private static Map convertMeasure( - Map measure, Pattern aliasHead, Map aliasToDataset) { + private static Map convertMeasure(Map measure, + Pattern aliasHead, Map aliasToDataset, DialectScope scope, + boolean filtered) { String name = requireStr(measure, "name", "measure"); String rawExpr = requireStr(measure, "expr", "measure '" + name + "'"); String expr = rewriteQualifiers(rawExpr, aliasHead, aliasToDataset::get); Map metric = new LinkedHashMap<>(); metric.put("name", name); - metric.put("expression", dialectExpr(expr)); + // A window, partition, or view filter changes the measure's value and rides only in the + // DATABRICKS stash; a portable label would let another tool compute a plain aggregate instead. + boolean portable = + !filtered && !measure.containsKey("window") && !measure.containsKey("partition"); + metric.put("expression", + dialectExpr(expr, portable ? scope.dialectOf(expr, name, true) : DIALECT_DATABRICKS)); if (truthy(get(measure, "comment"))) { metric.put("description", get(measure, "comment")); } @@ -545,9 +563,10 @@ private static Map convertMeasure( return metric; } - private static Map dialectExpr(String expr) { + /** An Apache Ossie expression with the single dialect entry {@code label: expr}. */ + private static Map dialectExpr(String expr, String label) { Map dialect = new LinkedHashMap<>(); - dialect.put("dialect", DIALECT_DATABRICKS); + dialect.put("dialect", label); dialect.put("expression", expr); List dialects = new ArrayList<>(); dialects.add(dialect); @@ -559,4 +578,61 @@ private static Map dialectExpr(String expr) { private static boolean isWildcard(Map col) { return !col.containsKey("name"); } + + /** + * Lower-cased bare names whose meaning comes from the view rather than a column: parameters, and + * dimensions other than a plain same-named column (a later expression naming one may resolve to + * the dimension instead of the column). + */ + private static Set viewDefinedNames(Map view, List dims) { + Set names = new HashSet<>(); + for (Object p : asList(get(view, "parameters"))) { + String name = str(get(asMap(p), "name")); + if (name != null) { + names.add(name.toLowerCase(Locale.ROOT)); + } + } + for (Object d : dims) { + Map dim = asMap(d); + String name = str(get(dim, "name")); + String expr = str(get(dim, "expr")); + if (name != null && (expr == null || !expr.trim().equalsIgnoreCase(name.trim()))) { + names.add(name.toLowerCase(Locale.ROOT)); + } + } + return names; + } + + /** + * Whether a dimension has no name and is not a fact wildcard: a joined wildcard such as + * {@code customer.*}, or a derived name, adds dimensions this import cannot see, and any of them + * may shadow a column of the same name. + */ + private static boolean hasHiddenNames(List dims) { + for (Object d : dims) { + Map dim = asMap(d); + String expr = str(get(dim, "expr")); + if (isWildcard(dim) && (expr == null || !FACT_WILDCARD_RE.matcher(expr).matches())) { + return true; + } + } + return false; + } + + /** + * What an expression may name besides a column, for labeling its dialect (ExpressionDialect). + * With {@code hiddenNames}, any bare name may be a dimension, so nothing is portable. + */ + private record DialectScope(Set datasets, Set viewNames, boolean hiddenNames) { + /** The most portable dialect for the expression of the column {@code self}. */ + String dialectOf(String expr, String self, boolean measure) { + if (hiddenNames) { + return DIALECT_DATABRICKS; + } + // A column naming itself refers to the source column, not to itself. + Set opaque = new HashSet<>(viewNames); + opaque.remove(self.toLowerCase(Locale.ROOT)); + return ExpressionDialect.classify(expr, datasets, opaque, measure); + } + } } diff --git a/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/ExpressionDialectSuite.java b/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/ExpressionDialectSuite.java new file mode 100644 index 000000000..d83a5a92c --- /dev/null +++ b/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/ExpressionDialectSuite.java @@ -0,0 +1,173 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ossie.converter.databricks; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +import java.util.Set; +import org.junit.jupiter.api.Test; + +public class ExpressionDialectSuite { + + private static final Set DATASETS = Set.of("sales", "customer"); + private static final Set VIEW_NAMES = Set.of("p_region", "status"); + + private static void assertDialect(String expected, String... expressions) { + for (String expr : expressions) { + assertEquals(expected, ExpressionDialect.classify(expr, DATASETS, VIEW_NAMES, false), expr); + } + } + + private static void assertMeasureDialect(String expected, String... expressions) { + for (String expr : expressions) { + assertEquals(expected, ExpressionDialect.classify(expr, DATASETS, VIEW_NAMES, true), expr); + } + } + + @Test + public void ossieLanguageExpressionsAreOssieSql() { + assertDialect("OSSIE_SQL_2026", + "amount", + "SUM(sales.amount)", + "COUNT(customer.c_name)", + "COUNT(*)", + "COUNT(DISTINCT customer_id)", + "SUM(CASE WHEN o_status = 'done' THEN amount ELSE 0 END)", + "DATE_TRUNC('month', order_date)", + "EXTRACT(YEAR FROM order_date)", + "CAST(amount AS DECIMAL(10, 2))", + "CAST(amount AS DOUBLE)", + "CAST(amount AS STRING)", + "COALESCE(a, b) + 1.5e3", + "a IS NOT DISTINCT FROM b", + "name ILIKE '%x%'", + "x IN ('a', 'b') AND NOT y IS NULL", + "amount % 2 != 0", + "PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY amount)", + "DATEADD(day, 7, order_date)", + "DATE '2024-01-15'", + "TIMESTAMP_NTZ '2024-01-15 10:00:00'", + "POSITION('a' IN s)", + "SUBSTRING(s, 2, 3)", + "CURRENT_DATE()", + "a || b", + "LOG(10, x)"); + } + + @Test + public void standardSqlOutsideTheOssieLanguageIsAnsiSql() { + assertDialect("ANSI_SQL", + "CHAR_LENGTH(name)", + "SUM(amount) FILTER (WHERE amount > 0)", + "CORR(a, b)", + "SUBSTRING(s FROM 2 FOR 3)", + "TRIM(BOTH ' ' FROM s)", + "s LIKE 'a!%' ESCAPE '!'", + "PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY amount NULLS LAST)"); + } + + @Test + public void aMeasureIsPortableOnlyWhenEveryNameSitsInAnAggregate() { + // Outside an aggregate call, a bare name in a measure refers to another measure. + assertMeasureDialect("OSSIE_SQL_2026", + "SUM(amount)", + "SUM(sales.amount) / NULLIF(COUNT(DISTINCT customer_id), 0)", + "SUM(CASE WHEN (a > 1) THEN b END) * 2", + "PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY amount)"); + assertMeasureDialect("ANSI_SQL", "SUM(amount) FILTER (WHERE amount > 0)"); + assertMeasureDialect("DATABRICKS", + "total_cost / row_count", + "running_revenue * 2", + "SUM(amount) + sales.amount", + "DATEADD(day, 1, MAX(order_date))"); + } + + @Test + public void databricksOnlyOrAmbiguousExpressionsStayDatabricks() { + assertDialect("DATABRICKS", + // Databricks reads "x" as a string where portable SQL reads an identifier, and a legacy + // Databricks setting reads 'a''b' as two concatenated literals rather than one. + "\"amount\"", + "'it''s'", + "'a' 'b'", + "'a\\'b'", + "`my col`", + // Databricks-only syntax and operators. + "x::int", + ":param * 2", + "x <=> y", + "x == y", + "arr[0]", + "t.*", + "a.b.c", + "transform(arr, x -> x + 1)", + "name RLIKE 'a.*'", + "INTERVAL 7 DAY", + "1L + 2", + "a -- comment", + "MEASURE(revenue)", + "schema.fn(x)", + "SUM(x) total", + "x IN (SELECT y FROM t)", + "CURRENT_USER()", + "localtime", + "o_flags >> 3", + "x << 2", + // Each half fits only one portable dialect: CHAR_LENGTH is standard SQL outside the Ossie + // language, while CURRENT_DATE() and TIMESTAMP_NTZ are Ossie forms outside standard SQL. + "CHAR_LENGTH(s) > 0 AND d = CURRENT_DATE()", + "CHAR_LENGTH(s) > 0 AND d = TIMESTAMP_NTZ '2024-01-15 10:00:00'", + // Names that are not a dataset column: the fact alias, a struct field, a parameter, and a + // dimension that may shadow a column of the same name. + "UPPER(source.o_status)", + "address.city", + "amount * p_region", + "UPPER(status)", + "SUM(sales.status)", + "customer.address.city", + "UPPER(customer.nation.n_name)", + "customer.fn(x)", + // Windows: Databricks sorts NULLs first by default, and the frame depends on query grain. + "SUM(x) OVER (ORDER BY v)", + "ROW_NUMBER() OVER (ORDER BY d NULLS LAST)", + "SUM(x) OVER w", + // Same function name, different meaning or form in Databricks. + "DATEDIFF(day, start_date, end_date)", + "DATEDIFF(end_date, start_date)", + "LOG(x)", + "COUNT(a, b)", + "CEIL(x, 2)", + "TRUNC(d, 'MM')", + "POSITION('a', s)", + "TRIM(s, 'x')", + "TRIM(BOTH FROM s, 'x')", + "REGEXP_EXTRACT(s, 'a')", + // CAST targets whose Databricks type differs: VARCHAR is a plain STRING, FLOAT is 32-bit, + // an integer cast truncates, and a bare DECIMAL is DECIMAL(10, 0). + "CAST(z AS VARCHAR(5))", + "CAST(x AS FLOAT)", + "CAST(x AS INT)", + "CAST(x AS DECIMAL)", + "CAST(d AS TIMESTAMP)", + "TIMESTAMP '2024-01-01'", + // Not a single well-formed expression. + "", + "(a", + "a, b"); + } +} diff --git a/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterRoundTripSuite.java b/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterRoundTripSuite.java index be899dba7..17fb62d9a 100644 --- a/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterRoundTripSuite.java +++ b/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterRoundTripSuite.java @@ -510,10 +510,15 @@ private void assertMvRoundTrip(Map mv, long seed) { assertEquals(e1, e2, "join nesting" + ctx); } + // Import labels each expression with the most portable dialect it fits, so a DATABRICKS + // expression can come back as OSSIE_SQL_2026 or ANSI_SQL; the round trip preserves its text. + private static final Set CONVERTER_DIALECTS = + Set.of("DATABRICKS", "ANSI_SQL", "OSSIE_SQL_2026"); + private static String exprOf(Map obj) { for (Object dObj : asList(asMap(obj.get("expression")).get("dialects"))) { Map d = asMap(dObj); - if ("DATABRICKS".equals(d.get("dialect"))) { + if (CONVERTER_DIALECTS.contains(d.get("dialect"))) { return (String) d.get("expression"); } } diff --git a/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterSuite.java b/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterSuite.java index 0c4dcf604..5527524a0 100644 --- a/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterSuite.java +++ b/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterSuite.java @@ -24,6 +24,7 @@ import java.io.InputStream; import java.nio.charset.StandardCharsets; +import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; @@ -1633,6 +1634,210 @@ public void importRejectsNonStringScalarsWhereStringsRequired() { assertTrue(e2.getMessage().contains("must be a string"), e2.getMessage()); } + // -- import direction: expression dialects --------------------------------- + + /** The single dialect entry of a converted field or metric, as {dialect, expression}. */ + @SuppressWarnings("unchecked") + private static List dialectOf(Map column) { + List dialects = + (List) ((Map) column.get("expression")).get("dialects"); + assertEquals(1, dialects.size(), dialects.toString()); + Map d = (Map) dialects.get(0); + return List.of((String) d.get("dialect"), (String) d.get("expression")); + } + + @Test + @SuppressWarnings("unchecked") + public void importLabelsEachExpressionWithItsMostPortableDialect() { + // A plain expression is labeled OSSIE_SQL_2026, standard SQL outside the Ossie language + // ANSI_SQL, and anything Databricks-specific stays DATABRICKS. + String mv = + "version: '1.1'\n" + + "source: c.s.orders\n" + + "dimensions:\n" + + "- {name: status, expr: o_status}\n" + + "- {name: name_length, expr: CHAR_LENGTH(o_name)}\n" + + "- {name: order_month, expr: \"date_format(o_date, 'yyyy-MM')\"}\n" + + "measures:\n" + + "- {name: revenue, expr: SUM(o_amount)}\n"; + Map model = (Map) importMv(mv); + List fields = (List) ((Map) + ((List) model.get("datasets")).get(0)).get("fields"); + assertEquals(List.of("OSSIE_SQL_2026", "o_status"), + dialectOf((Map) fields.get(0))); + assertEquals(List.of("ANSI_SQL", "CHAR_LENGTH(o_name)"), + dialectOf((Map) fields.get(1))); + assertEquals(List.of("DATABRICKS", "date_format(o_date, 'yyyy-MM')"), + dialectOf((Map) fields.get(2))); + Map revenue = + (Map) ((List) model.get("metrics")).get(0); + assertEquals(List.of("OSSIE_SQL_2026", "SUM(o_amount)"), dialectOf(revenue)); + } + + @Test + public void importKeepsWindowAndPartitionMeasuresDatabricks() { + // A window or partition measure's value depends on settings that live only in the DATABRICKS + // stash, so its expression keeps the DATABRICKS label even when the text alone is portable. + String mv = + "version: '1.1'\n" + + "source: c.s.orders\n" + + "dimensions:\n" + + "- {name: order_date, expr: o_date}\n" + + "measures:\n" + + "- {name: revenue, expr: SUM(o_amount)}\n" + + "- name: running_revenue\n" + + " expr: SUM(o_amount)\n" + + " window:\n" + + " - {order: order_date, range: cumulative, semiadditive: last}\n" + + "- name: daily_revenue\n" + + " expr: SUM(o_amount)\n" + + " partition: {include: [order_date], outer_aggregate: SUM}\n"; + assertEquals(Map.of("order_date", "OSSIE_SQL_2026", "revenue", "OSSIE_SQL_2026", + "running_revenue", "DATABRICKS", "daily_revenue", "DATABRICKS"), dialectsByName(mv)); + } + + @Test + public void importKeepsEveryMeasureOfAFilteredViewDatabricks() { + // A view filter changes every measure's value and lives only in the DATABRICKS stash, so the + // measures stay DATABRICKS; a dimension's per-row value does not depend on it. + String mv = + "version: '1.1'\n" + + "source: c.s.orders\n" + + "filter: o_status = 'done'\n" + + "dimensions:\n" + + "- {name: status, expr: o_status}\n" + + "measures:\n" + + "- {name: revenue, expr: SUM(o_amount)}\n"; + assertEquals(Map.of("status", "OSSIE_SQL_2026", "revenue", "DATABRICKS"), dialectsByName(mv)); + } + + @Test + public void importKeepsNamesDefinedByTheViewDatabricks() { + // Portable only when every name is a column or `dataset.column`: a parameter, a dimension + // that may shadow a column, the `source.` fact alias, and a struct field are not. A + // dimension naming itself (`status`) reads the source column. + String mv = + "version: '1.1'\n" + + "source: c.s.orders\n" + + "parameters:\n" + + "- {name: min_amount, data_type: DOUBLE}\n" + + "joins:\n" + + "- {name: customer, source: c.s.customer, on: source.o_custkey = customer.c_custkey}\n" + + "dimensions:\n" + + "- {name: status, expr: UPPER(status)}\n" + + "- {name: status_code, expr: 'LEFT(status, 1)'}\n" + + "- {name: source_status, expr: UPPER(source.o_status)}\n" + + "- {name: city, expr: address.city}\n" + + "- {name: customer_name, expr: UPPER(customer.c_name)}\n" + + "measures:\n" + + "- {name: big_orders, expr: 'COUNT(IF(o_amount > min_amount, 1, NULL))'}\n" + + "- {name: revenue, expr: SUM(o_amount)}\n"; + assertEquals(Map.of("status", "OSSIE_SQL_2026", "status_code", "DATABRICKS", + "source_status", "DATABRICKS", "city", "DATABRICKS", "customer_name", "OSSIE_SQL_2026", + "big_orders", "DATABRICKS", "revenue", "OSSIE_SQL_2026"), dialectsByName(mv)); + } + + @Test + public void importKeepsMeasureReferencesAndShadowedQualifiedColumnsDatabricks() { + // A bare name outside an aggregate refers to another measure, which may carry a window; and + // `source.o_amount` reads the raw column, where the Apache Ossie `orders.o_amount` would name + // the shadowing `o_amount` dimension. + String mv = + "version: '1.1'\n" + + "source: c.s.orders\n" + + "joins:\n" + + "- {name: customer, source: c.s.customer, on: source.o_custkey = customer.c_custkey}\n" + + "dimensions:\n" + + "- {name: order_date, expr: o_date}\n" + + "- {name: o_amount, expr: o_amount * 100}\n" + + "- {name: c_acctbal, expr: customer.c_acctbal * 2}\n" + + "measures:\n" + + "- {name: cost, expr: SUM(cost)}\n" + + "- {name: row_count, expr: COUNT(*)}\n" + + "- {name: avg_cost, expr: cost / row_count}\n" + + "- name: running_cost\n" + + " expr: SUM(cost)\n" + + " window:\n" + + " - {order: order_date, range: cumulative, semiadditive: last}\n" + + "- {name: running_x2, expr: running_cost * 2}\n" + + "- {name: raw_amount, expr: SUM(source.o_amount)}\n" + + "- {name: raw_balance, expr: SUM(customer.c_acctbal)}\n"; + assertEquals(Map.of("order_date", "OSSIE_SQL_2026", "o_amount", "OSSIE_SQL_2026", + "c_acctbal", "OSSIE_SQL_2026", "cost", "OSSIE_SQL_2026", "row_count", "OSSIE_SQL_2026", + "avg_cost", "DATABRICKS", "running_cost", "DATABRICKS", "running_x2", "DATABRICKS", + "raw_amount", "DATABRICKS", "raw_balance", "DATABRICKS"), dialectsByName(mv)); + } + + @Test + public void importKeepsViewsWithUnseenDimensionNamesDatabricks() { + // A joined wildcard or a derived name adds dimensions the import cannot see, and any of them + // may shadow a column (`customer.*` adds `comment`), so nothing in the view is portable. A + // fact wildcard only adds the fact's own columns under their own names. + String head = + "version: '1.1'\n" + + "source: c.s.orders\n" + + "joins:\n" + + "- {name: customer, source: c.s.customer, on: source.o_custkey = customer.c_custkey}\n" + + "dimensions:\n" + + "- {name: status, expr: o_status}\n"; + String measures = "measures:\n- {name: commenters, expr: COUNT(DISTINCT comment)}\n"; + Map hidden = Map.of("status", "DATABRICKS", "commenters", "DATABRICKS"); + assertEquals(hidden, dialectsByName(head + "- expr: customer.*\n" + measures)); + assertEquals(hidden, dialectsByName(head + "- expr: customer.comment\n" + measures)); + assertEquals(Map.of("status", "OSSIE_SQL_2026", "commenters", "OSSIE_SQL_2026"), + dialectsByName(head + "- expr: source.*\n" + measures)); + } + + /** The dialect label of every field and metric imported from {@code mv}, by name. */ + @SuppressWarnings("unchecked") + private static Map dialectsByName(String mv) { + Map model = (Map) importMv(mv); + List columns = new ArrayList<>(); + for (Object ds : (List) model.get("datasets")) { + columns.addAll((List) ((Map) ds).getOrDefault("fields", List.of())); + } + columns.addAll((List) model.getOrDefault("metrics", List.of())); + Map out = new HashMap<>(); + for (Object c : columns) { + Map column = (Map) c; + out.put((String) column.get("name"), dialectOf(column).get(0)); + } + return out; + } + + /** A single sales dataset and one SUM metric, written in {@code dialect}. */ + private static String helloWorldModel(String dialect) { + return "version: 0.2.0.dev0\n" + + "name: hello_world\n" + + "datasets:\n" + + "- name: sales\n" + + " source: hello.public.sales\n" + + " fields:\n" + + " - {name: amount, expression: {dialects: [{dialect: " + dialect + + ", expression: amount}]}}\n" + + "metrics:\n" + + "- {name: total_amount, expression: {dialects: [{dialect: " + dialect + + ", expression: SUM(sales.amount)}]}}\n"; + } + + @Test + @SuppressWarnings("unchecked") + public void portableDialectSurvivesOssieToMetricViewToOssie() { + // Apache Ossie -> Metric View -> Apache Ossie keeps a portable expression portable: an + // OSSIE_SQL_2026 model comes back OSSIE_SQL_2026, and an ANSI_SQL one comes back in the + // Ossie language (ANSI SQL's portable subset) rather than as DATABRICKS. + for (String dialect : new String[] {"OSSIE_SQL_2026", "ANSI_SQL"}) { + String mv = OssieConverter.convertOssieToMetricView(helloWorldModel(dialect), null).yaml; + Map back = (Map) importMv(mv); + Map amount = (Map) ((List) ((Map) + ((List) back.get("datasets")).get(0)).get("fields")).get(0); + Map total = + (Map) ((List) back.get("metrics")).get(0); + assertEquals("OSSIE_SQL_2026", dialectOf(amount).get(0), dialect); + assertEquals("OSSIE_SQL_2026", dialectOf(total).get(0), dialect); + } + } + // -- hardening: strict schema/stash parsing and structured failures ------- @Test diff --git a/converters/databricks/java/src/test/resources/ossie_fixtureB_ossie.yaml b/converters/databricks/java/src/test/resources/ossie_fixtureB_ossie.yaml index 21d7531e0..0ec8396f1 100644 --- a/converters/databricks/java/src/test/resources/ossie_fixtureB_ossie.yaml +++ b/converters/databricks/java/src/test/resources/ossie_fixtureB_ossie.yaml @@ -20,7 +20,9 @@ # Fixture B -- expected Apache Ossie produced from fixtureB_metric_view.yaml. MV-only # features are stashed in custom_extensions[DATABRICKS], keyed by their exact v1.1 # field name. Exporting this back must reproduce fixtureB_metric_view.yaml. -# Model name is derived from the fact table (`lineitem`). +# Model name is derived from the fact table (`lineitem`). The field carries the most portable +# dialect it fits (OSSIE_SQL_2026); the metrics stay DATABRICKS because the view `filter`, and +# the `partition` on `order_count`, change their values and live only in the DATABRICKS stash. version: "0.2.0.dev0" @@ -33,7 +35,7 @@ datasets: - name: line_number expression: dialects: - - dialect: DATABRICKS + - dialect: OSSIE_SQL_2026 expression: l_linenumber custom_extensions: - vendor_name: DATABRICKS