Repository navigation
fix(gooddata): preserve grain for aliased primary-key fields - #347
Conversation
| attr = _convert_to_attribute(field_def, ds_name) | ||
| attributes.append(attr) | ||
| if field_name in pk_columns: | ||
| if attr.source_column in pk_columns: |
There was a problem hiding this comment.
This matches PK columns by comparing attr.source_column against pk_columns, which is a string/value match rather than a unique field match. If two fields alias the same physical column (e.g. a natural key customer_id and a display alias customer_key that both resolve to ANSI_SQL: customer_id), both get added to grain_ids, producing a bogus multi-attribute composite grain for what's actually a single-column key.
I suggest keyring grain membership off the field's identity (e.g. matching source columns to specific field defs and taking the first match, or building grain from primary_key entries directly rather than scanning all attributes) instead of plain "is this source_column in pk_columns" check.
There was a problem hiding this comment.
building grain from primary_key entries directly rather than scanning all attributes
Great catch. Grain now follows primary_key entries directly, so each key column resolves to one attribute.
There was a problem hiding this comment.
Thanks again. Grain still follows each primary_key entry to one attribute after the rebase.
| if not is_date: | ||
| for f in ds.get("fields", []): | ||
| src = _get_source_column(f) | ||
| col_to_attr[src] = f"attr.{ds_name}.{f['name']}" |
There was a problem hiding this comment.
I found it related while reviewing 😄
col_to_attr is a dict keyed by resolved source_column, with no collision check. If two fields resolve to the same physical column, whichever is processed last silently overwrites the earlier entry. Since _convert_relationship looks up target attributes through this map, a relationship meant to reference one attribute can get silently rewired to a different one (no error, just a wrong GdReferenceTarget in the output).
Given _convert_relationship already raises ValueError for an unresolved column, this should probably raise (or at least warn) on a source_column collision here too, rather than overwriting silently.
There was a problem hiding this comment.
this should probably raise on a source_column collision here too
Agreed. The converter now raises ValueError on duplicate resolved source columns, with regression coverage for ambiguous grain and relationship targets.
There was a problem hiding this comment.
By the way, checking all fields here introduces two issues imho:
- Non-dimension fields become
GdFactwith IDs likefact.{ds_name}.{name}.col_to_attris used by_convert_relationshipto resolve foreign key reference targets (GdReferenceTarget(type="attribute")). Facts cannot be reference targets in GoodData and should not be mapped to phantomattr.*IDs. - If an Ossie dataset defines both a dimension and a fact referencing the same physical column (e.g.
customer_iddimension and acustomer_idcount fact), or defines multiple facts on the same column, this raises aValueErroreven though there is no ambiguity.
We should filter for attribute fields before registering into col_to_attr:
if not is_date:
for f in ds.get("fields", []):
is_attr = f.get("dimension") is not None or _detect_type_from_maql(f) == "attribute"
if not is_attr:
continue
src = _get_source_column(f)
if src in col_to_attr:
raise ValueError(
f"Dataset '{ds_name}': source column '{src}' maps to multiple attributes."
)
col_to_attr[src] = f"attr.{ds_name}.{f['name']}"
There was a problem hiding this comment.
Thanks for the follow-up. The target map now includes attributes only, while duplicate attribute sources still raise an error.
|
@mattfaltyn thanks for the update. I'm doing a new pass. |
jbonofre
left a comment
There was a problem hiding this comment.
Thanks for addressing the grain resolution logic in _convert_ossie_dataset @mattfaltyn !
I did a new full pass and there are two remaining points before we can merge:
- We have to rebase from
mainand use the flat root document format. - Scope
_build_target_infocollision check to attributes.
I've left inline comments 😄
| model = { | ||
| "semantic_model": [ | ||
| { | ||
| "name": "m", | ||
| "datasets": [ | ||
| { | ||
| "name": "customers", | ||
| "source": "db.s.customers", | ||
| "primary_key": ["customer_id"], | ||
| "fields": [field], | ||
| } | ||
| ], | ||
| } |
There was a problem hiding this comment.
main has been updated removing the root semantic_model: [...] wrapper list.
Once rebased onto main, these tests fail with:
ValueError: Ossie model properties must be at the root; semantic_model wrappers are not supported
Please rebase on main and flatten the test fixtures to put name, datasets, and relationships at the root:
model = {
"name": "m",
"datasets": [
{
"name": "customers",
"source": "db.s.customers",
"primary_key": ["customer_id"],
"fields": [field],
}
],
}
(same update needed in test_duplicate_source_columns_are_rejected below)
There was a problem hiding this comment.
Thanks. Rebased onto current main and flattened both test fixtures. The full GoodData suite passes on Python 3.12, 3.13, and 3.14.
| } | ||
| ] | ||
| } | ||
|
|
There was a problem hiding this comment.
Maybe worth to add a test case verifying that a dataset with an attribute and a fact referencing the same physical column (or two facts on the same column) converts without raising an error.
Something like:
| def test_attribute_and_fact_sharing_source_column_are_permitted(): | |
| model = { | |
| "name": "m", | |
| "datasets": [ | |
| { | |
| "name": "sales", | |
| "primary_key": ["order_id"], | |
| "fields": [ | |
| _direct_field("order_id", dimension={}), | |
| _direct_field("order_id"), # Fact referencing the same source column | |
| ], | |
| } | |
| ], | |
| } | |
| dataset = ossie_to_gooddata(model).ldm.datasets[0] | |
| assert len(dataset.attributes) == 1 | |
| assert len(dataset.facts) == 1 | |
| assert dataset.grain[0].id == "attr.sales.order_id" |
There was a problem hiding this comment.
Thanks for the suggestion. Added a regression with one attribute and two facts sharing a source column, including grain and relationship target checks.
Signed-off-by: Matt Faltyn <faltyn.matthew@gmail.com>
Signed-off-by: Matt Faltyn <faltyn.matthew@gmail.com>
Signed-off-by: Matt Faltyn <faltyn.matthew@gmail.com>
68f7dd9 to
55e152a
Compare
Signed-off-by: Matt Faltyn <faltyn.matthew@gmail.com>
jbonofre
left a comment
There was a problem hiding this comment.
Thanks @mattfaltyn for rebasing on main and adopting the flat root document format.
Also the additional test coverage for repeated keys and attribute/fact column sharing looks great (and all checks are green 😄 )!
Summary
primary_keyentries against the converted attribute'ssource_column, which is already derived from the ANSI SQL expression.This keeps the existing behavior for fields whose logical and physical names match while preventing silent loss of key/grain semantics for aliased fields.
Related Issues
Closes #346
Validation
Checklist
Specification
Converters
Documentation
Tests
Compliance