Skip to content

fix(gooddata): preserve grain for aliased primary-key fields - #347

Merged
jbonofre merged 4 commits into
apache:mainfrom
mattfaltyn:fix/gooddata-aliased-primary-key-grain
Sep 28, 2026
Merged

jbonofre merged 4 commits into
apache:mainfrom
mattfaltyn:fix/gooddata-aliased-primary-key-grain

Conversation

@mattfaltyn

@mattfaltyn mattfaltyn commented Aug 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Preserve GoodData dataset grain when an Ossie attribute uses a logical field name that differs from its physical primary-key column.
  • Match primary_key entries against the converted attribute's source_column, which is already derived from the ANSI SQL expression.
  • Cover both explicit dimensions and MAQL-detected attributes with regression cases.

This keeps the existing behavior for fields whose logical and physical names match while preventing silent loss of key/grain semantics for aliased fields.

Related Issues

Closes #346

Validation

Python 3.12: uv run --python 3.12 --frozen pytest -q  -> 94 passed
Python 3.13: uv run --python 3.13 --frozen pytest -q  -> 94 passed
Python 3.14: uv run --python 3.14 --frozen pytest -q  -> 94 passed
uv run --frozen ruff check src tests                  -> passed
git diff --check                                      -> passed

Checklist

Specification

  • No specification changes

Converters

  • Converter logic is updated
  • Regression coverage is included under the GoodData test directory

Documentation

  • Existing documentation was reviewed; no update is needed because this restores the documented primary-key-to-grain mapping without changing the interface

Tests

  • All GoodData tests pass across the CI Python matrix
  • New behavior is covered by tests

Compliance

  • No new source files or dependencies

@jbonofre
jbonofre self-requested a review September 12, 2026 06:12
attr = _convert_to_attribute(field_def, ds_name)
attributes.append(attr)
if field_name in pk_columns:
if attr.source_column in pk_columns:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This matches PK columns by comparing attr.source_column against pk_columns, which is a string/value match rather than a unique field match. If two fields alias the same physical column (e.g. a natural key customer_id and a display alias customer_key that both resolve to ANSI_SQL: customer_id), both get added to grain_ids, producing a bogus multi-attribute composite grain for what's actually a single-column key.

I suggest keyring grain membership off the field's identity (e.g. matching source columns to specific field defs and taking the first match, or building grain from primary_key entries directly rather than scanning all attributes) instead of plain "is this source_column in pk_columns" check.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

building grain from primary_key entries directly rather than scanning all attributes

Great catch. Grain now follows primary_key entries directly, so each key column resolves to one attribute.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again. Grain still follows each primary_key entry to one attribute after the rebase.

if not is_date:
for f in ds.get("fields", []):
src = _get_source_column(f)
col_to_attr[src] = f"attr.{ds_name}.{f['name']}"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found it related while reviewing 😄

col_to_attr is a dict keyed by resolved source_column, with no collision check. If two fields resolve to the same physical column, whichever is processed last silently overwrites the earlier entry. Since _convert_relationship looks up target attributes through this map, a relationship meant to reference one attribute can get silently rewired to a different one (no error, just a wrong GdReferenceTarget in the output).

Given _convert_relationship already raises ValueError for an unresolved column, this should probably raise (or at least warn) on a source_column collision here too, rather than overwriting silently.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should probably raise on a source_column collision here too

Agreed. The converter now raises ValueError on duplicate resolved source columns, with regression coverage for ambiguous grain and relationship targets.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, checking all fields here introduces two issues imho:

  1. Non-dimension fields become GdFact with IDs like fact.{ds_name}.{name}. col_to_attr is used by _convert_relationship to resolve foreign key reference targets (GdReferenceTarget(type="attribute")). Facts cannot be reference targets in GoodData and should not be mapped to phantom attr.* IDs.
  2. If an Ossie dataset defines both a dimension and a fact referencing the same physical column (e.g. customer_id dimension and a customer_id count fact), or defines multiple facts on the same column, this raises a ValueError even though there is no ambiguity.

We should filter for attribute fields before registering into col_to_attr:

           if not is_date:
               for f in ds.get("fields", []):
                   is_attr = f.get("dimension") is not None or _detect_type_from_maql(f) == "attribute"
                   if not is_attr:
                       continue
                   src = _get_source_column(f)
                   if src in col_to_attr:
                       raise ValueError(
                           f"Dataset '{ds_name}': source column '{src}' maps to multiple attributes."
                       )
                   col_to_attr[src] = f"attr.{ds_name}.{f['name']}"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the follow-up. The target map now includes attributes only, while duplicate attribute sources still raise an error.

@mattfaltyn
mattfaltyn requested a review from jbonofre September 12, 2026 16:08
@jbonofre

Copy link
Copy Markdown
Member

@mattfaltyn thanks for the update. I'm doing a new pass.

@jbonofre jbonofre left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the grain resolution logic in _convert_ossie_dataset @mattfaltyn !

I did a new full pass and there are two remaining points before we can merge:

  1. We have to rebase from main and use the flat root document format.
  2. Scope _build_target_info collision check to attributes.

I've left inline comments 😄

Comment on lines +363 to +375
model = {
"semantic_model": [
{
"name": "m",
"datasets": [
{
"name": "customers",
"source": "db.s.customers",
"primary_key": ["customer_id"],
"fields": [field],
}
],
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

main has been updated removing the root semantic_model: [...] wrapper list.

Once rebased onto main, these tests fail with:

  ValueError: Ossie model properties must be at the root; semantic_model wrappers are not supported

Please rebase on main and flatten the test fixtures to put name, datasets, and relationships at the root:

       model = {
           "name": "m",
           "datasets": [
               {
                   "name": "customers",
                   "source": "db.s.customers",
                   "primary_key": ["customer_id"],
                   "fields": [field],
               }
           ],
       }

(same update needed in test_duplicate_source_columns_are_rejected below)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Rebased onto current main and flattened both test fixtures. The full GoodData suite passes on Python 3.12, 3.13, and 3.14.

}
]
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe worth to add a test case verifying that a dataset with an attribute and a fact referencing the same physical column (or two facts on the same column) converts without raising an error.

Something like:

Suggested change
def test_attribute_and_fact_sharing_source_column_are_permitted():
model = {
"name": "m",
"datasets": [
{
"name": "sales",
"primary_key": ["order_id"],
"fields": [
_direct_field("order_id", dimension={}),
_direct_field("order_id"), # Fact referencing the same source column
],
}
],
}
dataset = ossie_to_gooddata(model).ldm.datasets[0]
assert len(dataset.attributes) == 1
assert len(dataset.facts) == 1
assert dataset.grain[0].id == "attr.sales.order_id"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. Added a regression with one attribute and two facts sharing a source column, including grain and relationship target checks.

Signed-off-by: Matt Faltyn <faltyn.matthew@gmail.com>
Signed-off-by: Matt Faltyn <faltyn.matthew@gmail.com>
Signed-off-by: Matt Faltyn <faltyn.matthew@gmail.com>
Signed-off-by: Matt Faltyn <faltyn.matthew@gmail.com>

@jbonofre jbonofre left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mattfaltyn for rebasing on main and adopting the flat root document format.
Also the additional test coverage for repeated keys and attribute/fact column sharing looks great (and all checks are green 😄 )!

@jbonofre
jbonofre merged commit 1ae977d into apache:main Sep 28, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GoodData export drops grain for aliased primary-key fields

2 participants