Repository navigation
feat(dloc): convert a Dataset Structure Model to data locations and variables - #146
Draft
ehennestad wants to merge 1 commit into
Draft
ehennestad wants to merge 1 commit into
ehennestad wants to merge 1 commit into
Conversation
…ariables Imports a dataset described by a Dataset Structure Model into a NANSEN project, mapping what NANSEN can represent and reporting the rest. nansen.config.dloc.dsm2DataLocationModel returns data location items, variable items and a table of the model elements that were not mapped, each with the reason. nansen.config.dloc.importDatasetStructureModel adds them to a project's models. Adding a data location assigns it a new uuid, so the import relinks each variable to the uuid its location was given. Choosing what a NANSEN session is: an entity type named session with a layout level, otherwise the entity type of the deepest level, because a session has to be found on disk. The subject is an entity type named subject, otherwise the session entity's parent. Both can be overridden. Three places where the two models differ needed a conversion rather than a copy, each checked against NANSEN's own extraction: - A model's regex rule yields its first capture group; NANSEN's expr mode yields the whole match. The text around the group moves into lookbehind and lookahead, so the whole match is the value. - A model's substring rule is a 0-based Python slice. It becomes an ind rule where both bounds are known positions, and an expr rule otherwise, because NANSEN's ind mode accepts end only as 1:end. - File patterns become wildcard file name expressions. Exclude patterns are not mapped: NANSEN's ignore list matches literal substrings, so a pattern such as ^\..* would become "." and exclude every file. Tested with the model written for Garad and Lessmann (2022), a flat folder of ABF recordings: each file becomes a session, the cell becomes the subject, and NANSEN's own session detection finds the ten sessions of a synthetic folder laid out the same way, with their ids and dates. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ehennestad
added this pull request to stack #122
September 19, 2026 16:41
ehennestad
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mechanism
A Dataset Structure Model (DSM) JSON config describes a dataset's folder layout, entity hierarchy and metadata extraction rules. NANSEN describes the same things with data locations (subfolder structure and metadata extraction rules) and data variables (file name expressions and file adapters).
Changes
nansen.config.dloc.dsm2DataLocationModel, which returns data location items, variable items and a report table of the model elements that were not mapped, each with the reason, andnansen.config.dloc.importDatasetStructureModel, which adds them to a project's models and relinks each variable to the uuid its data location was given.regexrule yields its first capture group, while NANSEN'sexprmode yields the whole match, so the text around the group becomes lookbehind and lookahead (convertRegexToWholeMatch). A DSMsubstringrule is a 0-based Python slice; it becomes anindrule where both bounds are known positions and anexprrule otherwise (convertSliceToExtraction). File patterns become wildcard file name expressions (convertFilePatternToWildcard). Exclude patterns are not mapped, because NANSEN's ignore list matches literal substrings.Dsm2DataLocationModelTest,ImportDatasetStructureModelTestwith the fixturedatasetstructure/garad-2022.json: each ABF file of a flat folder becomes a session, the cell becomes the subject, and NANSEN's session detection finds the ten sessions of a synthetic folder with their IDs and dates.🤖 Generated with Claude Code