Carry the terms in the identifier, so they travel with it

[jwrosewell]

Draft. The specification this implements has merged as 51Degrees/specifications#27, and this stays in draft until the cloud has been updated to write both fields. pipeline-dotnet#409 goes first, because the cloud is .NET and needs that package published before it can write anything.

What this is

Two fields of the 51Did payload, read by fiftyone_pipeline_did.

The Terms is one byte after the match key, saying which terms document the identifier was created under so that the terms travel with the identifier instead of alongside it. A receiver can then tell which document was in force when the identifier was made, rather than having to be told separately and rather than trusting an address whose contents can be edited later.

The payload version is bits 4 and 5 of the flags byte, saying which layout the payload follows.

The Terms

The byte is an index into a table in the specification and not a version number, so that a later document can live at any address rather than only at one the specification could compose from a number. An index is never reused or repointed once published, because an identifier issued under it has to stay readable years later.

fod_id.terms answers with the address of the document. The package turns the index into the address, so a caller never handles the byte.

Index	Document	fod_id.terms
0	Not stated in the identifier	None
1	Model Terms for Marketing, version 2	https://m4ow.uk/mtm/2.txt
any other	One this package cannot name	None

That one member is the whole of the addition to the public surface. The address for index 1 is exactly https://m4ow.uk/mtm/2.txt, the versioned text of the Model Terms for Marketing version 2, and no other form of it. The package answers with the address and never fetches it, because what to do with the document is the receiver's decision.

The named value the specification gives for each index lives in _terms.py, private to the package as _layout.py is, so the names are recorded once and a caller never handles the byte.

The payload version

This package reads version 0 and refuses every other version with FodIdParseStatus.UNSUPPORTED_PAYLOAD_VERSION, and the raising readers name the version they found in the message.

No field is read under the layout this package knows once the version says otherwise. A later version exists precisely because a field moved, so reading such a payload here would answer with values that are wrong rather than absent, which is worse than refusing. A version that nothing checks protects nothing.

The version is not exposed. Either this package read the layout, in which case the accessors are the answer, or it did not, in which case there is no identifier to read fields from.

The parts that are easy to get wrong

No address is ever built from an index this package cannot name. That would name a document nobody wrote, and a receiver would record having accepted terms that do not exist. An index of zero and an index added after this release therefore give a caller the same answer, which is deliberate, since both say the identifier does not give the terms and the answer has to come from somewhere else.

The byte is read at the offset the match key ends at. The identifier type fixes the match key length and so the type says where the field starts, which is why _layout.py carries TERMS_LENGTH and no offset constant. A reader with one fixed offset reads a match key byte on a RANDOM identifier.

A payload with no byte after the match key reads as index 0. That is the same answer a zero byte gives, so absence and zero mean the same thing and no presence flag exists to tell them apart. No field before the Terms moves, because the byte sits after the match key and every offset before it is fixed by the header and the type.

An identifier of the RESERVED type reads as index 0. No match key length is defined for that type, so every byte after the header is its match key and none is left to read. That is the right answer and not a defect, and no special case is written for it. There is a comment saying so at _read_terms_index and a test pinning it.

No address does not mean the identifier is unrestricted. It says only that the identifier does not carry the answer, so the answer has to come from the surrounding protocol, being the Terms Document Locator in an OpenRTB request or whatever else it provides. Where an identifier may go is a separate question fod_id.usage answers.

The Terms sits before the creator context section. An issuer writing a context section writes the Terms byte before it, and this package holds no upper bound on a payload, so a longer one is accepted and the same fields are read.

The version is read on its own bits. A reader masking the wrong bits would refuse a version 0 identifier of some usages, or let a later version through, so there is a test over every combination of the usage and type bits.

Files changed

File	What it holds
src/fiftyone_pipeline_did/_terms.py	The named value the specification gives for each index, from_index and url. Renamed from terms.py and no longer exported, since the package turns the index into an address.
src/fiftyone_pipeline_did/fod_id.py	Reads the version before any field, refuses a version it does not know, adds UNSUPPORTED_PAYLOAD_VERSION, and answers terms with the address read at the offset the match key ends at
src/fiftyone_pipeline_did/_layout.py	TERMS_LENGTH, ABSENT_TERMS_INDEX and SUPPORTED_PAYLOAD_VERSION, private as the other layout constants are
src/fiftyone_pipeline_did/__init__.py	No longer exports the named value
tests/test_fodid.py	The fixtures and the tests below
readme.md	The terms section, a payload version section and the payload layout summary
examples/fodid_example.py	The sample payload carries both new fields and the example prints the address

The test fixtures write what an issuer writes

canonical_payload() and canonical_random_payload() are the creating side, so they now carry the payload version 0 in the flags byte and the Terms byte their usage calls for, being index 1 for the personalized marketing canonical payload and index 0 for the non-marketing Random one. payload_ending_at_match_key() and random_payload_ending_at_match_key() are the fixtures for a payload that carries no Terms byte, which a reader takes as index 0 and which no issuer would write.

Tests

FodIdTermsTests, building the payload byte by byte the way the existing tests do:

• test_payload_ending_at_the_match_key_has_no_address (both match key lengths)
• test_an_absent_byte_and_a_zero_byte_read_the_same
• test_index_one_answers_with_the_model_terms_address (both match key lengths)
• test_the_address_is_the_versioned_document
• test_an_unknown_index_has_no_address
• test_an_unknown_index_answers_as_no_terms_stated_does
• test_every_index_the_package_does_not_know_has_no_address
• test_terms_is_read_before_a_creator_context_section (both match key lengths)
• test_a_context_section_alone_does_not_state_terms
• test_reserved_type_takes_every_byte_as_its_match_key
• test_every_reader_reads_the_same_terms
• test_base64_roundtrip_preserves_the_terms
• test_terms_does_not_change_how_the_other_fields_read
• test_the_payload_rules_are_unchanged_by_the_terms

FodIdVersionTests:

• test_version_zero_reads_every_field
• test_the_layout_names_the_version_this_package_reads
• test_an_unassigned_version_is_refused
• test_the_raising_readers_name_the_version
• test_the_version_is_read_apart_from_the_usage_and_type_bits

TermsTests, over the vocabulary alone, including test_the_named_value_is_not_part_of_the_package_surface.

The terms table

The address, the index and the name are one table, so adding a terms document is one row and not a search for every place a number was written down. That is the point of the byte being an index rather than a version number, and the first version of this change did not honour it, spreading the same fact over three, being the member, an index to member dict and an address dict. The address lookup also subscripted its dict directly, so a member missing from it would raise KeyError at the caller rather than answer with no address places that could disagree.

Each member now carries its index and its address, and the index to member map is built from the members rather than written out again, so there is nothing left to raise.

A test that claimed a member without an address entry would raise there described the old design and could not fail under the new one. It is replaced by two that check what now holds, being that every member reads back from its own index and that a member naming a document has an https address, and that no two members share an index.

What was run and what it reported

Python 3.14.5 on Windows, with the owid-python submodule on the path and the OWID source copied in by ci/copy-owid-source.ps1, which is what setup.ps1 does.

python -m pytest tests -q
172 passed, 2 skipped, 29 subtests passed

The 2 skipped are the live cloud tests in test_did_client_live.py, which need a resource key.

The offline example runs and prints the address:

python examples/fodid_example.py
  Match key : 202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f
  Terms     : https://m4ow.uk/mtm/2.txt

tox itself could not be installed on the machine used, because the virtualenv wheel would not download. The command above is the one tox.ini runs, executed directly against the same environment.

The tests were checked against a wrong reader

A green suite proves nothing on its own, so two deliberate faults were put into the reader to see which tests caught them.

Reading the byte at the end of the payload instead of at the offset the match key ends at:

3 failed, 82 passed
FAILED test_a_context_section_alone_does_not_state_terms
SUBFAILED[probabilistic] test_terms_is_read_before_a_creator_context_section
SUBFAILED[random] test_terms_is_read_before_a_creator_context_section

Reading an index the package does not know as index 0 with an address:

8 failed, 80 passed
FAILED test_an_unknown_index_has_no_address
FAILED test_from_index_names_every_other_index_unknown
SUBFAILED(index=2,3,127,128,255) test_every_index_the_package_does_not_know_has_no_address

Both faults were then removed and the suite returned to green.

Outstanding

• The other five 51Did packages carry the same change. All six were run over thirteen identifiers built by hand outside every package and gave the same terms and match key on every one, refusing versions 1, 2 and 3.
• The cloud has to write both fields before any real identifier carries them. This is the reader only.