Add access-log summary field groups (Summary Metrics) - #2256
Open
sandersonb wants to merge 3 commits into
Open
sandersonb wants to merge 3 commits into
sandersonb wants to merge 3 commits into
Conversation
sandersonb
force-pushed
the
feature-ghostwire-web-requests-summary
branch
2 times, most recently
from
September 15, 2026 23:18
6dce30f to
9083129
Compare
…ary-metrics) Model aggregated access-log summaries (CDN weblog, mobile, and thin-app requests) for delivery to Customer Journey Analytics as summary data, on the XDM Summary Metrics class. Geo/device/environment shapes reuse the standard datatypes so they match what the Edge orchestration layer emits on interactive ExperienceEvents. New field groups (intendedToExtend summarymetrics): - access-log-environment-summary: placeContext/device/environment via the standard placecontext, device, and environment datatypes. - access-log-bot-detection-summary: bot classification as three mutually exclusive, additive request counts (botRequests / uncertainRequests / notBotRequests) derived upstream from customer-chosen thresholds, plus an optional bot name. Counts re-aggregate correctly at any CJA grain. Reference composite: - schemas/access-log/access-log-summary: Summary Metrics class + the above field groups (and the existing, unmodified CDN Requests Summary field group), with an end-to-end example that doubles as the Data Prep mapping target. Design notes: - Normalized device type is carried in the standard device.type soft-enum field (mobile / tablet / desktop), not a bespoke field group. - Aggregation grain is set per flow by the upstream GROUP BY; numeric fields are never group-by dimensions, so the schema stays a permissive superset. - Client IP is excluded from summary output for PII; enforced at the Data Prep mapping layer. - The existing CDN Requests Summary field group is adopted unmodified; bot name lives only in botDetection.name (cdn.botType left for CDN-declared labels). - meta:titleId / meta:descriptionId omitted; generated by the localization pipeline. All schemas and examples pass npm test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sandersonb
force-pushed
the
feature-ghostwire-web-requests-summary
branch
from
September 16, 2026 00:09
9083129 to
dca0963
Compare
Add access-log-consent-summary (intendedToExtend summarymetrics) so consented vs non-consented traffic can be broken down in the summary. Reuses the Adobe consent standard at the standard path xdm:consents.xdm:collect via a $ref to the consent-field datatype; xdm:val carries the standard choice values (y/n/p/u/dy/dn) and is intended as a GROUP BY dimension. The dy/dn (default opt-in / default opt-out) values represent the customer's configured default consent, applied by the Ghostwire pipeline to requests with no explicit consent signal (CMP or WebSDK opt-out cookie) — letting defaulted-in traffic be distinguished from explicitly consented traffic. Scoped to data collection (collect) for now; share can be added later. Added to the access-log-summary composite schema and example. All schemas and examples pass npm test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sandersonb
marked this pull request as ready for review
September 16, 2026 19:00
Address Chris Luby's review: make the examples reflect a real CDN summary grain
rather than every property the reused ExperienceEvent datatypes allow.
- Trim examples to the default group-by dimension set + aggregate metrics; drop
properties not expected out of the box (city/postalCode/lat-long, raw device
id, screen dims, per-request timezone, etc.).
- Add environment.type (browser/application/iot) to the default set — the
web / mobile-app / thin-app channel dimension.
- cdn.url stays in every joinable row (CJA join key into the customer's rich
web dataset).
- Two composite examples for two grains:
- example.1: a 15-minute bucket at the default granularity (human traffic,
mixed bot bands, no matched bot name).
- example.2: a daily rollup with fewer dimensions (named-bot total).
- Remove city from all examples (country/state is the default geo granularity;
city is a later optional group-by set).
All schemas and examples pass npm test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add access-log summary field groups (Summary Metrics)
Resolves #2258
Breaking changes: None. This change is purely additive — three new field groups plus a reference composite schema; no existing component is modified.
Summary
Adds field groups that extend the XDM Summary Metrics class to model aggregated access-log summaries — CDN weblog, mobile, and thin-app requests — for delivery to Customer Journey Analytics as summary data. Covers geo/device/environment, bot classification, and data-collection consent.
Geo/device/environment shapes reuse the standard
placecontext,device, andenvironmentdatatypes at their canonical paths, so the summary model matches what the Edge orchestration layer emits on interactive ExperienceEvents — one consistent enrichment model across interactive and summary datasets.The existing CDN Requests Summary field group is adopted as-is (unmodified) for transport dimensions and the request count.
New field groups (
meta:intendedToExtend: summarymetrics,meta:status: experimental)access-log-environment-summary—xdm:placeContext,xdm:device,xdm:environmentvia$refto the standard datatypes (geo, IANA timezone, IP-local offset; device hardware; ISP/carrier/domain/connection & proxy type; OS and browser details). The normalized device type is carried in the standarddevice.typesoft-enum field (mobile/tablet/desktop).access-log-bot-detection-summary— bot classification as three mutually exclusive, additive request counts:botRequests/uncertainRequests/notBotRequests, derived upstream from customer-chosen thresholds (equal thresholds collapse to a bot / not-bot split), plus an optionalname. Counts re-aggregate correctly at any CJA grain.access-log-consent-summary— data-collection consent at the standard pathxdm:consents.xdm:collectvia$refto the standardconsent-fielddatatype.xdm:valuses the Adobe consent choice values (y/n/p/u/dy/dn), used as a GROUP BY dimension so consented vs non-consented traffic can be broken down.dy/dn(default opt-in / default opt-out) represent the customer's configured default consent, applied to requests with no explicit consent signal.Reference composite schema
schemas/access-log/access-log-summary(+ example) — composes the Summary Metrics class with the three field groups above and the existing (unmodified) CDN Requests Summary field group; the example is one fully-populated summary record and doubles as the Data Prep mapping target. It is a reference/example schema, not an attachable field group.Design notes
experienceevent-*field groups can't be reused directly (theyintendedToExtendExperienceEvent), but the underlying datatypes (placecontext,device,environment,consent-field) are class-agnostic and are reused here at their standard paths.choice-valueenum already includesdy/dnfor default consent. The per-IMSOrg default is applied in the Ghostwire mapping layer, not the schema.environmentdatatype still declaresipV4/ipV6(fields can't be stripped from a$ref'd standard datatype); exclusion is enforced at the Data Prep mapping layer by never mapping source IP into those paths.device.type) accept new values without a schema change, matching existing XDM practice.meta:titleId/meta:descriptionIdare omitted on new fields, matching current contribution practice; they are generated by the localization pipeline.Testing
npm test— all passing (schema meta-validity, example presence, and example validity for every new schema, including the composite reference schema).Deferred / follow-ups
requestBytes/responseBytes) were considered as additions to CDN Requests Summary but are deferred to avoid modifying another team's field group; may be revisited later (would need that team's sign-off at that time).collect) only;sharecan be added if data-sharing consent is needed.device.typeis the intended low-cardinality dimension); revisit only if both raw and normalized are needed in the same row.