Skip to content

Add access-log summary field groups (Summary Metrics) - #2256

Open
sandersonb wants to merge 3 commits into
adobe:masterfrom
sandersonb:feature-ghostwire-web-requests-summary
Open

sandersonb wants to merge 3 commits into
adobe:masterfrom
sandersonb:feature-ghostwire-web-requests-summary

Conversation

@sandersonb

@sandersonb sandersonb commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Add access-log summary field groups (Summary Metrics)

Resolves #2258

Breaking changes: None. This change is purely additive — three new field groups plus a reference composite schema; no existing component is modified.

Summary

Adds field groups that extend the XDM Summary Metrics class to model aggregated access-log summaries — CDN weblog, mobile, and thin-app requests — for delivery to Customer Journey Analytics as summary data. Covers geo/device/environment, bot classification, and data-collection consent.

Geo/device/environment shapes reuse the standard placecontext, device, and environment datatypes at their canonical paths, so the summary model matches what the Edge orchestration layer emits on interactive ExperienceEvents — one consistent enrichment model across interactive and summary datasets.

The existing CDN Requests Summary field group is adopted as-is (unmodified) for transport dimensions and the request count.

New field groups (meta:intendedToExtend: summarymetrics, meta:status: experimental)

  • access-log-environment-summaryxdm:placeContext, xdm:device, xdm:environment via $ref to the standard datatypes (geo, IANA timezone, IP-local offset; device hardware; ISP/carrier/domain/connection & proxy type; OS and browser details). The normalized device type is carried in the standard device.type soft-enum field (mobile/tablet/desktop).
  • access-log-bot-detection-summary — bot classification as three mutually exclusive, additive request counts: botRequests / uncertainRequests / notBotRequests, derived upstream from customer-chosen thresholds (equal thresholds collapse to a bot / not-bot split), plus an optional name. Counts re-aggregate correctly at any CJA grain.
  • access-log-consent-summary — data-collection consent at the standard path xdm:consents.xdm:collect via $ref to the standard consent-field datatype. xdm:val uses the Adobe consent choice values (y/n/p/u/dy/dn), used as a GROUP BY dimension so consented vs non-consented traffic can be broken down. dy/dn (default opt-in / default opt-out) represent the customer's configured default consent, applied to requests with no explicit consent signal.

Reference composite schema

  • schemas/access-log/access-log-summary (+ example) — composes the Summary Metrics class with the three field groups above and the existing (unmodified) CDN Requests Summary field group; the example is one fully-populated summary record and doubles as the Data Prep mapping target. It is a reference/example schema, not an attachable field group.

Design notes

  • Class fit. Summary Metrics is used because the source is pre-aggregated (GROUP BY dimensions + a request count). Aggregation grain is set per flow by the upstream GROUP BY; numeric/continuous fields are never group-by dimensions, so the schema is a permissive superset and the grain lives upstream, not in the schema.
  • Reuse over redefinition. The interactive experienceevent-* field groups can't be reused directly (they intendedToExtend ExperienceEvent), but the underlying datatypes (placecontext, device, environment, consent-field) are class-agnostic and are reused here at their standard paths.
  • Bot counts, not scores. A raw or pre-averaged bot score can't be re-aggregated across summary rows; mutually exclusive counts can. Thresholds are applied upstream.
  • Consent reuses the standard. No new consent vocabulary — the standard choice-value enum already includes dy/dn for default consent. The per-IMSOrg default is applied in the Ghostwire mapping layer, not the schema.
  • PII / IP. Client IP is intentionally excluded from summary output. The shared environment datatype still declares ipV4/ipV6 (fields can't be stripped from a $ref'd standard datatype); exclusion is enforced at the Data Prep mapping layer by never mapping source IP into those paths.
  • Soft enums (device.type) accept new values without a schema change, matching existing XDM practice.
  • meta:titleId/meta:descriptionId are omitted on new fields, matching current contribution practice; they are generated by the localization pipeline.

Testing

npm test — all passing (schema meta-validity, example presence, and example validity for every new schema, including the composite reference schema).

Deferred / follow-ups

  • Byte metrics (requestBytes/responseBytes) were considered as additions to CDN Requests Summary but are deferred to avoid modifying another team's field group; may be revisited later (would need that team's sign-off at that time).
  • Consent scope is data collection (collect) only; share can be added if data-sharing consent is needed.
  • Raw device hardware type is not carried in the summary (the normalized device.type is the intended low-cardinality dimension); revisit only if both raw and normalized are needed in the same row.

@sandersonb
sandersonb force-pushed the feature-ghostwire-web-requests-summary branch 2 times, most recently from 6dce30f to 9083129 Compare September 15, 2026 23:18
…ary-metrics)

Model aggregated access-log summaries (CDN weblog, mobile, and thin-app
requests) for delivery to Customer Journey Analytics as summary data, on the XDM
Summary Metrics class. Geo/device/environment shapes reuse the standard
datatypes so they match what the Edge orchestration layer emits on interactive
ExperienceEvents.

New field groups (intendedToExtend summarymetrics):
- access-log-environment-summary: placeContext/device/environment via the
  standard placecontext, device, and environment datatypes.
- access-log-bot-detection-summary: bot classification as three mutually
  exclusive, additive request counts (botRequests / uncertainRequests /
  notBotRequests) derived upstream from customer-chosen thresholds, plus an
  optional bot name. Counts re-aggregate correctly at any CJA grain.

Reference composite:
- schemas/access-log/access-log-summary: Summary Metrics class + the above field
  groups (and the existing, unmodified CDN Requests Summary field group), with
  an end-to-end example that doubles as the Data Prep mapping target.

Design notes:
- Normalized device type is carried in the standard device.type soft-enum field
  (mobile / tablet / desktop), not a bespoke field group.
- Aggregation grain is set per flow by the upstream GROUP BY; numeric fields are
  never group-by dimensions, so the schema stays a permissive superset.
- Client IP is excluded from summary output for PII; enforced at the Data Prep
  mapping layer.
- The existing CDN Requests Summary field group is adopted unmodified; bot name
  lives only in botDetection.name (cdn.botType left for CDN-declared labels).
- meta:titleId / meta:descriptionId omitted; generated by the localization
  pipeline.

All schemas and examples pass npm test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@sandersonb
sandersonb force-pushed the feature-ghostwire-web-requests-summary branch from 9083129 to dca0963 Compare September 16, 2026 00:09
Add access-log-consent-summary (intendedToExtend summarymetrics) so consented vs
non-consented traffic can be broken down in the summary. Reuses the Adobe consent
standard at the standard path xdm:consents.xdm:collect via a $ref to the
consent-field datatype; xdm:val carries the standard choice values
(y/n/p/u/dy/dn) and is intended as a GROUP BY dimension.

The dy/dn (default opt-in / default opt-out) values represent the customer's
configured default consent, applied by the Ghostwire pipeline to requests with
no explicit consent signal (CMP or WebSDK opt-out cookie) — letting defaulted-in
traffic be distinguished from explicitly consented traffic. Scoped to data
collection (collect) for now; share can be added later.

Added to the access-log-summary composite schema and example. All schemas and
examples pass npm test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@sandersonb sandersonb changed the title Add web-requests summary field groups and extend CDN Requests Summary (summary-metrics) Add access-log summary field groups (Summary Metrics) Sep 16, 2026
@sandersonb
sandersonb marked this pull request as ready for review September 16, 2026 19:00
Address Chris Luby's review: make the examples reflect a real CDN summary grain
rather than every property the reused ExperienceEvent datatypes allow.

- Trim examples to the default group-by dimension set + aggregate metrics; drop
  properties not expected out of the box (city/postalCode/lat-long, raw device
  id, screen dims, per-request timezone, etc.).
- Add environment.type (browser/application/iot) to the default set — the
  web / mobile-app / thin-app channel dimension.
- cdn.url stays in every joinable row (CJA join key into the customer's rich
  web dataset).
- Two composite examples for two grains:
  - example.1: a 15-minute bucket at the default granularity (human traffic,
    mixed bot bands, no matched bot name).
  - example.2: a daily rollup with fewer dimensions (named-bot total).
- Remove city from all examples (country/state is the default geo granularity;
  city is a later optional group-by set).

All schemas and examples pass npm test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Need to add Summary Metrics field groups to expose CDN Log Data & Classifications

1 participant