diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index 6e0d39842b72..83c4567bfaa4 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -1,6 +1,7 @@ ## 11.1.0 #### Project +* Update the e2e mock sender's OTLP proto to v1.11.1. A current OpenTelemetry Collector adds fields the old copy lacked, such as a metric's `metadata`, and the sender's strict JSON parser refused a recording that carries them. The Windows e2e now also replays such a recording, of windows_exporter `0.31.8` through OpenTelemetry Collector `0.158.0` taken on a GitHub-hosted Windows runner, and asserts its memory metrics. * Replace the GenAI e2e cases' Spring AI application with an in-repo `e2e-spring-ai-service` module pinned to the released Spring AI 2.0.1, and move the mock LLM endpoints out of `e2e-service-provider` into a dedicated `e2e-mock-llm-server` module that all three GenAI cases share. The Spring AI application was previously built at test time by cloning `spring-projects/spring-ai-examples` and running Maven inside the image build, which resolved `spring-ai:2.0.0-SNAPSHOT`: a line that has since been abandoned, whose surviving builds all predate the 2.0.0 GA by a day and whose older builds have been pruned, so the fixture was pinned to nothing released and could no longer be rebuilt from the same bytes. Both modules are built by the existing e2e reactor and published to `ghcr.io/apache/skywalking` alongside the other e2e service images; `e2e-spring-ai-service` needs Java 17 and Spring Boot 4, so it is gated behind a new `jdk-17` profile in the e2e reactor and the jobs that build it now pin JDK 17 explicitly; `make -C test docker` refuses up front on an older JDK rather than packaging a missing or stale jar, since `clean` never reaches a module the profile excluded and compose would turn the absent jar into a directory. #### OAP Server @@ -28,6 +29,8 @@ * Bring the TraceQL module closer to the Tempo API. Every datasource lists only the spans that matched a search in each span set, capped at `spss`, fills `serviceStats`, filters `span.http.status_code`, honours `scope` and the per-scope `limit` on the tag endpoints, whose `intrinsic` scope lists what each datasource can filter, accepts Tempo's unscoped `.key` tag names on the tag-value endpoints, and answers a tag-name or tag-value lookup filtered by `q` from a sample of the newest 50 matching traces, so Grafana's dependent dropdowns narrow down; the `status` and `kind` dropdowns always list their full enum. The `/otlp` datasource also accepts Tempo's `span:name`, `span:kind`, `span:status` and `span:duration` spellings. `/api/metrics/query_range` and `/api/metrics/query` answer `501 Not Implemented` with an error body and accept Tempo's `q` parameter, instead of `200` with a string Grafana cannot parse. * Refuse unsupported TraceQL with `400` instead of dropping the predicate and answering with unfiltered traces: syntax errors (`||` inside a spanset, regex, an attribute without a scope or leading dot), `!=` and other unsupported operators, negation, attribute-existence checks, multiple spansets, unknown intrinsics and values, and conditions a datasource cannot filter (`kind` on Zipkin and SkyWalking, `resource.instance` on Zipkin, `resource.remote.service` and `status = unset` on SkyWalking, `resource.instance` or `name` without a service on SkyWalking). The deprecated `tags` search parameter is parsed as logfmt on every datasource and goes through the same mapping and refusals. `kind = server` and `status = error` accept the bare keyword as Tempo does. Every search-result span carries a `status` attribute (`error`, `ok`, `unset`) next to `service.name` and `span.kind`, so a trace list can show failures. A trace the storage matched but none of whose spans satisfies every condition is left out instead of listed with every span, and the `/otlp` datasource compares names the way the receiver indexed them and keeps the `resource.` and `span.` scopes apart in its tag index, so `span.env` no longer matches a resource attribute. `duration >` and `<` are strict at microsecond precision. (#14093) * Support the OpenTelemetry Collector `hostmetrics` receiver as an alternative source for Linux and Windows host monitoring. `vm.yaml` and `windows.yaml` map it to the same `meter_vm_*` / `meter_win_*` metrics as node-exporter and windows_exporter, whose metrics keep their meaning, and add metrics only hostmetrics provides: CPU core count and normalized CPU usage, plus CPU load, file-system usage, system handle count and pagefile usage on Windows. node-exporter metrics without a matching hostmetrics source (`tcp_alloc`, `sockets_used`, `udp_inuse`, `filefd_allocated`) stay node-exporter only. The new `process-hostmetrics-linux` and `process-hostmetrics-windows` rules, enabled by default, report each process name as an instance of its host (`mp_process_linux_*` / `mp_process_windows_*`: process count, threads, CPU, resident memory, open handles and oldest process uptime). Reference Collector configurations for both systems are under `docs/en/setup/backend/`; they set the `job_name` and normalized `process_name` labels the rules route and group on, and sum same-named processes before export. +* Windows monitoring through windows_exporter now requires windows_exporter `0.29.0` or later. windows_exporter `0.31.0` removed the `cs` collector and the deprecated `os` memory metrics (`windows_cs_physical_memory_bytes`, `windows_os_physical_memory_free_bytes`, `windows_os_virtual_memory_bytes`, `windows_os_virtual_memory_free_bytes`) that `windows.yaml` read, so on `0.31.0`+ `meter_win_memory_total` / `_available` / `_used` and the `meter_win_memory_virtual_memory_*` commit metrics were empty. The rules now read their replacements from the `memory` collector (`windows_memory_physical_total_bytes`, `windows_memory_physical_free_bytes`, `windows_memory_commit_limit`, `windows_memory_committed_bytes`), all of which windows_exporter provides since `0.29.0`; on an older windows_exporter these memory metrics stay empty until it is upgraded. Also fix `meter_win_cpu_total_percentage` counting overlapping Windows CPU modes twice: windows_exporter's `privileged` already includes `interrupt` and `dpc` time, and OpenTelemetry hostmetrics' `system` already includes `interrupt` time, so busy CPU is now `user` + `privileged` or `user` + `system`. +* Fix the OpenTelemetry receiver creating no Linux or Windows host service from an OpenTelemetry Collector `0.127.0` or later. The Collector's Prometheus receiver sends a scraped target's host only as `server.address` from that release, no longer as `net.host.name`, and `vm.yaml` and `windows.yaml` name a host by `node_identifier_host_name`, which the receiver took only from `net.host.name` or `host.name`. It now falls back to `server.address` when neither is present. * AI agent conversations adopt AI Sessionizer `130601c`. Calls to MCP servers: the `execution` kind the Sessionizer's Claude Code plugin writes, `streams//execution--.sd`, is stored like any other file; the document lists every `execution/1` record under `tool_executions`, joined to its step by tool-use id and kept once by its own id, a tool step names its records under `executions`, and a call to an MCP server carries `mcp_server` and `mcp_tool` in its `attrs`. The new `otel-rules/ai-agent/mcp_endpoint.yaml` gives one endpoint per MCP server and tool, `/`, under the agent's service, with `meter_ai_agent_mcp_calls`, `meter_ai_agent_mcp_calls_by_outcome` and `meter_ai_agent_mcp_duration` from the Sessionizer's `agent.mcp.calls` and `agent.mcp.duration`. The token rules read `agent.token.usage`, the Sessionizer's name for the metric. The document follows the Sessionizer's: an `llm.call` takes its provider bodies from the round's `provider_bodies` attribute and the session its count from `provider_bodies_landed`, and a round whose bodies do not read or point past its range is refused; talks are in the order they began across streams; only a `child` stream's talk is a child's, not an `auxiliary` one's; a stream's `opened_by`, the relations and a step's edges are in the order they happened, by record position inside one stream or workflow run and by time across them, never by id; change and execution records of one instant are in the order they were read; a record is read by the fields its format lists, and one of another shape is skipped; a round with a frame field of another type, or a negative sequence, row or round number, does not read; a record time is RFC 3339 as Session Data defines it, to the second, and anything else is no time; record times compare as instants. * Fix the Elasticsearch `BulkProcessor` treating a bulk write as fully successful whenever the overall HTTP status was 200, even when Elasticsearch rejected individual items in the same response, for example a 429 `cluster_block_exception` when an index is switched to `read_only_allow_delete` by the flood-stage disk watermark. The response body's `errors` flag and per-item `status`/`error` are now decoded and checked on every 200 response; a rejected request's future now completes exceptionally instead of being treated as a success, while the rest of the batch still completes normally. One ERROR line per `_bulk` request summarizes the rejections grouped by status code and error type, never a document id or the raw ES error reason. A response whose `items` array is shorter than the request (a malformed or truncated response) also fails those requests instead of silently treating them as succeeded. This also fixes a related batching bug where, once a flush was split into multiple `_bulk` HTTP requests by `batchOfBytes`, the completion of one chunk's request completed or failed every request in the whole flush instead of just its own chunk, and a bug where requests whose bulk failed to even build (e.g. an encoding error) were left with a future that never completed, which could block a persistence round forever. @@ -35,6 +38,7 @@ * Add a Virtual GenAI evaluation-record page and evaluation-score chart in Horizon UI, so operators can inspect evaluation result, level, reason, judge model, timestamp, trace linkage, and the `gen_ai_model_evaluation_score_ppm` trend for evaluated records. #### Documentation +* The Windows monitoring doc's example OpenTelemetry Collector configuration uses the `debug` exporter. The `logging` exporter it used was removed from the Collector, so the example no longer loaded. * Document the BanyanDB trace tail sampling metrics in the BanyanDB self-observability dashboard catalog, and point the "Operating it" section of the trace tail sampling guide at them — the OAP-collected metrics show what the sampler plugins *proposed* next to what storage *committed*, which the data node's raw metrics endpoint alone does not. * Rename Envoy AI Gateway to Agent Router in the monitoring docs, following the project's move to the Agentic AI Foundation. Only the prose changes: the `ENVOY_AI_GATEWAY` layer, the `job_name=envoy-ai-gateway` routing tag, the `envoy-ai-gateway` rule files and the `meter_envoy_ai_gw_` metric names are kept, as Agent Router itself kept its deployed names and the telemetry it emits. diff --git a/docs/en/setup/backend/backend-vm-monitoring.md b/docs/en/setup/backend/backend-vm-monitoring.md index 621548c08e90..3e6b39312c9d 100644 --- a/docs/en/setup/backend/backend-vm-monitoring.md +++ b/docs/en/setup/backend/backend-vm-monitoring.md @@ -59,7 +59,7 @@ Likewise, `meter_vm_tcp_alloc`, `meter_vm_sockets_used`, and `meter_vm_udp_inuse | TCP Established / Close-Wait | count | `meter_vm_tcp_curr_estab` | TCP connections in ESTABLISHED or CLOSE-WAIT state | Yes | Yes | Yes | | TCP Time Wait | count | `meter_vm_tcp_tw` | TCP connections in TIME-WAIT state | Yes | Yes | Yes | | TCP Allocated | count | `meter_vm_tcp_alloc` | Allocated TCP sockets | Yes | No | Yes | -| Sockets Used | count | `meter_vm_sockets_used` | Kernel sockets currently in use | Yes | No | Yes | +| Sockets Used | count | `meter_vm_sockets_used` | Kernel sockets currently in use | Yes | No | — | | UDP In Use | count | `meter_vm_udp_inuse` | UDP sockets currently in use | Yes | No | Yes | | Filefd Allocated | count | `meter_vm_filefd_allocated` | Host-level allocated file descriptors from Linux `/proc/sys/fs/file-nr` | Yes | No | — | diff --git a/docs/en/setup/backend/backend-win-monitoring.md b/docs/en/setup/backend/backend-win-monitoring.md index c4c901c5c512..8c432712d9e0 100644 --- a/docs/en/setup/backend/backend-win-monitoring.md +++ b/docs/en/setup/backend/backend-win-monitoring.md @@ -9,10 +9,14 @@ Windows entity as a `Service` in OAP and on the `Layer: OS_WINDOWS`. 3. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results. ## Setup **For OpenTelemetry receiver:** -1. Setup [Prometheus windows_exporter](https://github.com/prometheus-community/windows_exporter). +1. Setup [Prometheus windows_exporter](https://github.com/prometheus-community/windows_exporter) `0.29.0` or later. + - Keep its default `cpu`, `memory`, `logical_disk` and `net` collectors enabled. + - Memory metrics are read from the `memory` collector. windows_exporter `0.31.0` removed the `cs` collector and the `os` memory metrics that older SkyWalking releases read. 2. Setup [OpenTelemetry Collector ](https://opentelemetry.io/docs/collector/). This is an example for OpenTelemetry Collector configuration [otel-collector-config.yaml](../../../../test/e2e-v2/cases/win/prometheus-windows_exporter/otel-collector-config.yaml). 3. Config SkyWalking [OpenTelemetry receiver](opentelemetry-receiver.md). +Each Windows host is a service named after its host: the `host.name` or `net.host.name` resource attribute when the Collector sends one, otherwise `server.address`, where the Collector's Prometheus receiver puts the scraped target's host from Collector `0.127.0`. + ### Native OpenTelemetry hostmetrics (expanded alternative) SkyWalking can also receive Windows host and process metrics directly from OpenTelemetry Collector Contrib, without Prometheus windows_exporter. @@ -29,8 +33,8 @@ The `OTel hostmetrics` column below refers to the complete OpenTelemetry Collect | Monitoring Panel | Unit | Metric Name | Description | windows_exporter | OTel hostmetrics | |---|---|---|---|:---:|:---:| -| CPU Usage | % | `meter_win_cpu_total_percentage` | Total CPU usage across all logical CPUs | Yes | Yes | -| CPU Average Used | % | `meter_win_cpu_average_used` | CPU usage by mode/state | Yes | Yes | +| CPU Usage | % | `meter_win_cpu_total_percentage` | Busy CPU across all logical CPUs: `user` + `privileged` (windows_exporter) or `user` + `system` (OTel) | Yes | Yes | +| CPU Average Used | % | `meter_win_cpu_average_used` | CPU usage by mode/state, summed across logical CPUs. Windows modes overlap: `privileged` (windows_exporter) includes `interrupt` and `dpc` time, and `system` (OTel) includes `interrupt` time | Yes | Yes | | CPU Cores | count | `meter_win_cpu_cores_num` | Number of logical CPUs | — | Yes | | Normalized CPU Usage | % | `meter_win_cpu_norm_percentage` | CPU usage normalized by the number of logical CPUs | — | Yes | | CPU Load | | `meter_win_cpu_load1`
`meter_win_cpu_load5`
`meter_win_cpu_load15` | CPU load metrics exposed by the Collector hostmetrics load scraper | — | Yes | diff --git a/docs/en/setup/backend/opentelemetry-receiver.md b/docs/en/setup/backend/opentelemetry-receiver.md index 90c022be8588..3c806f3fe37a 100644 --- a/docs/en/setup/backend/opentelemetry-receiver.md +++ b/docs/en/setup/backend/opentelemetry-receiver.md @@ -23,8 +23,8 @@ receiver-otel: ``` The receiver adds label with key `node_identifier_host_name` to the collected data samples, -and its value is from `net.host.name` (or `host.name` for some OTLP versions) resource attributes defined in OpenTelemetry proto, -for identification of the metric data. +and its value is from the `net.host.name` or `host.name` resource attribute, or, when neither is present, +`server.address`, for identification of the metric data. **Label name conversion:** Dots (`.`) in attribute key names are converted to underscores (`_`) for both resource attributes and data point (metric-level) attributes. For example, `gen_ai.token.type` becomes @@ -40,11 +40,12 @@ in the resource attributes, the fallback is skipped. | `service.name` | `job_name` | The [OTel Collector Prometheus Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md) automatically converts the Prometheus `job` label to `service.name`. This fallback ensures it is available as `job_name` for MAL rule filtering. | | `net.host.name` | `node_identifier_host_name` | Legacy: used by VM/Windows MAL rules | | `host.name` | `node_identifier_host_name` | Legacy: used by VM/Windows MAL rules | +| `server.address` | `node_identifier_host_name` | Only when neither `net.host.name` nor `host.name` is present. The OTel Collector Prometheus Receiver sends a scraped target's host only as `server.address` from Collector `0.127.0`. | When `job_name` is set explicitly in `OTEL_RESOURCE_ATTRIBUTES` (e.g., `job_name=envoy-ai-gateway` for [Agent Router](backend-envoy-ai-gateway-monitoring.md)), it takes precedence and the `service.name` fallback is skipped. -**Note:** The `net.host.name` and `host.name` mappings are legacy. New integrations should use +**Note:** The `net.host.name`, `host.name` and `server.address` mappings are legacy. New integrations should use the natural dot-to-underscore conversion (e.g., `host.name` → `host_name` in MAL rules). **Points of one request are analysed a minute at a time**, oldest minute first. A MAL rule folds every sample of diff --git a/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/process-hostmetrics-linux.data.yaml b/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/process-hostmetrics-linux.data.yaml index 6f3457dcee1b..d2a4fcc0dbee 100644 --- a/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/process-hostmetrics-linux.data.yaml +++ b/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/process-hostmetrics-linux.data.yaml @@ -62,11 +62,11 @@ input: value: 3000.0 process_memory_utilization: - labels: {node_identifier_host_name: test-host, process_name: java} - value: 0.01 + value: 1.0 - labels: {node_identifier_host_name: test-host, process_name: java} - value: 0.02 + value: 2.0 - labels: {node_identifier_host_name: test-host, process_name: java} - value: 0.03 + value: 3.0 process_open_handles: - labels: {node_identifier_host_name: test-host, process_name: java} value: 5.0 @@ -110,7 +110,7 @@ expected: mp_process_linux_memory_resident_percent: samples: - labels: {node_identifier_host_name: test-host, process_name: java} - value: 6.0 + value: 600.0 mp_process_linux_open_handles: samples: - labels: {node_identifier_host_name: test-host, process_name: java} diff --git a/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/process-hostmetrics-windows.data.yaml b/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/process-hostmetrics-windows.data.yaml index 62864d7c7e79..ed5158ec98ab 100644 --- a/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/process-hostmetrics-windows.data.yaml +++ b/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/process-hostmetrics-windows.data.yaml @@ -62,11 +62,11 @@ input: value: 3000.0 process_memory_utilization: - labels: {node_identifier_host_name: test-host, process_name: svchost.exe} - value: 0.01 + value: 1.0 - labels: {node_identifier_host_name: test-host, process_name: svchost.exe} - value: 0.02 + value: 2.0 - labels: {node_identifier_host_name: test-host, process_name: svchost.exe} - value: 0.03 + value: 3.0 process_open_handles: - labels: {node_identifier_host_name: test-host, process_name: svchost.exe} value: 5.0 @@ -110,7 +110,7 @@ expected: mp_process_windows_memory_resident_percent: samples: - labels: {node_identifier_host_name: test-host, process_name: svchost.exe} - value: 6.0 + value: 600.0 mp_process_windows_open_handles: samples: - labels: {node_identifier_host_name: test-host, process_name: svchost.exe} diff --git a/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/windows.data.yaml b/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/windows.data.yaml index f90570bec591..f373fcdf4690 100644 --- a/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/windows.data.yaml +++ b/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/windows.data.yaml @@ -20,6 +20,22 @@ input: node_identifier_host_name: test-host mode: user value: 100.0 + - labels: + node_identifier_host_name: test-host + mode: privileged + value: 40.0 + - labels: + node_identifier_host_name: test-host + mode: interrupt + value: 8.0 + - labels: + node_identifier_host_name: test-host + mode: dpc + value: 4.0 + - labels: + node_identifier_host_name: test-host + mode: idle + value: 200.0 # Native OpenTelemetry hostmetrics contract after Collector normalization. # Values are fixed for deterministic MAL assertions. system_cpu_logical_count: @@ -81,18 +97,18 @@ input: device: 'C:\\pagefile.sys' state: free value: 70.0 - windows_cs_physical_memory_bytes: + windows_memory_physical_total_bytes: - labels: value: 100.0 - windows_os_physical_memory_free_bytes: + windows_memory_physical_free_bytes: - labels: - value: 100.0 - windows_os_virtual_memory_free_bytes: + value: 40.0 + windows_memory_commit_limit: - labels: value: 100.0 - windows_os_virtual_memory_bytes: + windows_memory_committed_bytes: - labels: - value: 100.0 + value: 40.0 windows_logical_disk_read_bytes_total: - labels: node_identifier_host_name: test-host @@ -118,13 +134,21 @@ expected: samples: - labels: node_identifier_host_name: test-host - value: 3500.0 + value: 4300.0 meter_win_cpu_average_used: entities: - scope: SERVICE service: test-host layer: OS_WINDOWS samples: + - labels: + node_identifier_host_name: test-host + mode: idle + value: 6500.0 + - labels: + node_identifier_host_name: test-host + mode: interrupt + value: 400.0 - labels: node_identifier_host_name: test-host mode: user @@ -142,21 +166,21 @@ expected: layer: OS_WINDOWS samples: - labels: - value: 100.0 + value: 40.0 meter_win_memory_used: entities: - scope: SERVICE layer: OS_WINDOWS samples: - labels: - value: 0.0 + value: 60.0 meter_win_memory_virtual_memory_free: entities: - scope: SERVICE layer: OS_WINDOWS samples: - labels: - value: 100.0 + value: 60.0 meter_win_memory_virtual_memory_total: entities: - scope: SERVICE @@ -170,7 +194,7 @@ expected: layer: OS_WINDOWS samples: - labels: - value: -0.0 + value: 40.0 meter_win_disk_read: entities: - scope: SERVICE diff --git a/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OpenTelemetryMetricRequestProcessor.java b/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OpenTelemetryMetricRequestProcessor.java index 3a9c2919bcf2..1d2955c0df54 100644 --- a/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OpenTelemetryMetricRequestProcessor.java +++ b/oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OpenTelemetryMetricRequestProcessor.java @@ -105,6 +105,14 @@ public class OpenTelemetryMetricRequestProcessor implements Service, MalConverte // in resource attributes (e.g., Envoy AI Gateway), it takes precedence via putIfAbsent. .put("service.name", "job_name") .build(); + + /** + * Where a Prometheus target's host arrives from an OTel Collector v0.127.0 or later: its Prometheus receiver + * sends {@code server.address} and no longer {@code net.host.name}. Read only when no legacy host attribute + * gave {@code node_identifier_host_name}, so an explicit {@code host.name} still wins. + */ + private static final String SERVER_ADDRESS = "server.address"; + private static final String NODE_IDENTIFIER_HOST_NAME = "node_identifier_host_name"; /** * Active MAL converters, keyed by {@code ":"} so boot-time entries and * runtime-rule entries share one namespace. A runtime {@code /addOrUpdate} for a rule that @@ -145,20 +153,7 @@ public void processMetricsRequest(final ExportMetricsServiceRequest requests) { log.debug("Resource attributes: {}", request.getResource().getAttributesList()); } - // First pass: collect all resource attributes with dots replaced by underscores - final Map nodeLabels = new HashMap<>(); - for (final var it : request.getResource().getAttributesList()) { - final String key = it.getKey().replace('.', '_'); - final String value = anyValueToString(it.getValue()); - nodeLabels.putIfAbsent(key, value); - } - // Second pass: apply fallback mappings — only if the target key is absent - for (final var it : request.getResource().getAttributesList()) { - final String targetKey = FALLBACK_LABEL_MAPPINGS.get(it.getKey()); - if (targetKey != null) { - nodeLabels.putIfAbsent(targetKey, anyValueToString(it.getValue())); - } - } + final Map nodeLabels = nodeLabels(request.getResource().getAttributesList()); // A request is analysed a minute at a time, oldest minute first. A MAL rule folds every sample of // an entity into one value stamped with the first sample's time, which is right for a scrape, whose @@ -255,6 +250,35 @@ public void start() throws ModuleStartException { } } + /** + * The labels every sample of a resource carries, from its resource attributes. + */ + static Map nodeLabels(final List attributes) { + // First pass: collect all resource attributes with dots replaced by underscores + final Map nodeLabels = new HashMap<>(); + for (final var it : attributes) { + final String key = it.getKey().replace('.', '_'); + final String value = anyValueToString(it.getValue()); + nodeLabels.putIfAbsent(key, value); + } + // Second pass: apply fallback mappings — only if the target key is absent + for (final var it : attributes) { + final String targetKey = FALLBACK_LABEL_MAPPINGS.get(it.getKey()); + if (targetKey != null) { + nodeLabels.putIfAbsent(targetKey, anyValueToString(it.getValue())); + } + } + if (!nodeLabels.containsKey(NODE_IDENTIFIER_HOST_NAME)) { + for (final var it : attributes) { + if (SERVER_ADDRESS.equals(it.getKey())) { + nodeLabels.put(NODE_IDENTIFIER_HOST_NAME, anyValueToString(it.getValue())); + break; + } + } + } + return nodeLabels; + } + private static Map buildLabels(List kvs) { return kvs .stream() diff --git a/oap-server/server-receiver-plugin/otel-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OpenTelemetryMetricRequestProcessorTest.java b/oap-server/server-receiver-plugin/otel-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OpenTelemetryMetricRequestProcessorTest.java index 883ece04295b..38ac0ded4451 100644 --- a/oap-server/server-receiver-plugin/otel-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OpenTelemetryMetricRequestProcessorTest.java +++ b/oap-server/server-receiver-plugin/otel-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OpenTelemetryMetricRequestProcessorTest.java @@ -17,6 +17,8 @@ package org.apache.skywalking.oap.server.receiver.otel.otlp; +import io.opentelemetry.proto.common.v1.AnyValue; +import io.opentelemetry.proto.common.v1.KeyValue; import io.opentelemetry.proto.metrics.v1.ExponentialHistogram; import io.opentelemetry.proto.metrics.v1.ExponentialHistogramDataPoint; import io.opentelemetry.proto.metrics.v1.Metric; @@ -131,4 +133,33 @@ public void testAdaptExponentialHistogram() throws NoSuchMethodException, Invoca assertTrue(histogramMetric.getBuckets().containsKey(-Math.pow(base, 17))); assertEquals(2, histogramMetric.getBuckets().get(-Math.pow(base, 17))); } + + private static KeyValue attribute(final String key, final String value) { + return KeyValue.newBuilder().setKey(key).setValue(AnyValue.newBuilder().setStringValue(value)).build(); + } + + // OTel Collector v0.127.0 and later send a Prometheus target's host as server.address only. + @Test + public void testHostNameFromServerAddress() { + final Map labels = OpenTelemetryMetricRequestProcessor.nodeLabels(List.of( + attribute("service.name", "windows-monitoring"), + attribute("server.address", "172.25.0.1"), + attribute("service.instance.id", "172.25.0.1:9182") + )); + assertEquals("172.25.0.1", labels.get("node_identifier_host_name")); + assertEquals("windows-monitoring", labels.get("job_name")); + assertEquals("172.25.0.1", labels.get("server_address")); + } + + @Test + public void testLegacyHostAttributesWinOverServerAddress() { + assertEquals("win-host", OpenTelemetryMetricRequestProcessor.nodeLabels(List.of( + attribute("server.address", "172.25.0.1"), + attribute("host.name", "win-host") + )).get("node_identifier_host_name")); + assertEquals("10.211.55.3", OpenTelemetryMetricRequestProcessor.nodeLabels(List.of( + attribute("net.host.name", "10.211.55.3"), + attribute("server.address", "10.211.55.3") + )).get("node_identifier_host_name")); + } } diff --git a/oap-server/server-starter/src/main/resources/otel-rules/process-hostmetrics-linux.yaml b/oap-server/server-starter/src/main/resources/otel-rules/process-hostmetrics-linux.yaml index 71356da371b6..8a84c009551a 100644 --- a/oap-server/server-starter/src/main/resources/otel-rules/process-hostmetrics-linux.yaml +++ b/oap-server/server-starter/src/main/resources/otel-rules/process-hostmetrics-linux.yaml @@ -67,10 +67,10 @@ metricsRules: - name: memory_resident_bytes exp: "process_memory_usage.sum(['node_identifier_host_name','process_name'])" - # process.memory.utilization is emitted as a [0,1] fraction. Multiply by - # 100 before SkyWalking fixed-point storage to preserve two decimal places - # of percentage precision (basis points). UI expressions must divide the - # stored metric by 100. + # process.memory.utilization is emitted as a percentage (0-100): the + # Collector records gopsutil's MemoryPercent, despite the metric's unit "1". + # Multiply by 100 before SkyWalking's integer storage to keep two decimal + # places (basis points). UI expressions must divide the stored metric by 100. - name: memory_resident_percent exp: > process_memory_utilization diff --git a/oap-server/server-starter/src/main/resources/otel-rules/process-hostmetrics-windows.yaml b/oap-server/server-starter/src/main/resources/otel-rules/process-hostmetrics-windows.yaml index bde9f5b73580..86657630e6bb 100644 --- a/oap-server/server-starter/src/main/resources/otel-rules/process-hostmetrics-windows.yaml +++ b/oap-server/server-starter/src/main/resources/otel-rules/process-hostmetrics-windows.yaml @@ -67,10 +67,10 @@ metricsRules: - name: memory_resident_bytes exp: "process_memory_usage.sum(['node_identifier_host_name','process_name'])" - # process.memory.utilization is emitted as a [0,1] fraction. Multiply by - # 100 before SkyWalking fixed-point storage to preserve two decimal places - # of percentage precision (basis points). UI expressions must divide the - # stored metric by 100. + # process.memory.utilization is emitted as a percentage (0-100): the + # Collector records gopsutil's MemoryPercent, despite the metric's unit "1". + # Multiply by 100 before SkyWalking's integer storage to keep two decimal + # places (basis points). UI expressions must divide the stored metric by 100. - name: memory_resident_percent exp: > process_memory_utilization diff --git a/oap-server/server-starter/src/main/resources/otel-rules/windows.yaml b/oap-server/server-starter/src/main/resources/otel-rules/windows.yaml index c7ad4a4d22a6..f071c12807c1 100644 --- a/oap-server/server-starter/src/main/resources/otel-rules/windows.yaml +++ b/oap-server/server-starter/src/main/resources/otel-rules/windows.yaml @@ -29,15 +29,18 @@ metricsRules: # --------------------------------------------------------------------------- # CPU + # Windows modes overlap: windows_exporter's `privileged` already includes + # `interrupt` and `dpc` time, and hostmetrics' `system` already includes + # `interrupt` time. Busy CPU is therefore user + privileged/system only. - name: cpu_total_percentage exp: > (windows_cpu_time_total * 100) - .tagNotEqual('mode','idle') + .tagMatch('mode','^(user|privileged)$') .sum(['node_identifier_host_name']) .rate('PT1M') + (system_cpu_time * 100) - .tagNotEqual('mode','idle') + .tagMatch('mode','^(user|system)$') .sum(['node_identifier_host_name']) .rate('PT1M') @@ -51,16 +54,18 @@ metricsRules: .sum(['node_identifier_host_name','mode']) .rate('PT1M') - # Physical memory + # Physical memory. The windows_exporter branch reads the `memory` collector + # (windows_exporter 0.29.0+); 0.31.0 removed the `cs` collector and the + # `os` memory metrics this rule used before. - name: memory_total exp: > - windows_cs_physical_memory_bytes + windows_memory_physical_total_bytes + system_memory_limit.sum(['node_identifier_host_name']) - name: memory_available exp: > - windows_os_physical_memory_free_bytes + windows_memory_physical_free_bytes + system_memory_usage .tagEqual('state','free') @@ -68,7 +73,7 @@ metricsRules: - name: memory_used exp: > - (windows_cs_physical_memory_bytes - windows_os_physical_memory_free_bytes) + (windows_memory_physical_total_bytes - windows_memory_physical_free_bytes) + ( system_memory_limit.sum(['node_identifier_host_name']) @@ -76,13 +81,14 @@ metricsRules: system_memory_usage.tagEqual('state','free').sum(['node_identifier_host_name']) ) - # Canonical virtual memory semantics. - # The hostmetrics branch uses Windows Performance Counters: + # Canonical virtual memory semantics: the commit limit and committed bytes. + # windows_exporter reads them from its `memory` collector; the hostmetrics + # branch uses Windows Performance Counters: # Memory\\Commit Limit # Memory\\Committed Bytes - name: memory_virtual_memory_free exp: > - windows_os_virtual_memory_free_bytes + (windows_memory_commit_limit - windows_memory_committed_bytes) + ( system_virtual_memory_commit_limit.sum(['node_identifier_host_name']) @@ -92,14 +98,14 @@ metricsRules: - name: memory_virtual_memory_total exp: > - windows_os_virtual_memory_bytes + windows_memory_commit_limit + system_virtual_memory_commit_limit.sum(['node_identifier_host_name']) - name: memory_virtual_memory_percentage exp: > ( - 100 - ((windows_os_virtual_memory_free_bytes * 100) / windows_os_virtual_memory_bytes) + windows_memory_committed_bytes * 100 / windows_memory_commit_limit ) + ( diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/assert-collector-output.sh b/test/e2e-v2/cases/vm/otel-hostmetrics/assert-collector-output.sh index 9e0b4c8ad38b..fc3dd3efe411 100755 --- a/test/e2e-v2/cases/vm/otel-hostmetrics/assert-collector-output.sh +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/assert-collector-output.sh @@ -65,7 +65,7 @@ function finish_block() { in_count = 0 } -/-> Name: process.count[[:space:]]*$/ { +/-> Name: process\.count([[:space:]]|$)/ { finish_block() in_count = 1 points = 0 @@ -105,21 +105,38 @@ END { # --------------------------------------------------------------------------- # Windows host normalization: -# native system.cpu.time state -> canonical mode. +# native system.cpu.time state -> canonical mode, for all four +# Windows states. Other host metrics keep their state attribute, so the +# check is scoped to the CPU metric block. # --------------------------------------------------------------------------- -grep -q -- '-> Name: system.cpu.time' \ - "${TMPDIR}/windows-host.log" +awk ' +/-> Name: / { + in_cpu = ($0 ~ /-> Name: system\.cpu\.time([[:space:]]|$)/) + if (in_cpu) { + seen = 1 + } + next +} -grep -q -- '-> mode: Str(idle)' \ - "${TMPDIR}/windows-host.log" +in_cpu && /-> state:/ { + bad = 1 +} -grep -q -- '-> mode: Str(user)' \ - "${TMPDIR}/windows-host.log" +in_cpu && /-> mode: Str\((user|system|interrupt|idle)\)/ { + match($0, /Str\([a-z]+\)/) + modes[substr($0, RSTART + 4, RLENGTH - 5)] = 1 +} -if grep -q -- '-> state:' "${TMPDIR}/windows-host.log"; then - echo "Windows system.cpu.time still contains state attribute" >&2 +END { + if (!seen || bad) { + exit 1 + } + exit(("user" in modes) && ("system" in modes) && ("interrupt" in modes) && ("idle" in modes) ? 0 : 1) +} +' "${TMPDIR}/windows-host.log" || { + echo "Windows system.cpu.time state was not normalized to mode" >&2 exit 1 -fi +} # --------------------------------------------------------------------------- @@ -129,7 +146,7 @@ fi grep -q -- '-> Name: process.open_handles' \ "${TMPDIR}/windows-process.log" -if grep -q -- '-> Name: process.handles[[:space:]]*$' \ +if grep -q -E -- '-> Name: process\.handles([[:space:]]|$)' \ "${TMPDIR}/windows-process.log"; then echo "Native process.handles survived normalization" >&2 exit 1 diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/docker-compose.yml b/test/e2e-v2/cases/vm/otel-hostmetrics/docker-compose.yml index 88168a2965f7..325d39d6f6d8 100644 --- a/test/e2e-v2/cases/vm/otel-hostmetrics/docker-compose.yml +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/docker-compose.yml @@ -53,14 +53,24 @@ services: - ./fixtures:/fixtures:ro networks: - e2e + # windows-host.json carries cumulative system.cpu.time placeholders + # (900000001.0 user, 900000002.0 system, 900000003.0 interrupt, + # 900000004.0 idle). Each replay sets them from the clock: per second, + # user +0.5, system +0.3, interrupt +0.05 (already inside system) and + # idle +3.2 CPU-seconds on 4 logical CPUs, so OAP's rate() yields 80% busy + # CPU and 20% normalized CPU. Each value is computed from the same NOW as the + # datapoint timestamp, so the counters rise with the timestamp even if the + # clock jumps, and rate() is exact however long a replay round takes. command: - sh - -c - | while true; do - TS="$$(date +%s)000000000" + NOW="$$(date +%s)" + TS="$${NOW}000000000" + CPU="$$(awk -v t="$${NOW}" 'BEGIN { printf "s/900000001[.]0/%.1f/g;s/900000002[.]0/%.1f/g;s/900000003[.]0/%.2f/g;s/900000004[.]0/%.1f/g", t*0.5, t*0.3, t*0.05, t*3.2 }')" echo "Replaying Windows OTLP fixtures timestamp=$${TS}" - sed "s/1788474000000000000/$${TS}/g" \ + sed -e "s/1788474000000000000/$${TS}/g" -e "$${CPU}" \ /fixtures/windows-host.json \ | curl -sS \ -o /dev/null \ diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/e2e.yaml b/test/e2e-v2/cases/vm/otel-hostmetrics/e2e.yaml index bb3c28839b8f..9aa86b5de3fa 100644 --- a/test/e2e-v2/cases/vm/otel-hostmetrics/e2e.yaml +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/e2e.yaml @@ -73,7 +73,30 @@ verify: # Synthetic native-Windows OTLP fixtures are normalized by Collector 0.158 # and then consumed by the production Windows MAL rules. - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_cpu_cores_num --service-name=win-fixture - expected: ../expected/metrics-has-value.yml + expected: expected/win-cpu-cores-4.yml + + # CPU from system.cpu.time on 4 logical CPUs. The replay grows user by 0.5, + # system by 0.3 and interrupt by 0.05 CPU-seconds per second; `system` + # already includes interrupt time, so busy CPU is 80% and normalized 20%. + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_cpu_total_percentage --service-name=win-fixture + expected: expected/win-cpu-total-80.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_cpu_norm_percentage --service-name=win-fixture + expected: expected/win-cpu-norm-20.yml + + # Memory, commit, pagefile, filesystem, handles and load: exact values the + # fixture's fixed gauges produce. + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_memory_used --service-name=win-fixture + expected: expected/win-memory-used-5368709120.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_memory_virtual_memory_percentage --service-name=win-fixture + expected: expected/win-commit-percentage-50.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_memory_pagefile_percentage --service-name=win-fixture + expected: expected/win-pagefile-percentage-25.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression="aggregate_labels(meter_win_filesystem_percentage,avg)" --service-name=win-fixture + expected: expected/win-filesystem-percentage-60.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_filehandles_allocated --service-name=win-fixture + expected: expected/win-filehandles-45000.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_cpu_load1 --service-name=win-fixture + expected: expected/win-cpu-load1-50.yml - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=mp_process_windows_num_procs --service-name=win-fixture --instance-name=svchost.exe expected: ../expected/metrics-has-value.yml diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-commit-percentage-50.yml b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-commit-percentage-50.yml new file mode 100644 index 000000000000..34f6c30e3e87 --- /dev/null +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-commit-percentage-50.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 50 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-cores-4.yml b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-cores-4.yml new file mode 100644 index 000000000000..9f8ffd75924a --- /dev/null +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-cores-4.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 4 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-load1-50.yml b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-load1-50.yml new file mode 100644 index 000000000000..34f6c30e3e87 --- /dev/null +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-load1-50.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 50 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-norm-20.yml b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-norm-20.yml new file mode 100644 index 000000000000..c78042b2bfb9 --- /dev/null +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-norm-20.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 20 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-total-80.yml b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-total-80.yml new file mode 100644 index 000000000000..7fe0a1d2aa5c --- /dev/null +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-total-80.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 80 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-filehandles-45000.yml b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-filehandles-45000.yml new file mode 100644 index 000000000000..934d2225b92d --- /dev/null +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-filehandles-45000.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 45000 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-filesystem-percentage-60.yml b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-filesystem-percentage-60.yml new file mode 100644 index 000000000000..da3142840071 --- /dev/null +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-filesystem-percentage-60.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 60 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-memory-used-5368709120.yml b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-memory-used-5368709120.yml new file mode 100644 index 000000000000..8d522919bfb0 --- /dev/null +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-memory-used-5368709120.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 5368709120 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-pagefile-percentage-25.yml b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-pagefile-percentage-25.yml new file mode 100644 index 000000000000..d74025d15a42 --- /dev/null +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-pagefile-percentage-25.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 25 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/fixtures/windows-host.json b/test/e2e-v2/cases/vm/otel-hostmetrics/fixtures/windows-host.json index 048391f8999a..6f66ed3fdccd 100644 --- a/test/e2e-v2/cases/vm/otel-hostmetrics/fixtures/windows-host.json +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/fixtures/windows-host.json @@ -1 +1 @@ -{"resourceMetrics":[{"resource":{"attributes":[{"key":"host.name","value":{"stringValue":"win-fixture"}}]},"scopeMetrics":[{"scope":{"name":"windows-host-fixture"},"metrics":[{"name":"system.cpu.logical.count","unit":"{cpu}","gauge":{"dataPoints":[{"timeUnixNano":"1788474000000000000","asInt":"4"}]}},{"name":"system.cpu.time","unit":"s","sum":{"dataPoints":[{"attributes":[{"key":"state","value":{"stringValue":"idle"}}],"timeUnixNano":"1788474000000000000","asDouble":100.0},{"attributes":[{"key":"state","value":{"stringValue":"user"}}],"timeUnixNano":"1788474000000000000","asDouble":20.0}],"aggregationTemporality":2,"isMonotonic":true}}]}]}]} +{"resourceMetrics":[{"resource":{"attributes":[{"key":"host.name","value":{"stringValue":"win-fixture"}}]},"scopeMetrics":[{"scope":{"name":"windows-host-fixture"},"metrics":[{"name":"system.cpu.logical.count","unit":"{cpu}","sum":{"dataPoints":[{"timeUnixNano":"1788474000000000000","asInt":"4"}],"aggregationTemporality":2,"isMonotonic":false}},{"name":"system.cpu.time","unit":"s","sum":{"dataPoints":[{"attributes":[{"key":"state","value":{"stringValue":"user"}}],"timeUnixNano":"1788474000000000000","asDouble":900000001.0},{"attributes":[{"key":"state","value":{"stringValue":"system"}}],"timeUnixNano":"1788474000000000000","asDouble":900000002.0},{"attributes":[{"key":"state","value":{"stringValue":"interrupt"}}],"timeUnixNano":"1788474000000000000","asDouble":900000003.0},{"attributes":[{"key":"state","value":{"stringValue":"idle"}}],"timeUnixNano":"1788474000000000000","asDouble":900000004.0}],"aggregationTemporality":2,"isMonotonic":true}},{"name":"system.memory.limit","unit":"By","sum":{"dataPoints":[{"timeUnixNano":"1788474000000000000","asInt":"8589934592"}],"aggregationTemporality":2,"isMonotonic":false}},{"name":"system.memory.usage","unit":"By","sum":{"dataPoints":[{"attributes":[{"key":"state","value":{"stringValue":"used"}}],"timeUnixNano":"1788474000000000000","asInt":"5368709120"},{"attributes":[{"key":"state","value":{"stringValue":"free"}}],"timeUnixNano":"1788474000000000000","asInt":"3221225472"}],"aggregationTemporality":2,"isMonotonic":false}},{"name":"system.paging.usage","unit":"By","sum":{"dataPoints":[{"attributes":[{"key":"device","value":{"stringValue":"C:\\pagefile.sys"}},{"key":"state","value":{"stringValue":"used"}}],"timeUnixNano":"1788474000000000000","asInt":"1073741824"},{"attributes":[{"key":"device","value":{"stringValue":"C:\\pagefile.sys"}},{"key":"state","value":{"stringValue":"free"}}],"timeUnixNano":"1788474000000000000","asInt":"3221225472"}],"aggregationTemporality":2,"isMonotonic":false}},{"name":"system.filesystem.usage","unit":"By","sum":{"dataPoints":[{"attributes":[{"key":"device","value":{"stringValue":"C:"}},{"key":"mountpoint","value":{"stringValue":"C:"}},{"key":"type","value":{"stringValue":"NTFS"}},{"key":"mode","value":{"stringValue":"rw"}},{"key":"state","value":{"stringValue":"used"}}],"timeUnixNano":"1788474000000000000","asInt":"64424509440"},{"attributes":[{"key":"device","value":{"stringValue":"C:"}},{"key":"mountpoint","value":{"stringValue":"C:"}},{"key":"type","value":{"stringValue":"NTFS"}},{"key":"mode","value":{"stringValue":"rw"}},{"key":"state","value":{"stringValue":"free"}}],"timeUnixNano":"1788474000000000000","asInt":"42949672960"},{"attributes":[{"key":"device","value":{"stringValue":"C:"}},{"key":"mountpoint","value":{"stringValue":"C:"}},{"key":"type","value":{"stringValue":"NTFS"}},{"key":"mode","value":{"stringValue":"rw"}},{"key":"state","value":{"stringValue":"reserved"}}],"timeUnixNano":"1788474000000000000","asInt":"0"}],"aggregationTemporality":2,"isMonotonic":false}},{"name":"system.cpu.load_average.1m","unit":"{thread}","gauge":{"dataPoints":[{"timeUnixNano":"1788474000000000000","asDouble":0.5}]}},{"name":"system.cpu.load_average.5m","unit":"{thread}","gauge":{"dataPoints":[{"timeUnixNano":"1788474000000000000","asDouble":0.4}]}},{"name":"system.cpu.load_average.15m","unit":"{thread}","gauge":{"dataPoints":[{"timeUnixNano":"1788474000000000000","asDouble":0.3}]}},{"name":"system.virtual.memory.commit_limit","unit":"By","gauge":{"dataPoints":[{"timeUnixNano":"1788474000000000000","asDouble":12884901888}]}},{"name":"system.virtual.memory.committed","unit":"By","gauge":{"dataPoints":[{"timeUnixNano":"1788474000000000000","asDouble":6442450944}]}},{"name":"system.filehandles.allocated","unit":"{count}","gauge":{"dataPoints":[{"timeUnixNano":"1788474000000000000","asDouble":45000}]}}]}]}]} diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/process-hostmetrics-linux.yaml b/test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/process-hostmetrics-linux.yaml index 71356da371b6..8a84c009551a 100644 --- a/test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/process-hostmetrics-linux.yaml +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/process-hostmetrics-linux.yaml @@ -67,10 +67,10 @@ metricsRules: - name: memory_resident_bytes exp: "process_memory_usage.sum(['node_identifier_host_name','process_name'])" - # process.memory.utilization is emitted as a [0,1] fraction. Multiply by - # 100 before SkyWalking fixed-point storage to preserve two decimal places - # of percentage precision (basis points). UI expressions must divide the - # stored metric by 100. + # process.memory.utilization is emitted as a percentage (0-100): the + # Collector records gopsutil's MemoryPercent, despite the metric's unit "1". + # Multiply by 100 before SkyWalking's integer storage to keep two decimal + # places (basis points). UI expressions must divide the stored metric by 100. - name: memory_resident_percent exp: > process_memory_utilization diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/process-hostmetrics-windows.yaml b/test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/process-hostmetrics-windows.yaml index bde9f5b73580..86657630e6bb 100644 --- a/test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/process-hostmetrics-windows.yaml +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/process-hostmetrics-windows.yaml @@ -67,10 +67,10 @@ metricsRules: - name: memory_resident_bytes exp: "process_memory_usage.sum(['node_identifier_host_name','process_name'])" - # process.memory.utilization is emitted as a [0,1] fraction. Multiply by - # 100 before SkyWalking fixed-point storage to preserve two decimal places - # of percentage precision (basis points). UI expressions must divide the - # stored metric by 100. + # process.memory.utilization is emitted as a percentage (0-100): the + # Collector records gopsutil's MemoryPercent, despite the metric's unit "1". + # Multiply by 100 before SkyWalking's integer storage to keep two decimal + # places (basis points). UI expressions must divide the stored metric by 100. - name: memory_resident_percent exp: > process_memory_utilization diff --git a/test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/windows.yaml b/test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/windows.yaml index c7ad4a4d22a6..f071c12807c1 100644 --- a/test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/windows.yaml +++ b/test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/windows.yaml @@ -29,15 +29,18 @@ metricsRules: # --------------------------------------------------------------------------- # CPU + # Windows modes overlap: windows_exporter's `privileged` already includes + # `interrupt` and `dpc` time, and hostmetrics' `system` already includes + # `interrupt` time. Busy CPU is therefore user + privileged/system only. - name: cpu_total_percentage exp: > (windows_cpu_time_total * 100) - .tagNotEqual('mode','idle') + .tagMatch('mode','^(user|privileged)$') .sum(['node_identifier_host_name']) .rate('PT1M') + (system_cpu_time * 100) - .tagNotEqual('mode','idle') + .tagMatch('mode','^(user|system)$') .sum(['node_identifier_host_name']) .rate('PT1M') @@ -51,16 +54,18 @@ metricsRules: .sum(['node_identifier_host_name','mode']) .rate('PT1M') - # Physical memory + # Physical memory. The windows_exporter branch reads the `memory` collector + # (windows_exporter 0.29.0+); 0.31.0 removed the `cs` collector and the + # `os` memory metrics this rule used before. - name: memory_total exp: > - windows_cs_physical_memory_bytes + windows_memory_physical_total_bytes + system_memory_limit.sum(['node_identifier_host_name']) - name: memory_available exp: > - windows_os_physical_memory_free_bytes + windows_memory_physical_free_bytes + system_memory_usage .tagEqual('state','free') @@ -68,7 +73,7 @@ metricsRules: - name: memory_used exp: > - (windows_cs_physical_memory_bytes - windows_os_physical_memory_free_bytes) + (windows_memory_physical_total_bytes - windows_memory_physical_free_bytes) + ( system_memory_limit.sum(['node_identifier_host_name']) @@ -76,13 +81,14 @@ metricsRules: system_memory_usage.tagEqual('state','free').sum(['node_identifier_host_name']) ) - # Canonical virtual memory semantics. - # The hostmetrics branch uses Windows Performance Counters: + # Canonical virtual memory semantics: the commit limit and committed bytes. + # windows_exporter reads them from its `memory` collector; the hostmetrics + # branch uses Windows Performance Counters: # Memory\\Commit Limit # Memory\\Committed Bytes - name: memory_virtual_memory_free exp: > - windows_os_virtual_memory_free_bytes + (windows_memory_commit_limit - windows_memory_committed_bytes) + ( system_virtual_memory_commit_limit.sum(['node_identifier_host_name']) @@ -92,14 +98,14 @@ metricsRules: - name: memory_virtual_memory_total exp: > - windows_os_virtual_memory_bytes + windows_memory_commit_limit + system_virtual_memory_commit_limit.sum(['node_identifier_host_name']) - name: memory_virtual_memory_percentage exp: > ( - 100 - ((windows_os_virtual_memory_free_bytes * 100) / windows_os_virtual_memory_bytes) + windows_memory_committed_bytes * 100 / windows_memory_commit_limit ) + ( diff --git a/test/e2e-v2/cases/win/expected/commit-limit-10597691392.yml b/test/e2e-v2/cases/win/expected/commit-limit-10597691392.yml new file mode 100644 index 000000000000..fc093d3b96b5 --- /dev/null +++ b/test/e2e-v2/cases/win/expected/commit-limit-10597691392.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 10597691392 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/win/expected/commit-limit-8838758400.yml b/test/e2e-v2/cases/win/expected/commit-limit-8838758400.yml new file mode 100644 index 000000000000..a5169b5dc6dc --- /dev/null +++ b/test/e2e-v2/cases/win/expected/commit-limit-8838758400.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 8838758400 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/win/expected/commit-percentage-50.yml b/test/e2e-v2/cases/win/expected/commit-percentage-50.yml new file mode 100644 index 000000000000..34f6c30e3e87 --- /dev/null +++ b/test/e2e-v2/cases/win/expected/commit-percentage-50.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 50 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/win/expected/memory-total-4294967296.yml b/test/e2e-v2/cases/win/expected/memory-total-4294967296.yml new file mode 100644 index 000000000000..d2c12f7f4293 --- /dev/null +++ b/test/e2e-v2/cases/win/expected/memory-total-4294967296.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 4294967296 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/win/expected/memory-total-8584425472.yml b/test/e2e-v2/cases/win/expected/memory-total-8584425472.yml new file mode 100644 index 000000000000..1ad27d50df0a --- /dev/null +++ b/test/e2e-v2/cases/win/expected/memory-total-8584425472.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 8584425472 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/win/expected/memory-used-2794070016.yml b/test/e2e-v2/cases/win/expected/memory-used-2794070016.yml new file mode 100644 index 000000000000..0937128410c9 --- /dev/null +++ b/test/e2e-v2/cases/win/expected/memory-used-2794070016.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 2794070016 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/win/expected/memory-used-3221225472.yml b/test/e2e-v2/cases/win/expected/memory-used-3221225472.yml new file mode 100644 index 000000000000..d42257bdaaae --- /dev/null +++ b/test/e2e-v2/cases/win/expected/memory-used-3221225472.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: 3221225472 + owner: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/win/expected/service.yml b/test/e2e-v2/cases/win/expected/service.yml index 940f9201ae95..3ef0fbbaebc3 100644 --- a/test/e2e-v2/cases/win/expected/service.yml +++ b/test/e2e-v2/cases/win/expected/service.yml @@ -21,4 +21,11 @@ normal: true layers: - OS_WINDOWS +- id: {{ b64enc "172.25.0.1" }}.1 + name: 172.25.0.1 + group: "" + shortname: 172.25.0.1 + normal: true + layers: + - OS_WINDOWS {{- end }} diff --git a/test/e2e-v2/cases/win/mock-data/otel-mock-metrics.json b/test/e2e-v2/cases/win/mock-data/otel-mock-metrics.json index 4cb9c82d67ae..bc30304c05e4 100644 --- a/test/e2e-v2/cases/win/mock-data/otel-mock-metrics.json +++ b/test/e2e-v2/cases/win/mock-data/otel-mock-metrics.json @@ -123,8 +123,8 @@ "scope": {}, "metrics": [ { - "name": "windows_os_virtual_memory_bytes", - "description": "OperatingSystem.TotalVirtualMemorySize", + "name": "windows_memory_commit_limit", + "description": "Amount of virtual memory, in bytes, that can be committed without having to extend the paging file(s)", "gauge": { "dataPoints": [ { @@ -134,6 +134,42 @@ ] } }, + { + "name": "windows_memory_committed_bytes", + "description": "Amount of committed virtual memory, in bytes", + "gauge": { + "dataPoints": [ + { + "timeUnixNano": "1676140375004000000", + "asDouble": 4.4193792E9 + } + ] + } + }, + { + "name": "windows_memory_physical_total_bytes", + "description": "Total bytes of physical memory available to the operating system", + "gauge": { + "dataPoints": [ + { + "timeUnixNano": "1676140375004000000", + "asDouble": 4.294967296E9 + } + ] + } + }, + { + "name": "windows_memory_physical_free_bytes", + "description": "Bytes of physical memory currently unused and available", + "gauge": { + "dataPoints": [ + { + "timeUnixNano": "1676140375004000000", + "asDouble": 1.073741824E9 + } + ] + } + }, { "name": "windows_logical_disk_write_seconds_total", "description": "Seconds that the disk was busy servicing write requests (LogicalDisk.PercentDiskWriteTime)", diff --git a/test/e2e-v2/cases/win/mock-data/windows-exporter-0.31.8.json b/test/e2e-v2/cases/win/mock-data/windows-exporter-0.31.8.json new file mode 100644 index 000000000000..77937f2d6dfc --- /dev/null +++ b/test/e2e-v2/cases/win/mock-data/windows-exporter-0.31.8.json @@ -0,0 +1,544 @@ +{ + "resourceMetrics": [ + { + "resource": { + "attributes": [ + { + "key": "service.name", + "value": { + "stringValue": "windows-monitoring" + } + }, + { + "key": "server.address", + "value": { + "stringValue": "172.25.0.1" + } + }, + { + "key": "service.instance.id", + "value": { + "stringValue": "172.25.0.1:9182" + } + }, + { + "key": "server.port", + "value": { + "stringValue": "9182" + } + }, + { + "key": "url.scheme", + "value": { + "stringValue": "http" + } + } + ] + }, + "scopeMetrics": [ + { + "scope": { + "name": "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver", + "version": "0.158.0" + }, + "metrics": [ + { + "name": "windows_cpu_time_total", + "description": "Time that processor spent in different modes (dpc, idle, interrupt, privileged, user)", + "sum": { + "dataPoints": [ + { + "attributes": [ + { + "key": "core", + "value": { + "stringValue": "0,0" + } + }, + { + "key": "mode", + "value": { + "stringValue": "dpc" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 1.9375 + }, + { + "attributes": [ + { + "key": "core", + "value": { + "stringValue": "0,0" + } + }, + { + "key": "mode", + "value": { + "stringValue": "idle" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 84.6875 + }, + { + "attributes": [ + { + "key": "core", + "value": { + "stringValue": "0,0" + } + }, + { + "key": "mode", + "value": { + "stringValue": "interrupt" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 3.0625 + }, + { + "attributes": [ + { + "key": "core", + "value": { + "stringValue": "0,0" + } + }, + { + "key": "mode", + "value": { + "stringValue": "privileged" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 60.25 + }, + { + "attributes": [ + { + "key": "core", + "value": { + "stringValue": "0,0" + } + }, + { + "key": "mode", + "value": { + "stringValue": "user" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 98.296875 + }, + { + "attributes": [ + { + "key": "core", + "value": { + "stringValue": "0,1" + } + }, + { + "key": "mode", + "value": { + "stringValue": "dpc" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 0.203125 + }, + { + "attributes": [ + { + "key": "core", + "value": { + "stringValue": "0,1" + } + }, + { + "key": "mode", + "value": { + "stringValue": "idle" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 85.59375 + }, + { + "attributes": [ + { + "key": "core", + "value": { + "stringValue": "0,1" + } + }, + { + "key": "mode", + "value": { + "stringValue": "interrupt" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 3.125 + }, + { + "attributes": [ + { + "key": "core", + "value": { + "stringValue": "0,1" + } + }, + { + "key": "mode", + "value": { + "stringValue": "privileged" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 58.171875 + }, + { + "attributes": [ + { + "key": "core", + "value": { + "stringValue": "0,1" + } + }, + { + "key": "mode", + "value": { + "stringValue": "user" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 99.453125 + } + ], + "aggregationTemporality": 2, + "isMonotonic": true + }, + "metadata": [ + { + "key": "prometheus.type", + "value": { + "stringValue": "counter" + } + } + ] + }, + { + "name": "windows_logical_disk_read_bytes_total", + "description": "The number of bytes transferred from the disk during read operations (LogicalDisk.DiskReadBytesPerSec)", + "sum": { + "dataPoints": [ + { + "attributes": [ + { + "key": "volume", + "value": { + "stringValue": "C:" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 1110196736 + }, + { + "attributes": [ + { + "key": "volume", + "value": { + "stringValue": "D:" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 17408 + }, + { + "attributes": [ + { + "key": "volume", + "value": { + "stringValue": "HarddiskVolume2" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 0 + }, + { + "attributes": [ + { + "key": "volume", + "value": { + "stringValue": "HarddiskVolume3" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 12578304 + } + ], + "aggregationTemporality": 2, + "isMonotonic": true + }, + "metadata": [ + { + "key": "prometheus.type", + "value": { + "stringValue": "counter" + } + } + ] + }, + { + "name": "windows_logical_disk_write_bytes_total", + "description": "The number of bytes transferred to the disk during write operations (LogicalDisk.DiskWriteBytesPerSec)", + "sum": { + "dataPoints": [ + { + "attributes": [ + { + "key": "volume", + "value": { + "stringValue": "C:" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 1111252480 + }, + { + "attributes": [ + { + "key": "volume", + "value": { + "stringValue": "D:" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 814047232 + }, + { + "attributes": [ + { + "key": "volume", + "value": { + "stringValue": "HarddiskVolume2" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 0 + }, + { + "attributes": [ + { + "key": "volume", + "value": { + "stringValue": "HarddiskVolume3" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 159744 + } + ], + "aggregationTemporality": 2, + "isMonotonic": true + }, + "metadata": [ + { + "key": "prometheus.type", + "value": { + "stringValue": "counter" + } + } + ] + }, + { + "name": "windows_memory_commit_limit", + "description": "(CommitLimit)", + "gauge": { + "dataPoints": [ + { + "timeUnixNano": "1790731774528000000", + "asDouble": 10597691392 + } + ] + }, + "metadata": [ + { + "key": "prometheus.type", + "value": { + "stringValue": "gauge" + } + } + ] + }, + { + "name": "windows_memory_committed_bytes", + "description": "(CommittedBytes)", + "gauge": { + "dataPoints": [ + { + "timeUnixNano": "1790731774528000000", + "asDouble": 2574090240 + } + ] + }, + "metadata": [ + { + "key": "prometheus.type", + "value": { + "stringValue": "gauge" + } + } + ] + }, + { + "name": "windows_memory_physical_free_bytes", + "description": "The amount of physical memory currently available, in bytes. This is the amount of physical memory that can be immediately reused without having to write its contents to disk first. It is the sum of the size of the standby, free, and zero lists.", + "gauge": { + "dataPoints": [ + { + "timeUnixNano": "1790731774528000000", + "asDouble": 5790355456 + } + ] + }, + "metadata": [ + { + "key": "prometheus.type", + "value": { + "stringValue": "gauge" + } + } + ] + }, + { + "name": "windows_memory_physical_total_bytes", + "description": "The amount of actual physical memory, in bytes.", + "gauge": { + "dataPoints": [ + { + "timeUnixNano": "1790731774528000000", + "asDouble": 8584425472 + } + ] + }, + "metadata": [ + { + "key": "prometheus.type", + "value": { + "stringValue": "gauge" + } + } + ] + }, + { + "name": "windows_net_bytes_received_total", + "description": "(Network.BytesReceivedPerSec)", + "sum": { + "dataPoints": [ + { + "attributes": [ + { + "key": "nic", + "value": { + "stringValue": "Mellanox ConnectX-4 Lx Virtual Ethernet Adapter" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 236607072 + }, + { + "attributes": [ + { + "key": "nic", + "value": { + "stringValue": "Microsoft Hyper-V Network Adapter _3" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 253580969 + } + ], + "aggregationTemporality": 2, + "isMonotonic": true + }, + "metadata": [ + { + "key": "prometheus.type", + "value": { + "stringValue": "counter" + } + } + ] + }, + { + "name": "windows_net_bytes_sent_total", + "description": "(Network.BytesSentPerSec)", + "sum": { + "dataPoints": [ + { + "attributes": [ + { + "key": "nic", + "value": { + "stringValue": "Mellanox ConnectX-4 Lx Virtual Ethernet Adapter" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 2690078 + }, + { + "attributes": [ + { + "key": "nic", + "value": { + "stringValue": "Microsoft Hyper-V Network Adapter _3" + } + } + ], + "timeUnixNano": "1790731774528000000", + "asDouble": 2613057 + } + ], + "aggregationTemporality": 2, + "isMonotonic": true + }, + "metadata": [ + { + "key": "prometheus.type", + "value": { + "stringValue": "counter" + } + } + ] + } + ] + } + ] + } + ] +} \ No newline at end of file diff --git a/test/e2e-v2/cases/win/prometheus-windows_exporter/otel-collector-config.yaml b/test/e2e-v2/cases/win/prometheus-windows_exporter/otel-collector-config.yaml index 872c23d4291a..8b51751f353a 100644 --- a/test/e2e-v2/cases/win/prometheus-windows_exporter/otel-collector-config.yaml +++ b/test/e2e-v2/cases/win/prometheus-windows_exporter/otel-collector-config.yaml @@ -17,7 +17,7 @@ receivers: prometheus: config: scrape_configs: - - job_name: "windows-monitoring" # make sure to use this in the vm.yaml to filter only VM metrics + - job_name: "windows-monitoring" # windows.yaml keeps only this job scrape_interval: 10s static_configs: - targets: ["win-service:9182"] @@ -33,12 +33,12 @@ exporters: tls: insecure: true # Exports data to the console - logging: - logLevel: debug + debug: + verbosity: detailed service: pipelines: metrics: receivers: [prometheus] processors: [batch] - exporters: [otlp, logging] + exporters: [otlp, debug] diff --git a/test/e2e-v2/cases/win/win-cases.yaml b/test/e2e-v2/cases/win/win-cases.yaml index ebfd25223e0c..6f3aa260955c 100644 --- a/test/e2e-v2/cases/win/win-cases.yaml +++ b/test/e2e-v2/cases/win/win-cases.yaml @@ -18,5 +18,21 @@ cases: - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql service ls expected: expected/service.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_memory_total --service-name=10.211.55.3 + expected: expected/memory-total-4294967296.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_memory_used --service-name=10.211.55.3 + expected: expected/memory-used-3221225472.yml - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_memory_virtual_memory_total --service-name=10.211.55.3 + expected: expected/commit-limit-8838758400.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_memory_virtual_memory_percentage --service-name=10.211.55.3 + expected: expected/commit-percentage-50.yml + # A recording of windows_exporter 0.31.8 through OpenTelemetry Collector 0.158.0, which names the host + # only in `server.address`. + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_memory_total --service-name=172.25.0.1 + expected: expected/memory-total-8584425472.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_memory_used --service-name=172.25.0.1 + expected: expected/memory-used-2794070016.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_memory_virtual_memory_total --service-name=172.25.0.1 + expected: expected/commit-limit-10597691392.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_win_memory_virtual_memory_percentage --service-name=172.25.0.1 expected: expected/metrics-has-value.yml diff --git a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/collector/logs/v1/logs_service.proto b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/collector/logs/v1/logs_service.proto index 8260d8aaeb82..8be5cf75ef17 100644 --- a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/collector/logs/v1/logs_service.proto +++ b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/collector/logs/v1/logs_service.proto @@ -28,8 +28,6 @@ option go_package = "go.opentelemetry.io/proto/otlp/collector/logs/v1"; // OpenTelemetry and an collector, or between an collector and a central collector (in this // case logs are sent/received to/from multiple Applications). service LogsService { - // For performance reasons, it is recommended to keep this RPC - // alive for the entire life of the application. rpc Export(ExportLogsServiceRequest) returns (ExportLogsServiceResponse) {} } diff --git a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/collector/metrics/v1/metrics_service.proto b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/collector/metrics/v1/metrics_service.proto index dd48f1ad3a16..bc0242844151 100644 --- a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/collector/metrics/v1/metrics_service.proto +++ b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/collector/metrics/v1/metrics_service.proto @@ -28,8 +28,6 @@ option go_package = "go.opentelemetry.io/proto/otlp/collector/metrics/v1"; // instrumented with OpenTelemetry and a collector, or between a collector and a // central collector. service MetricsService { - // For performance reasons, it is recommended to keep this RPC - // alive for the entire life of the application. rpc Export(ExportMetricsServiceRequest) returns (ExportMetricsServiceResponse) {} } diff --git a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/collector/trace/v1/trace_service.proto b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/collector/trace/v1/trace_service.proto index d6fe67f9e553..efbbedbe4545 100644 --- a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/collector/trace/v1/trace_service.proto +++ b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/collector/trace/v1/trace_service.proto @@ -28,8 +28,6 @@ option go_package = "go.opentelemetry.io/proto/otlp/collector/trace/v1"; // OpenTelemetry and a collector, or between a collector and a central collector (in this // case spans are sent/received to/from multiple Applications). service TraceService { - // For performance reasons, it is recommended to keep this RPC - // alive for the entire life of the application. rpc Export(ExportTraceServiceRequest) returns (ExportTraceServiceResponse) {} } diff --git a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/common/v1/common.proto b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/common/v1/common.proto index d233677c109a..85bd3f2c00ac 100644 --- a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/common/v1/common.proto +++ b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/common/v1/common.proto @@ -22,7 +22,7 @@ option java_package = "io.opentelemetry.proto.common.v1"; option java_outer_classname = "CommonProto"; option go_package = "go.opentelemetry.io/proto/otlp/common/v1"; -// AnyValue is used to represent any type of attribute value. AnyValue may contain a +// Represents any type of attribute value. AnyValue may contain a // primitive value such as a string or integer or it may contain an arbitrary nested // object containing arrays, key-value lists and primitives. message AnyValue { @@ -36,6 +36,19 @@ message AnyValue { ArrayValue array_value = 5; KeyValueList kvlist_value = 6; bytes bytes_value = 7; + // Reference to the string value in ProfilesDictionary.string_table. + // + // Note: This is currently used exclusively in the Profiling signal. + // Implementers of OTLP receivers for signals other than Profiling should + // treat the presence of this value as a non-fatal issue. + // Log an error or warning indicating an unexpected field intended for the + // Profiling signal and process the data as if this value were absent or + // empty, ignoring its semantic content for the non-Profiling signal. + // + // Status: [Alpha] + // + // [Since v1.10.0] + int32 string_value_strindex = 8; } } @@ -54,24 +67,94 @@ message ArrayValue { message KeyValueList { // A collection of key/value pairs of key-value pairs. The list may be empty (may // contain 0 elements). + // // The keys MUST be unique (it is not allowed to have more than one // value with the same key). + // The behavior of software that receives duplicated keys can be unpredictable. repeated KeyValue values = 1; } -// KeyValue is a key-value pair that is used to store Span attributes, Link +// Represents a key-value pair that is used to store Span attributes, Link // attributes, etc. message KeyValue { + // The key name of the pair. + // key_strindex MUST NOT be set if key is used. string key = 1; + + // The value of the pair. AnyValue value = 2; + + // Reference to the string key in ProfilesDictionary.string_table. + // key MUST NOT be set if key_strindex is used. + // + // Note: This is currently used exclusively in the Profiling signal. + // Implementers of OTLP receivers for signals other than Profiling should + // treat the presence of this key as a non-fatal issue. + // Log an error or warning indicating an unexpected field intended for the + // Profiling signal and process the data as if this value were absent or + // empty, ignoring its semantic content for the non-Profiling signal. + // + // Status: [Alpha] + // + // [Since v1.10.0] + int32 key_strindex = 3; } // InstrumentationScope is a message representing the instrumentation scope information // such as the fully qualified name and version. message InstrumentationScope { + // A name denoting the Instrumentation scope. // An empty instrumentation scope name means the name is unknown. string name = 1; + + // Defines the version of the instrumentation scope. + // An empty instrumentation scope version means the version is unknown. string version = 2; + + // Additional attributes that describe the scope. [Optional]. + // Attribute keys MUST be unique (it is not allowed to have more than one + // attribute with the same key). + // The behavior of software that receives duplicated keys can be unpredictable. repeated KeyValue attributes = 3; + + // The number of attributes that were discarded. Attributes + // can be discarded because their keys are too long or because there are too many + // attributes. If this value is 0, then no attributes were dropped. uint32 dropped_attributes_count = 4; } + +// A reference to an Entity. +// Entity represents an object of interest associated with produced telemetry: e.g spans, metrics, profiles, or logs. +// +// Status: [Development] +// +// [Since v1.6.0] +message EntityRef { + // The Schema URL, if known. This is the identifier of the Schema that the entity data + // is recorded in. To learn more about Schema URL see + // https://opentelemetry.io/docs/specs/otel/schemas/#schema-url + // + // This schema_url applies to the data in this message and to the Resource attributes + // referenced by id_keys and description_keys. + // TODO: discuss if we are happy with this somewhat complicated definition of what + // the schema_url applies to. + // + // This field obsoletes the schema_url field in ResourceMetrics/ResourceSpans/ResourceLogs. + string schema_url = 1; + + // Defines the type of the entity. MUST not change during the lifetime of the entity. + // For example: "service" or "host". This field is required and MUST not be empty + // for valid entities. + string type = 2; + + // Attribute Keys that identify the entity. + // MUST not change during the lifetime of the entity. The Id must contain at least one attribute. + // These keys MUST exist in the containing {message}.attributes. + repeated string id_keys = 3; + + // Descriptive (non-identifying) attribute keys of the entity. + // MAY change over the lifetime of the entity. MAY be empty. + // These attribute keys are not part of entity's identity. + // These keys MUST exist in the containing {message}.attributes. + repeated string description_keys = 4; +} \ No newline at end of file diff --git a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/logs/v1/logs.proto b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/logs/v1/logs.proto index 9d0e376cad93..612bdadeacf2 100644 --- a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/logs/v1/logs.proto +++ b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/logs/v1/logs.proto @@ -55,6 +55,10 @@ message ResourceLogs { // A list of ScopeLogs that originate from a resource. repeated ScopeLogs scope_logs = 2; + // The Schema URL, if known. This is the identifier of the Schema that the resource data + // is recorded in. Notably, the last part of the URL path is the version number of the + // schema: http[s]://server[:port]/path/. To learn more about Schema URL see + // https://opentelemetry.io/docs/specs/otel/schemas/#schema-url // This schema_url applies to the data in the "resource" field. It does not apply // to the data in the "scope_logs" field which have their own schema_url field. string schema_url = 3; @@ -70,13 +74,17 @@ message ScopeLogs { // A list of log records. repeated LogRecord log_records = 2; - // This schema_url applies to all logs in the "logs" field. + // The Schema URL, if known. This is the identifier of the Schema that the log data + // is recorded in. Notably, the last part of the URL path is the version number of the + // schema: http[s]://server[:port]/path/. To learn more about Schema URL see + // https://opentelemetry.io/docs/specs/otel/schemas/#schema-url + // This schema_url applies to the data in the "scope" field and all logs in the + // "log_records" field. string schema_url = 3; } // Possible values for LogRecord.SeverityNumber. enum SeverityNumber { - // UNSPECIFIED is the default SeverityNumber, it MUST NOT be used. SEVERITY_NUMBER_UNSPECIFIED = 0; SEVERITY_NUMBER_TRACE = 1; SEVERITY_NUMBER_TRACE2 = 2; @@ -104,10 +112,23 @@ enum SeverityNumber { SEVERITY_NUMBER_FATAL4 = 24; } -// Masks for LogRecord.flags field. +// LogRecordFlags represents constants used to interpret the +// LogRecord.flags field, which is protobuf 'fixed32' type and is to +// be used as bit-fields. Each non-zero value defined in this enum is +// a bit-mask. To extract the bit-field, for example, use an +// expression like: +// +// (logRecord.flags & LOG_RECORD_FLAGS_TRACE_FLAGS_MASK) +// enum LogRecordFlags { - LOG_RECORD_FLAG_UNSPECIFIED = 0; - LOG_RECORD_FLAG_TRACE_FLAGS_MASK = 0x000000FF; + // The zero value for the enum. Should not be used for comparisons. + // Instead use bitwise "and" with the appropriate mask as shown above. + LOG_RECORD_FLAGS_DO_NOT_USE = 0; + + // Bits 0-7 are used for trace flags. + LOG_RECORD_FLAGS_TRACE_FLAGS_MASK = 0x000000FF; + + // Bits 8-31 are reserved for future use. } // A log record according to OpenTelemetry Log Data Model: @@ -153,6 +174,7 @@ message LogRecord { // Additional attributes that describe the specific event occurrence. [Optional]. // Attribute keys MUST be unique (it is not allowed to have more than one // attribute with the same key). + // The behavior of software that receives duplicated keys can be unpredictable. repeated opentelemetry.proto.common.v1.KeyValue attributes = 6; uint32 dropped_attributes_count = 7; @@ -160,18 +182,47 @@ message LogRecord { // defined in W3C Trace Context specification. 24 most significant bits are reserved // and must be set to 0. Readers must not assume that 24 most significant bits // will be zero and must correctly mask the bits when reading 8-bit trace flag (use - // flags & TRACE_FLAGS_MASK). [Optional]. + // flags & LOG_RECORD_FLAGS_TRACE_FLAGS_MASK). [Optional]. fixed32 flags = 8; // A unique identifier for a trace. All logs from the same trace share - // the same `trace_id`. The ID is a 16-byte array. An ID with all zeroes - // is considered invalid. Can be set for logs that are part of request processing - // and have an assigned trace id. [Optional]. + // the same `trace_id`. The ID is a 16-byte array. An ID with all zeroes OR + // of length other than 16 bytes is considered invalid (empty string in OTLP/JSON + // is zero-length and thus is also invalid). + // + // This field is optional. + // + // The receivers SHOULD assume that the log record is not associated with a + // trace if any of the following is true: + // - the field is not present, + // - the field contains an invalid value. bytes trace_id = 9; // A unique identifier for a span within a trace, assigned when the span - // is created. The ID is an 8-byte array. An ID with all zeroes is considered - // invalid. Can be set for logs that are part of a particular processing span. - // If span_id is present trace_id SHOULD be also present. [Optional]. + // is created. The ID is an 8-byte array. An ID with all zeroes OR of length + // other than 8 bytes is considered invalid (empty string in OTLP/JSON + // is zero-length and thus is also invalid). + // + // This field is optional. If the sender specifies a valid span_id then it SHOULD also + // specify a valid trace_id. + // + // The receivers SHOULD assume that the log record is not associated with a + // span if any of the following is true: + // - the field is not present, + // - the field contains an invalid value. bytes span_id = 10; + + // A unique identifier of event category/type. + // All events with the same event_name are expected to conform to the same + // schema for both their attributes and their body. + // + // Recommended to be fully qualified and short (no longer than 256 characters). + // + // Presence of event_name on the log record identifies this record + // as an event. + // + // [Optional]. + // + // [Since v1.5.0] + string event_name = 12; } diff --git a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/metrics/v1/metrics.proto b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/metrics/v1/metrics.proto index 101c8ccbbf56..3f06bb83d161 100644 --- a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/metrics/v1/metrics.proto +++ b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/metrics/v1/metrics.proto @@ -29,6 +29,24 @@ option go_package = "go.opentelemetry.io/proto/otlp/metrics/v1"; // storage, OR can be embedded by other protocols that transfer OTLP metrics // data but do not implement the OTLP protocol. // +// MetricsData +// └─── ResourceMetrics +// ├── Resource +// ├── SchemaURL +// └── ScopeMetrics +// ├── Scope +// ├── SchemaURL +// └── Metric +// ├── Name +// ├── Description +// ├── Unit +// └── data +// ├── Gauge +// ├── Sum +// ├── Histogram +// ├── ExponentialHistogram +// └── Summary +// // The main difference between this message and collector protocol is that // in this message there will not be any "control" or "metadata" specific to // OTLP protocol. @@ -55,6 +73,10 @@ message ResourceMetrics { // A list of metrics that originate from a resource. repeated ScopeMetrics scope_metrics = 2; + // The Schema URL, if known. This is the identifier of the Schema that the resource data + // is recorded in. Notably, the last part of the URL path is the version number of the + // schema: http[s]://server[:port]/path/. To learn more about Schema URL see + // https://opentelemetry.io/docs/specs/otel/schemas/#schema-url // This schema_url applies to the data in the "resource" field. It does not apply // to the data in the "scope_metrics" field which have their own schema_url field. string schema_url = 3; @@ -70,7 +92,12 @@ message ScopeMetrics { // A list of metrics that originate from an instrumentation library. repeated Metric metrics = 2; - // This schema_url applies to all metrics in the "metrics" field. + // The Schema URL, if known. This is the identifier of the Schema that the metric data + // is recorded in. Notably, the last part of the URL path is the version number of the + // schema: http[s]://server[:port]/path/. To learn more about Schema URL see + // https://opentelemetry.io/docs/specs/otel/schemas/#schema-url + // This schema_url applies to the data in the "scope" field and all metrics in the + // "metrics" field. string schema_url = 3; } @@ -79,7 +106,6 @@ message ScopeMetrics { // // https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/data-model.md // -// // The data model and relation between entities is shown in the // diagram below. Here, "DataPoint" is the term used to refer to any // one of the specific data point value types, and "points" is the term used @@ -91,7 +117,7 @@ message ScopeMetrics { // - DataPoint contains timestamps, attributes, and one of the possible value type // fields. // -// Metric +// Metric // +------------+ // |name | // |description | @@ -162,18 +188,18 @@ message ScopeMetrics { message Metric { reserved 4, 6, 8; - // name of the metric, including its DNS name prefix. It must be unique. + // The name of the metric. string name = 1; - // description of the metric, which can be used in documentation. + // A description of the metric, which can be used in documentation. string description = 2; - // unit in which the metric value is reported. Follows the format - // described by http://unitsofmeasure.org/ucum.html. + // The unit in which the metric value is reported. Follows the format + // described by https://ucum.org/ucum and https://units-of-measurement.org/ string unit = 3; // Data determines the aggregation type (if any) of the metric, what is the - // reported value type for the data points, as well as the relatationship to + // reported value type for the data points, as well as the relationship to // the time interval over which they are reported. oneof data { Gauge gauge = 5; @@ -182,6 +208,18 @@ message Metric { ExponentialHistogram exponential_histogram = 10; Summary summary = 11; } + + // Additional metadata attributes that describe the metric. [Optional]. + // Attributes are non-identifying. + // Consumers SHOULD NOT need to be aware of these attributes. + // These attributes MAY be used to encode information allowing + // for lossless roundtrip translation to / from another data model. + // Attribute keys MUST be unique (it is not allowed to have more than one + // attribute with the same key). + // The behavior of software that receives duplicated keys can be unpredictable. + // + // [Since v1.2.0] + repeated opentelemetry.proto.common.v1.KeyValue metadata = 12; } // Gauge represents the type of a scalar metric that always exports the @@ -194,25 +232,31 @@ message Metric { // AggregationTemporality is not included. Consequently, this also means // "StartTimeUnixNano" is ignored for all data points. message Gauge { + // The time series data points. + // Note: Multiple time series may be included (same timestamp, different attributes). repeated NumberDataPoint data_points = 1; } // Sum represents the type of a scalar metric that is calculated as a sum of all // reported measurements over a time interval. message Sum { + // The time series data points. + // Note: Multiple time series may be included (same timestamp, different attributes). repeated NumberDataPoint data_points = 1; // aggregation_temporality describes if the aggregator reports delta changes // since last report time, or cumulative changes since a fixed start time. AggregationTemporality aggregation_temporality = 2; - // If "true" means that the sum is monotonic. + // Represents whether the sum is monotonic. bool is_monotonic = 3; } // Histogram represents the type of a metric that is calculated by aggregating // as a Histogram of all reported measurements over a time interval. message Histogram { + // The time series data points. + // Note: Multiple time series may be included (same timestamp, different attributes). repeated HistogramDataPoint data_points = 1; // aggregation_temporality describes if the aggregator reports delta changes @@ -223,6 +267,8 @@ message Histogram { // ExponentialHistogram represents the type of a metric that is calculated by aggregating // as a ExponentialHistogram of all reported double measurements over a time interval. message ExponentialHistogram { + // The time series data points. + // Note: Multiple time series may be included (same timestamp, different attributes). repeated ExponentialHistogramDataPoint data_points = 1; // aggregation_temporality describes if the aggregator reports delta changes @@ -232,11 +278,16 @@ message ExponentialHistogram { // Summary metric data are used to convey quantile summaries, // a Prometheus (see: https://prometheus.io/docs/concepts/metric_types/#summary) -// and OpenMetrics (see: https://github.com/OpenObservability/OpenMetrics/blob/4dbf6075567ab43296eed941037c12951faafb92/protos/prometheus.proto#L45) +// and OpenMetrics (see: https://github.com/prometheus/OpenMetrics/blob/4dbf6075567ab43296eed941037c12951faafb92/protos/prometheus.proto#L45) // data type. These data points cannot always be merged in a meaningful way. // While they can be useful in some applications, histogram data points are // recommended for new applications. +// Summary metrics do not have an aggregation temporality field. This is +// because the count and sum fields of a SummaryDataPoint are assumed to be +// cumulative values. message Summary { + // The time series data points. + // Note: Multiple time series may be included (same timestamp, different attributes). repeated SummaryDataPoint data_points = 1; } @@ -316,15 +367,17 @@ enum AggregationTemporality { // enum is a bit-mask. To test the presence of a single flag in the flags of // a data point, for example, use an expression like: // -// (point.flags & FLAG_NO_RECORDED_VALUE) == FLAG_NO_RECORDED_VALUE +// (point.flags & DATA_POINT_FLAGS_NO_RECORDED_VALUE_MASK) == DATA_POINT_FLAGS_NO_RECORDED_VALUE_MASK // enum DataPointFlags { - FLAG_NONE = 0; + // The zero value for the enum. Should not be used for comparisons. + // Instead use bitwise "and" with the appropriate mask as shown above. + DATA_POINT_FLAGS_DO_NOT_USE = 0; // This DataPoint is valid but has no recorded value. This value // SHOULD be used to reflect explicitly missing data in a series, as // for an equivalent to the Prometheus "staleness marker". - FLAG_NO_RECORDED_VALUE = 1; + DATA_POINT_FLAGS_NO_RECORDED_VALUE_MASK = 1; // Bits 2-31 are reserved for future use. } @@ -338,6 +391,7 @@ message NumberDataPoint { // where this point belongs. The list may be empty (may contain 0 elements). // Attribute keys MUST be unique (it is not allowed to have more than one // attribute with the same key). + // The behavior of software that receives duplicated keys can be unpredictable. repeated opentelemetry.proto.common.v1.KeyValue attributes = 7; // StartTimeUnixNano is optional but strongly encouraged, see the @@ -386,6 +440,7 @@ message HistogramDataPoint { // where this point belongs. The list may be empty (may contain 0 elements). // Attribute keys MUST be unique (it is not allowed to have more than one // attribute with the same key). + // The behavior of software that receives duplicated keys can be unpredictable. repeated opentelemetry.proto.common.v1.KeyValue attributes = 9; // StartTimeUnixNano is optional but strongly encouraged, see the @@ -413,7 +468,7 @@ message HistogramDataPoint { // events, and is assumed to be monotonic over the values of these events. // Negative events *can* be recorded, but sum should not be filled out when // doing so. This is specifically to enforce compatibility w/ OpenMetrics, - // see: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#histogram + // see: https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#histogram optional double sum = 5; // bucket_counts is an optional field contains the count values of histogram @@ -422,7 +477,9 @@ message HistogramDataPoint { // The sum of the bucket_counts must equal the value in the count field. // // The number of elements in bucket_counts array must be by one greater than - // the number of elements in explicit_bounds array. + // the number of elements in explicit_bounds array. The exception to this rule + // is when the length of bucket_counts is 0, then the length of explicit_bounds + // must also be 0. repeated fixed64 bucket_counts = 6; // explicit_bounds specifies buckets with explicitly defined bounds for values. @@ -438,6 +495,9 @@ message HistogramDataPoint { // Histogram buckets are inclusive of their upper boundary, except the last // bucket where the boundary is at infinity. This format is intentionally // compatible with the OpenMetrics histogram definition. + // + // If bucket_counts length is 0 then explicit_bounds length must also be 0, + // otherwise the data point is invalid. repeated double explicit_bounds = 7; // (Optional) List of exemplars collected from @@ -465,6 +525,7 @@ message ExponentialHistogramDataPoint { // where this point belongs. The list may be empty (may contain 0 elements). // Attribute keys MUST be unique (it is not allowed to have more than one // attribute with the same key). + // The behavior of software that receives duplicated keys can be unpredictable. repeated opentelemetry.proto.common.v1.KeyValue attributes = 1; // StartTimeUnixNano is optional but strongly encouraged, see the @@ -480,21 +541,21 @@ message ExponentialHistogramDataPoint { // 1970. fixed64 time_unix_nano = 3; - // count is the number of values in the population. Must be + // The number of values in the population. Must be // non-negative. This value must be equal to the sum of the "bucket_counts" // values in the positive and negative Buckets plus the "zero_count" field. fixed64 count = 4; - // sum of the values in the population. If count is zero then this field + // The sum of the values in the population. If count is zero then this field // must be zero. // // Note: Sum should only be filled out when measuring non-negative discrete // events, and is assumed to be monotonic over the values of these events. // Negative events *can* be recorded, but sum should not be filled out when // doing so. This is specifically to enforce compatibility w/ OpenMetrics, - // see: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#histogram + // see: https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#histogram optional double sum = 5; - + // scale describes the resolution of the histogram. Boundaries are // located at powers of the base, where: // @@ -512,7 +573,7 @@ message ExponentialHistogramDataPoint { // values depend on the range of the data. sint32 scale = 6; - // zero_count is the count of values that are either exactly zero or + // The count of values that are either exactly zero or // within the region considered zero by the instrumentation at the // tolerated degree of precision. This bucket stores values that // cannot be expressed using the standard exponential formula as @@ -531,14 +592,14 @@ message ExponentialHistogramDataPoint { // Buckets are a set of bucket counts, encoded in a contiguous array // of counts. message Buckets { - // Offset is the bucket index of the first entry in the bucket_counts array. - // + // The bucket index of the first entry in the bucket_counts array. + // // Note: This uses a varint encoding as a simple form of compression. sint32 offset = 1; - // Count is an array of counts, where count[i] carries the count - // of the bucket at index (offset+i). count[i] is the count of - // values greater than base^(offset+i) and less or equal to than + // An array of count values, where bucket_counts[i] carries + // the count of the bucket at index (offset+i). bucket_counts[i] is the count + // of values greater than base^(offset+i) and less than or equal to // base^(offset+i+1). // // Note: By contrast, the explicit HistogramDataPoint uses @@ -546,7 +607,7 @@ message ExponentialHistogramDataPoint { // especially zeros, so uint64 has been selected to ensure // varint encoding. repeated uint64 bucket_counts = 2; - } + } // Flags that apply to this specific data point. See DataPointFlags // for the available flags and their meaning. @@ -556,15 +617,24 @@ message ExponentialHistogramDataPoint { // measurements that were used to form the data point repeated Exemplar exemplars = 11; - // min is the minimum value over (start_time, end_time]. + // The minimum value over (start_time, end_time]. optional double min = 12; - // max is the maximum value over (start_time, end_time]. + // The maximum value over (start_time, end_time]. optional double max = 13; + + // ZeroThreshold may be optionally set to convey the width of the zero + // region. Where the zero region is defined as the closed interval + // [-ZeroThreshold, ZeroThreshold]. + // When ZeroThreshold is 0, zero count bucket stores values that cannot be + // expressed using the standard exponential formula as well as values that + // have been rounded to zero. + double zero_threshold = 14; } // SummaryDataPoint is a single data point in a timeseries that describes the -// time-varying values of a Summary metric. +// time-varying values of a Summary metric. The count and sum fields represent +// cumulative values. message SummaryDataPoint { reserved 1; @@ -572,6 +642,7 @@ message SummaryDataPoint { // where this point belongs. The list may be empty (may contain 0 elements). // Attribute keys MUST be unique (it is not allowed to have more than one // attribute with the same key). + // The behavior of software that receives duplicated keys can be unpredictable. repeated opentelemetry.proto.common.v1.KeyValue attributes = 7; // StartTimeUnixNano is optional but strongly encouraged, see the @@ -597,7 +668,7 @@ message SummaryDataPoint { // events, and is assumed to be monotonic over the values of these events. // Negative events *can* be recorded, but sum should not be filled out when // doing so. This is specifically to enforce compatibility w/ OpenMetrics, - // see: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#summary + // see: https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#summary double sum = 5; // Represents the value at a given quantile of a distribution. diff --git a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/resource/v1/resource.proto b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/resource/v1/resource.proto index 6637560bc354..118bfed175d5 100644 --- a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/resource/v1/resource.proto +++ b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/resource/v1/resource.proto @@ -29,9 +29,19 @@ message Resource { // Set of attributes that describe the resource. // Attribute keys MUST be unique (it is not allowed to have more than one // attribute with the same key). + // The behavior of software that receives duplicated keys can be unpredictable. repeated opentelemetry.proto.common.v1.KeyValue attributes = 1; - // dropped_attributes_count is the number of dropped attributes. If the value is 0, then + // The number of dropped attributes. If the value is 0, then // no attributes were dropped. uint32 dropped_attributes_count = 2; + + // Set of entities that participate in this Resource. + // + // Note: keys in the references MUST exist in attributes of this message. + // + // Status: [Development] + // + // [Since v1.6.0] + repeated opentelemetry.proto.common.v1.EntityRef entity_refs = 3; } diff --git a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/trace/v1/trace.proto b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/trace/v1/trace.proto index 5903550742dc..235b54e8e78b 100644 --- a/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/trace/v1/trace.proto +++ b/test/e2e-v2/java-test-service/opentelemetry-proto/src/main/proto/opentelemetry/proto/trace/v1/trace.proto @@ -55,6 +55,10 @@ message ResourceSpans { // A list of ScopeSpans that originate from a resource. repeated ScopeSpans scope_spans = 2; + // The Schema URL, if known. This is the identifier of the Schema that the resource data + // is recorded in. Notably, the last part of the URL path is the version number of the + // schema: http[s]://server[:port]/path/. To learn more about Schema URL see + // https://opentelemetry.io/docs/specs/otel/schemas/#schema-url // This schema_url applies to the data in the "resource" field. It does not apply // to the data in the "scope_spans" field which have their own schema_url field. string schema_url = 3; @@ -70,7 +74,12 @@ message ScopeSpans { // A list of Spans that originate from an instrumentation scope. repeated Span spans = 2; - // This schema_url applies to all spans and span events in the "spans" field. + // The Schema URL, if known. This is the identifier of the Schema that the span data + // is recorded in. Notably, the last part of the URL path is the version number of the + // schema: http[s]://server[:port]/path/. To learn more about Schema URL see + // https://opentelemetry.io/docs/specs/otel/schemas/#schema-url + // This schema_url applies to the data in the "scope" field and all spans and span + // events in the "spans" field. string schema_url = 3; } @@ -79,21 +88,17 @@ message ScopeSpans { // The next available field id is 17. message Span { // A unique identifier for a trace. All spans from the same trace share - // the same `trace_id`. The ID is a 16-byte array. An ID with all zeroes - // is considered invalid. - // - // This field is semantically required. Receiver should generate new - // random trace_id if empty or invalid trace_id was received. + // the same `trace_id`. The ID is a 16-byte array. An ID with all zeroes OR + // of length other than 16 bytes is considered invalid (empty string in OTLP/JSON + // is zero-length and thus is also invalid). // // This field is required. bytes trace_id = 1; // A unique identifier for a span within a trace, assigned when the span - // is created. The ID is an 8-byte array. An ID with all zeroes is considered - // invalid. - // - // This field is semantically required. Receiver should generate new - // random span_id if empty or invalid span_id was received. + // is created. The ID is an 8-byte array. An ID with all zeroes OR of length + // other than 8 bytes is considered invalid (empty string in OTLP/JSON + // is zero-length and thus is also invalid). // // This field is required. bytes span_id = 2; @@ -107,6 +112,31 @@ message Span { // field must be empty. The ID is an 8-byte array. bytes parent_span_id = 4; + // Flags, a bit field. + // + // Bits 0-7 (8 least significant bits) are the trace flags as defined in W3C Trace + // Context specification. To read the 8-bit W3C trace flag, use + // `flags & SPAN_FLAGS_TRACE_FLAGS_MASK`. + // + // See https://www.w3.org/TR/trace-context-2/#trace-flags for the flag definitions. + // + // Bits 8 and 9 represent the 3 states of whether a span's parent + // is remote. The states are (unknown, is not remote, is remote). + // To read whether the value is known, use `(flags & SPAN_FLAGS_CONTEXT_HAS_IS_REMOTE_MASK) != 0`. + // To read whether the span is remote, use `(flags & SPAN_FLAGS_CONTEXT_IS_REMOTE_MASK) != 0`. + // + // When creating span messages, if the message is logically forwarded from another source + // with an equivalent flags fields (i.e., usually another OTLP span message), the field SHOULD + // be copied as-is. If creating from a source that does not have an equivalent flags field + // (such as a runtime representation of an OpenTelemetry span), the high 22 bits MUST + // be set to zero. + // Readers MUST NOT assume that bits 10-31 (22 most significant bits) will be zero. + // + // [Optional]. + // + // [Since v1.1.0] + fixed32 flags = 16; + // A description of the span's operation. // // For example, the name can be a qualified method name or a file name @@ -155,7 +185,7 @@ message Span { // and `SERVER` (callee) to identify queueing latency associated with the span. SpanKind kind = 6; - // start_time_unix_nano is the start time of the span. On the client side, this is the time + // The start time of the span. On the client side, this is the time // kept by the local machine where the span execution starts. On the server side, this // is the time when the server's application handler starts running. // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January 1970. @@ -163,7 +193,7 @@ message Span { // This field is semantically required and it is expected that end_time >= start_time. fixed64 start_time_unix_nano = 7; - // end_time_unix_nano is the end time of the span. On the client side, this is the time + // The end time of the span. On the client side, this is the time // kept by the local machine where the span execution ends. On the server side, this // is the time when the server application handler stops running. // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January 1970. @@ -171,21 +201,20 @@ message Span { // This field is semantically required and it is expected that end_time >= start_time. fixed64 end_time_unix_nano = 8; - // attributes is a collection of key/value pairs. Note, global attributes - // like server name can be set using the resource API. Examples of attributes: + // A collection of key/value pairs. Note, global attributes + // like service name can be set using the resource API. Examples of attributes: // - // "/http/user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36" - // "/http/server_latency": 300 - // "abc.com/myattribute": true - // "abc.com/score": 10.239 + // "user_agent.original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36" + // "http.response.status_code": 200 + // "com.example.myattribute": true + // "com.example.score": 10.239 // - // The OpenTelemetry API specification further restricts the allowed value types: - // https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/common/README.md#attribute // Attribute keys MUST be unique (it is not allowed to have more than one // attribute with the same key). + // The behavior of software that receives duplicated keys can be unpredictable. repeated opentelemetry.proto.common.v1.KeyValue attributes = 9; - // dropped_attributes_count is the number of attributes that were discarded. Attributes + // The number of attributes that were discarded. Attributes // can be discarded because their keys are too long or because there are too many // attributes. If this value is 0, then no attributes were dropped. uint32 dropped_attributes_count = 10; @@ -193,27 +222,28 @@ message Span { // Event is a time-stamped annotation of the span, consisting of user-supplied // text description and key-value pairs. message Event { - // time_unix_nano is the time the event occurred. + // The time the event occurred. fixed64 time_unix_nano = 1; - // name of the event. + // The name of the event. // This field is semantically required to be set to non-empty string. string name = 2; - // attributes is a collection of attribute key/value pairs on the event. + // A collection of attribute key/value pairs on the event. // Attribute keys MUST be unique (it is not allowed to have more than one // attribute with the same key). + // The behavior of software that receives duplicated keys can be unpredictable. repeated opentelemetry.proto.common.v1.KeyValue attributes = 3; - // dropped_attributes_count is the number of dropped attributes. If the value is 0, + // The number of dropped attributes. If the value is 0, // then no attributes were dropped. uint32 dropped_attributes_count = 4; } - // events is a collection of Event items. + // A collection of Event items. repeated Event events = 11; - // dropped_events_count is the number of dropped events. If the value is 0, then no + // The number of dropped events. If the value is 0, then no // events were dropped. uint32 dropped_events_count = 12; @@ -232,21 +262,43 @@ message Span { // The trace_state associated with the link. string trace_state = 3; - // attributes is a collection of attribute key/value pairs on the link. + // A collection of attribute key/value pairs on the link. // Attribute keys MUST be unique (it is not allowed to have more than one // attribute with the same key). + // The behavior of software that receives duplicated keys can be unpredictable. repeated opentelemetry.proto.common.v1.KeyValue attributes = 4; - // dropped_attributes_count is the number of dropped attributes. If the value is 0, + // The number of dropped attributes. If the value is 0, // then no attributes were dropped. uint32 dropped_attributes_count = 5; + + // Flags, a bit field. + // + // Bits 0-7 (8 least significant bits) are the trace flags as defined in W3C Trace + // Context specification. To read the 8-bit W3C trace flag, use + // `flags & SPAN_FLAGS_TRACE_FLAGS_MASK`. + // + // See https://www.w3.org/TR/trace-context-2/#trace-flags for the flag definitions. + // + // Bits 8 and 9 represent the 3 states of whether the link is remote. + // The states are (unknown, is not remote, is remote). + // To read whether the value is known, use `(flags & SPAN_FLAGS_CONTEXT_HAS_IS_REMOTE_MASK) != 0`. + // To read whether the link is remote, use `(flags & SPAN_FLAGS_CONTEXT_IS_REMOTE_MASK) != 0`. + // + // Readers MUST NOT assume that bits 10-31 (22 most significant bits) will be zero. + // When creating new spans, bits 10-31 (most-significant 22-bits) MUST be zero. + // + // [Optional]. + // + // [Since v1.1.0] + fixed32 flags = 6; } - // links is a collection of Links, which are references from this span to a span + // A collection of Links, which are references from this span to a span // in the same or different trace. repeated Link links = 13; - // dropped_links_count is the number of dropped links after the maximum size was + // The number of dropped links after the maximum size was // enforced. If this value is 0, then no links were dropped. uint32 dropped_links_count = 14; @@ -278,3 +330,39 @@ message Status { // The status code. StatusCode code = 3; } + +// SpanFlags represents constants used to interpret the +// Span.flags field, which is protobuf 'fixed32' type and is to +// be used as bit-fields. Each non-zero value defined in this enum is +// a bit-mask. To extract the bit-field, for example, use an +// expression like: +// +// (span.flags & SPAN_FLAGS_TRACE_FLAGS_MASK) +// +// See https://www.w3.org/TR/trace-context-2/#trace-flags for the flag definitions. +// +// Note that Span flags were introduced in version 1.1 of the +// OpenTelemetry protocol. Older Span producers do not set this +// field, consequently consumers should not rely on the absence of a +// particular flag bit to indicate the presence of a particular feature. +// +// [Since v1.1.0] +enum SpanFlags { + // The zero value for the enum. Should not be used for comparisons. + // Instead use bitwise "and" with the appropriate mask as shown above. + SPAN_FLAGS_DO_NOT_USE = 0; + + // Bits 0-7 are used for trace flags. + SPAN_FLAGS_TRACE_FLAGS_MASK = 0x000000FF; + + // Bits 8 and 9 are used to indicate that the parent span or link span is remote. + // Bit 8 (`HAS_IS_REMOTE`) indicates whether the value is known. + // Bit 9 (`IS_REMOTE`) indicates whether the span or link is remote. + // + // [Since v1.2.0] + SPAN_FLAGS_CONTEXT_HAS_IS_REMOTE_MASK = 0x00000100; + // [Since v1.2.0] + SPAN_FLAGS_CONTEXT_IS_REMOTE_MASK = 0x00000200; + + // Bits 10-31 are reserved for future use. +}