Fix Windows monitoring on windows_exporter 0.31.0+ and OTel Collector 0.127.0+, and CPU total counting overlapping modes - #14116
Merged
Conversation
wu-sheng
added
backend
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
Review effort: Lite
Findings: 2
Open (3)
grep -q -- '-> Name: system.cpu.time'check… · Newdate +%s). If… · New
What changed in this PR
Updates Windows host monitoring rules and fixtures to support windows_exporter 0.31.0+ memory metrics and fix CPU “total” double-counting due to overlapping Windows CPU modes, with expanded e2e/MAL coverage and doc updates.
Changes:
- Switch Windows memory rules from deprecated
cs/osmetrics tomemorycollector metrics (windows_memory_*) and update test fixtures/expectations. - Fix CPU total calculation by summing only non-overlapping modes (
user+privilegedoruser+system) and add assertions for exact CPU values. - Update docs/CHANGES to reflect the new windows_exporter minimum version and corrected metric semantics.
| File | Description |
|---|---|
| test/e2e-v2/cases/win/win-cases.yaml | Adds e2e assertions for Windows memory total/used/commit metrics. |
| test/e2e-v2/cases/win/mock-data/otel-mock-metrics.json | Updates mock OTLP metrics to new windows_exporter memory collector names. |
| test/e2e-v2/cases/win/expected/memory-used-3221225472.yml | Adds expected output for meter_win_memory_used. |
| test/e2e-v2/cases/win/expected/memory-total-4294967296.yml | Adds expected output for meter_win_memory_total. |
| test/e2e-v2/cases/win/expected/commit-percentage-50.yml | Adds expected output for commit percentage metric. |
| test/e2e-v2/cases/win/expected/commit-limit-8838758400.yml | Adds expected output for commit limit metric. |
| test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/windows.yaml | Fixes CPU total mode overlap and updates Windows memory metric names/semantics. |
| test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/process-hostmetrics-windows.yaml | Corrects process memory utilization comment/assumptions (0–100 scale). |
| test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/process-hostmetrics-linux.yaml | Corrects process memory utilization comment/assumptions (0–100 scale). |
| test/e2e-v2/cases/vm/otel-hostmetrics/fixtures/windows-host.json | Updates fixture to include more realistic CPU/memory/pagefile/filesystem metrics. |
| test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-pagefile-percentage-25.yml | Adds exact-value expectation for pagefile usage percentage. |
| test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-memory-used-5368709120.yml | Adds exact-value expectation for memory used. |
| test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-filesystem-percentage-60.yml | Adds exact-value expectation for filesystem usage percentage. |
| test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-filehandles-45000.yml | Adds exact-value expectation for allocated file handles. |
| test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-total-80.yml | Adds exact-value expectation for total CPU usage. |
| test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-norm-20.yml | Adds exact-value expectation for normalized CPU usage. |
| test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-load1-50.yml | Adds exact-value expectation for 1m CPU load. |
| test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-cpu-cores-4.yml | Adds exact-value expectation for logical CPU count. |
| test/e2e-v2/cases/vm/otel-hostmetrics/expected/win-commit-percentage-50.yml | Adds exact-value expectation for commit percentage. |
| test/e2e-v2/cases/vm/otel-hostmetrics/e2e.yaml | Expands verification to assert concrete CPU/memory/pagefile/filesystem/handles values. |
| test/e2e-v2/cases/vm/otel-hostmetrics/docker-compose.yml | Makes fixture CPU counters advance over time for non-zero rate() assertions. |
| test/e2e-v2/cases/vm/otel-hostmetrics/assert-collector-output.sh | Tightens collector-output verification to scope checks to CPU metric block. |
| oap-server/server-starter/src/main/resources/otel-rules/windows.yaml | Mirrors Windows rule fixes into production rules. |
| oap-server/server-starter/src/main/resources/otel-rules/process-hostmetrics-windows.yaml | Mirrors corrected process memory utilization comment into production rules. |
| oap-server/server-starter/src/main/resources/otel-rules/process-hostmetrics-linux.yaml | Mirrors corrected process memory utilization comment into production rules. |
| oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/windows.data.yaml | Updates MAL inputs/expected for CPU mode overlap and new Windows memory metrics. |
| oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/process-hostmetrics-windows.data.yaml | Updates MAL test data to reflect 0–100 process memory utilization scale. |
| oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/process-hostmetrics-linux.data.yaml | Updates MAL test data to reflect 0–100 process memory utilization scale. |
| docs/en/setup/backend/backend-win-monitoring.md | Documents windows_exporter minimum version and clarifies CPU metric meaning. |
| docs/en/setup/backend/backend-vm-monitoring.md | Corrects Linux sockets_used source table entry. |
| docs/en/changes/changes.md | Records Windows monitoring fixes and the new windows_exporter version requirement. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
… 0.127.0+, and CPU total counting overlapping modes windows_exporter 0.31.0 removed the `cs` collector and the deprecated `os` memory metrics windows.yaml read, so memory total/available/used and the commit metrics were empty on current exporters. Read the `memory` collector's replacements instead; windows_exporter 0.29.0 or later is now required. Windows CPU modes overlap: windows_exporter's `privileged` includes `interrupt` and `dpc`, and the Collector's `system` includes `interrupt`. cpu_total_percentage summed every non-idle mode and counted that time twice; it now sums user + privileged / user + system. The OTel Collector's Prometheus receiver sends a scraped target's host only as `server.address` from 0.127.0, so vm.yaml and windows.yaml, which name a host by node_identifier_host_name, got no host and no service. The OTel receiver now falls back to `server.address` when neither `net.host.name` nor `host.name` is present. The otel-hostmetrics e2e replays growing system.cpu.time counters so rate() runs on real increases, and both Windows e2e cases assert exact values. The win e2e also replays a recording of windows_exporter 0.31.8 through OTel Collector 0.158.0, taken on a GitHub-hosted Windows runner; the e2e mock sender's OTLP proto is updated to v1.11.1 so its strict JSON parser accepts it. The documented Collector config uses the debug exporter, since the logging exporter was removed. Also correct the process-hostmetrics comment on the scale of process.memory.utilization, and the Telegraf row in the Linux metrics table.
wu-sheng
force-pushed
the
fix/windows-host-metrics
branch
from
September 30, 2026 01:52
8656824 to
091965c
Compare
wu-sheng
changed the title
1 task done
wankai123
approved these changes
Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix Windows monitoring on windows_exporter 0.31.0+ and OTel Collector 0.127.0+, and CPU total counting overlapping modes
Checking
windows.yamlagainst the current windows_exporter and OpenTelemetry Collector found three bugs. Each was confirmed on real output, not only from source: windows_exporter 0.31.8, 0.31.0 and 0.30.9 were run on a GitHub-hostedwindows-latestrunner with their default collectors, beside OpenTelemetry Collector contrib 0.158.0.Memory is empty on windows_exporter 0.31.0 and later. That release removed the
cscollector and the deprecatedosmemory metrics (windows_cs_physical_memory_bytes,windows_os_physical_memory_free_bytes,windows_os_virtual_memory_bytes,windows_os_virtual_memory_free_bytes), whichwindows.yamlstill read. On a current exporter,meter_win_memory_total/_available/_usedand themeter_win_memory_virtual_memory_*commit metrics had no data. The rules now read the replacements that the exporter's own deprecation notes name, from itsmemorycollector:windows_memory_physical_total_bytes,windows_memory_physical_free_bytes,windows_memory_commit_limitandwindows_memory_committed_bytes. All four exist since windows_exporter 0.29.0, which becomes the minimum version. The setup doc and CHANGES say so. Reading both the old and new names would double-count on 0.29 and 0.30, where both exist.CPU total counts overlapping modes twice. Windows CPU modes overlap. windows_exporter's
privilegedalready includesinterruptanddpctime. The Collector'ssystemstate is kernel time minus idle, so it already includesinterrupttime.cpu_total_percentagesummed every non-idle mode, so that time was counted twice. It now sumsuser+privileged(windows_exporter) oruser+system(Collector), the same filtercpu_norm_percentagealready uses.cpu_average_usedstill shows every mode; the doc notes that they overlap. On the runner, per core over 61 s under load,idle+user+privileged= 1.0000 s/s and all five windows_exporter modes = 1.005–1.009 s/s.A current Collector's Linux and Windows hosts get no service. From Collector 0.127.0 the Prometheus receiver sends a scraped target's host only as
server.address, no longer asnet.host.name(feature gatereceiver.prometheusreceiver.RemoveLegacyResourceAttributes, beta in 0.127.0, legacy attributes removed in 0.129.0).vm.yamlandwindows.yamlname a host bynode_identifier_host_name, which the OTel receiver took only fromnet.host.nameorhost.name, so no host service was created. The receiver now falls back toserver.addresswhen neither is present, so an explicithost.namestill wins. The e2e cases pin Collector 0.102.1, from before the change, which is why they never saw it.On the runner, the four old memory metrics are absent on 0.31.x and the four new ones present; 0.30.9 carries both with identical values, and the new ones match
Win32_ComputerSystem.TotalPhysicalMemoryand the\Memory\Commit Limit/Committed Bytes/Available Bytescounters. The Collector sentservice.name,server.address,service.instance.id,server.port,url.schemeand nonet.host.name.Tests:
windows.data.yaml: the windows_exporter CPU input carries all five modes, and the memory inputs use the new names with distinct values.vm/otel-hostmetrics:system.cpu.timecounters grow with the clock foruser,system,interruptandidle, so OAP'srate()runs on real increases. They used to be fixed values, so every rate was 0 and no CPU metric was asserted.stateattribute.system.cpu.logical.countis now a non-monotonic Sum, as the Collector sends it.win:memorycollector names, and the case asserts exact memory total and used, commit limit and commit %.windows.yamlreads and otherwise as the Collector wrote it. It names its host only inserver.address; the case asserts that host's service and its exact memory total and used and commit limit, which needs the receiver fallback.metadata, which its strict JSON parser refused. Every existing mock file still parses.OpenTelemetryMetricRequestProcessorTestcoversserver.addressalone, andhost.name/net.host.namewinning over it.ghcr.io/apache/skywalking/oapat masterc4ae53dab2with these rules mounted:vm/otel-hostmetrics24/24 andwin5/5. A deliberately wrong expected value fails, so the exact assertions do check values.Also corrected in the docs:
The process-hostmetrics rules said
process.memory.utilizationarrives as a 0–1 fraction. The Collector records gopsutil'sMemoryPercent, which is 0–100. The arithmetic was already right; the comment and the MAL test data now use the real scale.The Linux metrics table no longer claims that Telegraf provides
sockets_used;telegraf-rules/vm.yamlhas no such rule.The Windows doc's example Collector configuration used the
loggingexporter, which the Collector removed;otelcol-contrib validaterejects it on 0.158.0. It now usesdebug.The OTel receiver doc lists the
server.addressfallback, and the Windows doc says how a host's service is named.If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
Update the
CHANGESlog.