Skip to content

openmetrics: fix histogram buckets with le="0.0" being silently dropped - #25283

Open
ian28223 wants to merge 1 commit into
masterfrom
ian.bucad/fix-openmetrics-histogram-zero-bucket-dropped
Open

ian28223 wants to merge 1 commit into
masterfrom
ian.bucad/fix-openmetrics-histogram-zero-bucket-dropped

Conversation

@ian28223

@ian28223 ian28223 commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes OpenMetrics histogram buckets with le="0.0" being silently dropped when histogram_buckets_as_distributions is enabled.

  • decumulate_histogram_buckets classified the smallest bucket as "negative domain" whenever its upper bound was not > 0. For le="0.0" (upper bound == 0), this gave it lower_bound = -Inf instead of 0.
  • The agent's sketch aggregation silently discards any bucket with an infinite bound, so that bucket's entire count vanished from the distribution metric, even though .count/.sum (submitted separately) still included it.
  • Also relaxed the "equal bounds" skip guard in the histogram transformer so it only fires for the genuinely degenerate -Inf/-Inf case, not for legitimate finite single-point buckets like [0, 0] that this fix now produces.

Motivation

Customer escalation AGENT-17215: a Solr-backed OpenMetrics histogram with occasional 0ms-latency requests was undercounting its distribution metric by exactly the le="0.0" bucket's count, because that bucket was silently dropped end-to-end.

  • Added test_histogram_buckets_as_distributions_with_zero_bucket, reproducing the reported histogram (le="0.0", le="5.0", le="+Inf") and asserting the zero bucket is submitted with lower_bound=0 instead of being dropped.
  • ddev test datadog_checks_base -- tests/base/checks/openmetrics/test_v2/ — 211 passed, 3 skipped, no regressions.
  • Manually verified on a live agent (Rocky Linux test VM) with a fake OpenMetrics endpoint serving the exact Solr-shaped histogram: trace logs show lower_bound:-inf (dropped) before the fix and lower_bound:0 (submitted) after.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

🤖 Generated with Claude Code

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Sep 21, 2026

Copy link
Copy Markdown

evalya-impact-summary

evalya impact analysis
Impact analysis: 0 selected, 0 skipped (of 0 test tasks)
Publish tasks:   2 (always emitted)
Diff (4 files):
  datadog_checks_base/changelog.d/25283.fixed
  datadog_checks_base/datadog_checks/base/checks/openmetrics/v2/transformers/histogram.py
  datadog_checks_base/datadog_checks/base/checks/openmetrics/v2/utils.py
  datadog_checks_base/tests/base/checks/openmetrics/test_v2/test_transformers/test_histogram.py

Debug a specific task: evalya plan impact --path <path> --task <task>

Learn more about CI impact filtering

@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Sep 21, 2026

Copy link
Copy Markdown

Tests  Code Coverage

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 76.92%
Overall Coverage: 89.27% (+0.09%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b543de9 | Docs | View more details | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Disk usage change

Commit b543de9 compared against 99606e5.

Uncompressed

linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
🔺 +135 B 🔺 +135 B 🔺 +135 B 🔺 +135 B 🔺 +135 B
Details
Name Version Type linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
datadog_checks_base 38.3.2 Integration 🔺 +135 B 🔺 +135 B 🔺 +135 B 🔺 +135 B 🔺 +135 B

Compressed

linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
🔺 +44 B 🔺 +44 B 🔺 +44 B 🔺 +44 B 🔺 +44 B
Details
Name Version Type linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
datadog_checks_base 38.3.2 Integration 🔺 +44 B 🔺 +44 B 🔺 +44 B 🔺 +44 B 🔺 +44 B

@ian28223 ian28223 added the qa/skip-qa Automatically skip this PR for the next QA label Sep 21, 2026
@dd-octo-sts

dd-octo-sts Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

🔄 Dispatcher tests · in progress

Dispatcher beta: informational only

Dispatcher is running alongside existing CI while we validate it. You can ignore this report and its statuses. Existing CI remains the merge signal.

Note

Tests are still running. 1 of 4 batches has not finished yet. 1 of 855 jobs have not reported.
This comment updates automatically as jobs progress and results are collected.

  854/855 jobs
✅ 853 passed · ❌ 1 failed · ⏳ 1 pending

Batches

BatchStateJobsWorkflow
batch-01✅ passed240/240run 35565210320
batch-02🔄 in progress216/217run 35565210493
batch-03✅ passed220/220run 35565210547
batch-04❌ failed178/178run 35565210470

❌ Failures

yarn / py3.13 / linux   view job

1 failed step
  • Run the tests
⏳ Dispatcher running — GitHub Run.

When histogram_buckets_as_distributions is enabled, the smallest bucket
was misclassified as a negative-domain bucket whenever its upper bound
was exactly 0, giving it lower_bound=-Inf. The agent's sketch
aggregation silently discards any bucket with an infinite bound, so
that bucket's count never reached the resulting distribution metric
even though it was still included in .count/.sum.

Also relax the equal-bounds guard in the histogram transformer so it
only skips the degenerate -Inf/-Inf case, not legitimate single-point
buckets like [0, 0].

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@ian28223
ian28223 force-pushed the ian.bucad/fix-openmetrics-histogram-zero-bucket-dropped branch from 7c91301 to b543de9 Compare September 21, 2026 05:33
@dd-octo-sts

dd-octo-sts Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Validation Report

All 21 validations passed.

Show details
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file
ci Validate CI configuration and code coverage settings
codeowners Validate every integration has a CODEOWNERS entry
config Validate default configuration files against spec.yaml
dep Verify dependency pins are consistent and Agent-compatible
http Validate integrations use the HTTP wrapper correctly
imports Validate check imports do not use deprecated modules
integration-style Validate check code style conventions
jmx-metrics Validate JMX metrics definition files and config
labeler Validate PR labeler config matches integration directories
legacy-signature Validate no integration uses the legacy Agent check signature
license-headers Validate Python files have proper license headers
licenses Validate third-party license attribution list
metadata Validate metadata.csv metric definitions
models Validate configuration data models match spec.yaml
openmetrics Validate OpenMetrics integrations disable the metric limit
package Validate Python package metadata and naming
qa-label Validate the pull request declares whether it needs QA for the next Agent release
readmes Validate README files have required sections
saved-views Validate saved view JSON file structure and fields
version Validate version consistency between package and changelog

View full run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant