Skip to content

[metrics] Count per-bucket errors in TabletServer response metrics - #4360

Open
fxbing wants to merge 2 commits into
apache:mainfrom
fxbing:feature/20260914-response-error-metrics
Open

fxbing wants to merge 2 commits into
apache:mainfrom
fxbing:feature/20260914-response-error-metrics

Conversation

@fxbing

@fxbing fxbing commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #4359

Follow up on #4213 to count per-bucket errors returned inside normal RPC responses. These errors currently bypass sendError() and are missing from the per-error metrics.

Brief change log

  • Generate recursive error-code collection for protobuf messages.
  • Count failed response entries in per-error metrics, excluding absent codes and NONE, while preserving the aggregate metric's whole-request failure semantics.
  • Prevent metric failures from triggering another response and avoid counting bucket errors when response encoding fails.

Tests

  • Added regression coverage for mixed-success batches, repeated and unknown errors, nested responses, message reuse, and encoding and metric failures.
  • Common and protogen tests, plus targeted RPC tests, passed.
  • Spotless, Checkstyle, and RAT passed. Full integration tests were not run.

API and Format

No public API, RPC wire-format, or storage-format changes. Per-error metrics now include bucket failures and therefore need not sum to the existing aggregate error metric.

Documentation

Updated the metrics reference to explain the counting semantics.

Generative AI disclosure

  • Yes — OpenAI Codex and Anthropic Claude.

@swuferhong
swuferhong self-requested a review September 18, 2026 05:00

@swuferhong swuferhong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @fxbing. Such an useful metrics improvement. I left some comments:

Generate recursive raw error-code collection for protobuf messages.
Count bucket failures without changing aggregate RPC error semantics.
Cover response errors, message reuse, and encoding and metrics failures.
Register missing TabletServer APIs and split ListOffsets client/follower metrics.

Exercise response errors with EmbeddedChannel tests and document the expanded metrics scope.
@fxbing
fxbing force-pushed the feature/20260914-response-error-metrics branch from c1982b8 to efd8c79 Compare September 23, 2026 02:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[metrics] Count per-bucket errors in TabletServer response metrics

2 participants