Skip to content

[metrics] Count per-bucket errors in TabletServer response metrics #4359

Description

@fxbing

Search before asking

  • I searched in the issues and found nothing similar.

Description

Motivation

The per-error metrics added in #4213 are updated only by sendError(). Batch APIs such as produceLog, fetchLog, putKv, and lookup can return normal RPC responses containing per-bucket errors, so these failures are not counted.

For example, two buckets returning NOT_LEADER_OR_FOLLOWER should add two events to that error's meter, even when the RPC response completes normally.

Solution

  • Generate recursive error-code collection for protobuf messages and update per-error metrics from the normal response path, following Kafka's response error-counting approach.
  • Count each failed response entry, excluding absent error codes and NONE. Preserve the existing aggregate metric's whole-request failure semantics and document that per-error metrics need not sum to the aggregate.
  • Cover mixed-success batches, repeated errors, nested responses, message reuse, and encoding failures. Metric failures must not trigger another response.

Anything else?

Follow-up to #4213 (review).

Willingness to contribute

  • I'm willing to submit a PR!

Activity

  1. added theissue type on Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions