feat: add covered-category-count and capped max-score fields for partial health scores (IN-1262)

[gaspergrom]

Summary

• Adds coveredCategoryCount (0-3) and healthMaxScore (60/65/75/100) to the Tinybird pipe chain — project_insights_copy_ds.datasource → project_insights_copy.pipe → project_insights.pipe, plus the two sink pipes — so a project missing exactly one Health Score v2 category (Maintainer 40 / Security & Supply Chain 35 / Development Activity 25) carries a single source-of-truth capped denominator and coverage count, instead of Insights, LFX One, and the Health Score Badge each recomputing the "- Partial" logic independently.
• healthMaxScore is null when fewer than 2 categories are covered (mirrors the existing healthScoreV2 null guard), 100 when all 3 are covered, otherwise 100 minus the weight of the one missing category.
• The korg project override in project_insights_copy.pipe explicitly nulls both new fields, matching how it already nulls other v2 fields for that slug.
• Repo-level records in project_insights_copy.pipe get both fields hardcoded to NULL — coverage/partial scoring is a project-level concept only.
• Also un-comments and activates the previously-inert EXPORT_SCHEDULE cron on health_score_v2_sink.pipe (was @on-demand-only pending stakeholder sign-off, now runs daily at 03:35 UTC — after project_insights_copy_ds's 03:00 refresh — and threads the two new fields through both health_score_sink.pipe and health_score_v2_sink.pipe.
• healthLabel in project_insights_copy.pipe and avgHealthScore in collection_insights_aggregate_projects_copy_pipe.pipe both bucket/average the score normalized to healthScoreV2 * 100.0 / healthMaxScore rather than the raw capped score, so a partial score (e.g. 45/60) benchmarks consistently as its full-scale equivalent (75/100) at both the project and collection level.
• Does not change how healthScoreV2 itself is computed, and does not touch the Insights/LFX One/Health Score Badge consumers — those are separate companion PRs.

Changes

File	What changed
services/libs/tinybird/datasources/project_insights_copy_ds.datasource	Adds coveredCategoryCount and healthMaxScore Nullable(UInt8) columns + doc comments
services/libs/tinybird/pipes/project_insights_copy.pipe	Computes both fields from the same category-presence checks the healthScoreV2 guard uses; nulls them for korg and for repo records
services/libs/tinybird/pipes/project_insights.pipe	Exposes both fields on the public project_insights endpoint, with doc comment
services/libs/tinybird/pipes/health_score_sink.pipe	Adds covered_category_count_v2 / health_max_score_v2 to the sink output
services/libs/tinybird/pipes/health_score_v2_sink.pipe	Activates the daily EXPORT_SCHEDULE at 03:35 UTC (was on-demand only) and adds coveredCategoryCount / healthMaxScore to the sink output
services/libs/tinybird/pipes/collection_insights_aggregate_projects_copy_pipe.pipe	Normalizes avgHealthScore to healthScoreV2 * 100.0 / healthMaxScore before averaging, to stay consistent with the project-level healthLabel normalization

JIRA

IN-1262 — Implement Partial Health Score labels and capped totals for projects with 1 missing category

This is crowd.dev's slice of a cross-repo ticket. Companion PRs exist in insights, lfx-self-serve, and lf-dbt to consume these fields for the "- Partial" label and capped total display.

Deploy order

1. This PR (crowd.dev Tinybird pipes) deploys first — already done and QA'd against live production Tinybird.
2. insights, lfx-self-serve, and lf-dbt companion PRs deploy after, since their features read coveredCategoryCount/healthMaxScore from these pipes and are not testable end-to-end until this is live.

Test plan

• Deployed to production Tinybird and validated against live data (coverage counts and capped denominators match expected values for projects missing one category)
• Confirm korg project still returns null for both new fields after deploy
• Confirm repo-type records in project_insights_copy.pipe return null for both fields
• Confirm the health_score_v2_sink.pipe daily export runs at 03:35 UTC without error on its first scheduled fire

Checklist

• git commit --signoff -S on every commit
• MIT license header on every new source file (no new files — existing pipes/datasource modified)
• PR diff < 1000 lines (48 lines)
• No unrelated changes bundled
• Cross-repo impact documented above (insights, lfx-self-serve, lf-dbt)

[cursor]

PR Summary

Medium Risk
Changes published health labels and collection averages for partial scores and enables a daily v2 Kafka export; downstream apps must deploy after this to consume the new fields safely.

Overview
Adds coveredCategoryCount and healthMaxScore to the Tinybird project-insights pipeline so partial Health Score v2 (one missing category) has a single capped denominator (60/65/75) and coverage count at the data layer, for downstream “Partial” UI without duplicating logic.

project_insights_copy.pipe computes both from the same category-presence rules as healthScoreV2, nulls them for korg and repo rows, and re-buckets healthLabel using healthScoreV2 * 100 / healthMaxScore instead of the raw capped points. Collection avgHealthScore uses the same normalization before averaging.

The fields flow through project_insights_copy_ds, project_insights.pipe, project_repo_insights.pipe, and Kafka sinks health_score_sink / health_score_v2_sink. health_score_v2_sink is scheduled daily at 03:35 UTC (after the 03:00 insights copy) instead of the prior on-demand-only setup; healthScoreV2 computation is unchanged.

^{Reviewed by Cursor Bugbot for commit e7c2e7c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

Adds project-level Health Score v2 coverage counts and capped maximum scores for downstream consumers.

Changes:

• Computes and stores category coverage and maximum score.
• Exposes fields through project insights and Kafka sink output.
• Documents the scheduled Health Score v2 export.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
project_insights.pipe	Exposes the new fields.
project_insights_copy.pipe	Computes project values and nulls unsupported records.
health_score_v2_sink.pipe	Updates export schedule documentation.
health_score_sink.pipe	Adds fields to Kafka output.
project_insights_copy_ds.datasource	Adds nullable storage columns.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7140994. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

services/libs/tinybird/pipes/health_score_v2_sink.pipe:3

• The 00:35 export runs before both inputs it publishes: project_insights_impact_breakdown_copy.pipe refreshes at 02:40 and project_insights_copy.pipe refreshes at 03:00. This can emit the previous day's health and impact snapshot; the existing dependency documentation explicitly identifies 03:30 as the intended sink time. Please schedule this after 03:00 (for example, EXPORT_SCHEDULE 35 3 * * *). Also, because the cron directive itself is unchanged in this diff, this edit only updates the comment rather than activating the export as the PR description claims.

    -- Live and scheduled: EXPORT_SCHEDULE below runs this daily at 00:35 UTC.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

services/libs/tinybird/pipes/health_score_v2_sink.pipe:42

• The diff reschedules an already-active export from 00:35 to 03:35; it does not activate an inert @on-demand sink as the PR description states, and the test plan still expects 00:35. Align the documented rollout/test expectation with the intended cron before merge.

EXPORT_SCHEDULE 35 3 * * *

services/libs/tinybird/pipes/project_repo_insights.pipe:16

• This combined endpoint also returns repository records, but both new fields are hardcoded to NULL for repositories. State that these semantics apply only to project records so API consumers do not expect repository coverage values.

    - `coveredCategoryCount` (0-3) and `healthMaxScore` (60/65/75/100, IN-1262) describe a partial Health Score v2: `healthMaxScore` is the capped denominator when exactly one category is missing (2 of 3 covered), 100 when all 3 are covered, and null when fewer than 2 are covered (same guard as `healthScoreV2`).

services/libs/tinybird/pipes/project_insights_copy.pipe:245

• This changes project healthLabel semantics for every two-category project (for example, 45/60 moves from fair to healthy), but the PR description says consumer behavior is unchanged and neither lists nor tests this relabeling. Please either remove this scope or document and validate the behavior before merge.

            (hv2.healthScoreV2 * 100.0 / hv2.healthMaxScore) >= 85,

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Suppressed comments (1)

services/libs/tinybird/pipes/health_score_v2_sink.pipe:26

• EXPORT_FORMAT csv emits headerless positional rows, so inserting these fields here immediately shifts every existing impact/date column for the current Kafka consumer. This conflicts with the documented producer-first rollout: append the new fields after the existing date column (and coordinate the rollout if the consumer rejects extra fields) so existing positions retain their meaning.

        p.coveredCategoryCount AS coveredCategoryCount,
        p.healthMaxScore AS healthMaxScore,

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

services/libs/tinybird/pipes/project_insights_copy.pipe:239

• This six-line block narrates the expression immediately below it. Repository guidance prohibits comments that merely explain what code does and limits allowed comments to two lines; the healthMaxScore name and pipe DESCRIPTION already capture this behavior, so remove this block.

        -- healthLabel buckets the score NORMALIZED to /100 (healthScoreV2 * 100 / healthMaxScore),
        -- not the raw capped healthScoreV2 - so a partial score like 45/60 benchmarks as its
        -- full-scale equivalent 75/100 ('healthy'), not the raw 45 ('fair') (IN-1262, per product
        -- decision - see collection_insights_aggregate_projects_copy_pipe.pipe for the same rule
        -- applied to the Collections average). healthMaxScore is 100 when all 3 categories are
        -- covered, so this is a no-op for non-partial scores.

services/libs/tinybird/pipes/health_score_v2_sink.pipe:5

• The scheduling-order invariant is a valid comment, but repository guidance limits allowed comments to two lines. Condense this three-line note while retaining the reason for the ordering.

    -- Live and scheduled: EXPORT_SCHEDULE below runs this daily at 03:35 UTC,
    -- after project_insights_copy_ds's 03:00 refresh and the impact breakdown's
    -- 02:40 refresh (see project_insights_impact_breakdown_copy.pipe).