Skip to content

[scheduler] Configurable outdated thresholds, fix early resource scrubbing - #154

Merged
cgalibern merged 2 commits into
opensvc:mainfrom
cgalibern:dev
Sep 25, 2026
Merged

cgalibern merged 2 commits into
opensvc:mainfrom
cgalibern:dev

Conversation

@cgalibern

Copy link
Copy Markdown
Contributor

Problem

scrub_resources flags live resources as undef too early:

scrub_1m: scrub_resources: set 999 resmon status to undef (no live instance) for ...

Two causes:

  • InstancePingFromNodeID returned early when no svcmon row needed a
    refresh, so resmon.updated and resmon_log_last.res_end were not
    refreshed. When svcmon was kept fresh by the daemon status feed,
    resmon.updated aged, and resources of live instances were scrubbed
    after 15 minutes.
  • The age limits used by the scheduler tasks were hardcoded in SQL, so
    they could not be tuned to a deployment.

Changes

[worker] Fix resmon not refreshed on daemon ping

The svcmon, svcmon_log_last, resmon and resmon_log_last refreshes in
InstancePingFromNodeID now run independently.

[scheduler] Make task outdated thresholds configurable

  • Each task that decides a value is outdated reads its age limit from
    scheduler.task.<task>.max_age. The defaults keep the previous
    hardcoded values, so behaviour is unchanged unless configured.
  • The value accepts time.ParseDuration units plus d (15m, 25h,
    2d, 1d12h) and must be at least 1m.
  • All scheduler.task.*.max_age keys are validated when the scheduler
    starts, so the scheduler and scheduler exec refuse to start with an
    invalid value.
  • scrub_object no longer uses the v_outdated_services view; its
    query is now inlined with a configurable age. The database schema is
    unchanged and the view is left in place.
Key (scheduler.task.<key>.max_age) Default
scrub_object, scrub_resources 15m
scrub_instances 21m
scrub_unfinished_actions 2h
scrub_resmon, scrub_pdf 1d
scrub_checks_live, scrub_diskinfo, scrub_svcdisks, scrub_stor_array 2d
scrub_node_hba, scrub_comp_status_unattached 7d
scrub_comp_status 31d
scrub_packages, scrub_patches 100d
scrub_static, scrub_tempviz 1h
alert_instances_not_updated 16m
alert_nodes_not_updated, alert_service_config_not_updated 25h
alert_checks_not_updated 1d
log_instances_not_updated 2h

Example:

scheduler:
  task:
    # max_age: the age after which the task considers a value outdated.
    # Units: s, m, h, d (e.g. 15m, 25h, 2d, 1d12h). Minimum 1m.
    # All max_age values are checked at startup; the scheduler refuses
    # to start if one is invalid.

    # alerts
    alert_checks_not_updated:
      max_age: 1d
    alert_instances_not_updated:
      max_age: 16m
    alert_nodes_not_updated:
      max_age: 25h
    alert_service_config_not_updated:
      max_age: 25h
    log_instances_not_updated:
      max_age: 2h

    # status scrubs (every minute)
    scrub_object:
      max_age: 15m
    scrub_resources:
      max_age: 15m
    scrub_instances:
      max_age: 21m

    # action scrubs (every 10 minutes)
    scrub_unfinished_actions:
      max_age: 2h

    # table purges (daily)
    scrub_checks_live:
      max_age: 2d
    scrub_comp_status:
      max_age: 31d
    scrub_comp_status_unattached:
      max_age: 7d
    scrub_diskinfo:
      max_age: 2d
    scrub_node_hba:
      max_age: 7d
    scrub_packages:
      max_age: 100d
    scrub_patches:
      max_age: 100d
    scrub_resmon:
      max_age: 1d
    scrub_stor_array:
      max_age: 2d
    scrub_svcdisks:
      max_age: 2d

    # file scrubs in directories.static
    scrub_static:
      max_age: 1h
    scrub_tempviz:
      max_age: 1h
    scrub_pdf:
      max_age: 1d

Operators changing these values should keep them consistent:
scrub_instances purges svcmon rows, so setting scrub_resources or
scrub_object above it means instances are purged before their
resources or objects would be flagged undef.

Tasks that decide a value is outdated now read their age limit from
scheduler.task.<task>.max_age instead of hardcoded SQL intervals. The
defaults keep the previous values.

The value accepts time.ParseDuration units plus "d" (15m, 25h, 2d,
1d12h) and must be >= 1m. All scheduler.task.*.max_age keys are
validated when the scheduler starts.

scrub_object no longer uses the v_outdated_services view. Its query is
now inlined with a configurable age. The view is left in the database.

Tunable tasks:
  scrub_object, scrub_resources, scrub_instances,
  scrub_unfinished_actions, scrub_resmon, scrub_checks_live,
  scrub_diskinfo, scrub_svcdisks, scrub_stor_array, scrub_node_hba,
  scrub_packages, scrub_patches, scrub_comp_status,
  scrub_comp_status_unattached, scrub_static, scrub_tempviz, scrub_pdf,
  alert_instances_not_updated, alert_nodes_not_updated,
  alert_service_config_not_updated, alert_checks_not_updated,
  log_instances_not_updated
InstancePingFromNodeID returned early when no svcmon row needed a
refresh, so resmon.updated and resmon_log_last.res_end were skipped.
When svcmon was kept fresh by the daemon status feed, resmon.updated
aged, and scrub_resources flagged the resources of live instances as
undef.

The svcmon and resmon refreshes now run independently.
@cgalibern
cgalibern merged commit dedf073 into opensvc:main Sep 25, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant