[scheduler] Configurable outdated thresholds, fix early resource scrubbing - #154
Merged
Merged
Conversation
Tasks that decide a value is outdated now read their age limit from scheduler.task.<task>.max_age instead of hardcoded SQL intervals. The defaults keep the previous values. The value accepts time.ParseDuration units plus "d" (15m, 25h, 2d, 1d12h) and must be >= 1m. All scheduler.task.*.max_age keys are validated when the scheduler starts. scrub_object no longer uses the v_outdated_services view. Its query is now inlined with a configurable age. The view is left in the database. Tunable tasks: scrub_object, scrub_resources, scrub_instances, scrub_unfinished_actions, scrub_resmon, scrub_checks_live, scrub_diskinfo, scrub_svcdisks, scrub_stor_array, scrub_node_hba, scrub_packages, scrub_patches, scrub_comp_status, scrub_comp_status_unattached, scrub_static, scrub_tempviz, scrub_pdf, alert_instances_not_updated, alert_nodes_not_updated, alert_service_config_not_updated, alert_checks_not_updated, log_instances_not_updated
InstancePingFromNodeID returned early when no svcmon row needed a refresh, so resmon.updated and resmon_log_last.res_end were skipped. When svcmon was kept fresh by the daemon status feed, resmon.updated aged, and scrub_resources flagged the resources of live instances as undef. The svcmon and resmon refreshes now run independently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
scrub_resourcesflags live resources asundeftoo early:Two causes:
InstancePingFromNodeIDreturned early when no svcmon row needed arefresh, so
resmon.updatedandresmon_log_last.res_endwere notrefreshed. When svcmon was kept fresh by the daemon status feed,
resmon.updatedaged, and resources of live instances were scrubbedafter 15 minutes.
they could not be tuned to a deployment.
Changes
[worker] Fix resmon not refreshed on daemon ping
The svcmon, svcmon_log_last, resmon and resmon_log_last refreshes in
InstancePingFromNodeIDnow run independently.[scheduler] Make task outdated thresholds configurable
scheduler.task.<task>.max_age. The defaults keep the previoushardcoded values, so behaviour is unchanged unless configured.
time.ParseDurationunits plusd(15m,25h,2d,1d12h) and must be at least1m.scheduler.task.*.max_agekeys are validated when the schedulerstarts, so the scheduler and
scheduler execrefuse to start with aninvalid value.
scrub_objectno longer uses thev_outdated_servicesview; itsquery is now inlined with a configurable age. The database schema is
unchanged and the view is left in place.
scheduler.task.<key>.max_age)scrub_object,scrub_resourcesscrub_instancesscrub_unfinished_actionsscrub_resmon,scrub_pdfscrub_checks_live,scrub_diskinfo,scrub_svcdisks,scrub_stor_arrayscrub_node_hba,scrub_comp_status_unattachedscrub_comp_statusscrub_packages,scrub_patchesscrub_static,scrub_tempvizalert_instances_not_updatedalert_nodes_not_updated,alert_service_config_not_updatedalert_checks_not_updatedlog_instances_not_updatedExample:
Operators changing these values should keep them consistent:
scrub_instances purges svcmon rows, so setting scrub_resources or
scrub_object above it means instances are purged before their
resources or objects would be flagged undef.