Skip to content

Kill the forked agent run if it outlives runtimeout - #642

Open
silug wants to merge 1 commit into
OpenVoxProject:mainfrom
silug:agent-child-run-deadline
Open

silug wants to merge 1 commit into
OpenVoxProject:mainfrom
silug:agent-child-run-deadline

Conversation

@silug

@silug silug commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Short description

The daemon forks a child for each catalog run and waited on it with a bare Process.waitpid2. The child enforces runtimeout on its own run, but nothing guaranteed that it would actually exit afterwards: a stranded thread or hung subprocess left the parent blocked forever and the agent stopped checking in until the service was restarted.

Poll the child with WNOHANG and, once runtimeout plus a grace period for sending the report has elapsed, SIGKILL it and reap it so the daemon carries on with its schedule. A runtimeout of 0 keeps the old blocking behaviour.

Fixes part of #485

Generated by Claude Code

Checklist

I have:

The daemon forks a child for each catalog run and waited on it with a
bare `Process.waitpid2`. The child enforces `runtimeout` on its own run,
but nothing guaranteed that it would actually exit afterwards: a stranded
thread or hung subprocess left the parent blocked forever and the agent
stopped checking in until the service was restarted.

Poll the child with `WNOHANG` and, once `runtimeout` plus a grace period
for sending the report has elapsed, SIGKILL it and reap it so the daemon
carries on with its schedule. A `runtimeout` of 0 keeps the old blocking
behaviour.

Fixes part of OpenVoxProject#485

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Steven Pritchard <steven.pritchard@gmail.com>

@miharp miharp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this against the wedge from #485 (analysis) with a full agent daemon against a real server, on x86_64. The agents were 9.0.0-rc1 (Ruby 4.0.6) on AlmaLinux 9 (glibc 2.34) and Ubuntu 24.04 (glibc 2.39). I ran puppet agent --no-daemonize under gdb and held the CRL-refresh resolver thread inside glibc's resolv.conf lock while the daemon forked the run, so the child inherited the lock locked. Settings were runinterval 90 s, runtimeout 30 s and both HTTP timeouts 10 s, which puts this PR's deadline at 50 s after the fork.

AlmaLinux 9 Ubuntu 24.04
Unpatched Child froze while loading facts. Daemon stopped running. Child froze while applying the cached catalog. Daemon stopped running.
This PR "did not exit within 20 seconds of the run timeout, killing it", then the next scheduled run applied a catalog normally. Same.

In both unpatched cases the child ignored SIGTERM, and runtimeout never fired. The child had reached a Kernel#fork of its own. Ruby joins its timer thread before forking, and that thread was stuck on the inherited lock while exiting, so the main thread waited forever with the GVL held. That is a second way to the same wedge, earlier than the hang at exit in @bootc's dump, and it means the parent-side deadline here is the only thing that recovers the agent.

One small note, not blocking: with default settings the deadline is 72 minutes (runtimeout plus the connect and read timeouts), so a wedged child costs about two missed runs. That seems a fair conservative choice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants