Conversation
The daemon forks a child for each catalog run and waited on it with a bare `Process.waitpid2`. The child enforces `runtimeout` on its own run, but nothing guaranteed that it would actually exit afterwards: a stranded thread or hung subprocess left the parent blocked forever and the agent stopped checking in until the service was restarted. Poll the child with `WNOHANG` and, once `runtimeout` plus a grace period for sending the report has elapsed, SIGKILL it and reap it so the daemon carries on with its schedule. A `runtimeout` of 0 keeps the old blocking behaviour. Fixes part of OpenVoxProject#485 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Steven Pritchard <steven.pritchard@gmail.com>
miharp
left a comment
There was a problem hiding this comment.
I tested this against the wedge from #485 (analysis) with a full agent daemon against a real server, on x86_64. The agents were 9.0.0-rc1 (Ruby 4.0.6) on AlmaLinux 9 (glibc 2.34) and Ubuntu 24.04 (glibc 2.39). I ran puppet agent --no-daemonize under gdb and held the CRL-refresh resolver thread inside glibc's resolv.conf lock while the daemon forked the run, so the child inherited the lock locked. Settings were runinterval 90 s, runtimeout 30 s and both HTTP timeouts 10 s, which puts this PR's deadline at 50 s after the fork.
| AlmaLinux 9 | Ubuntu 24.04 | |
|---|---|---|
| Unpatched | Child froze while loading facts. Daemon stopped running. | Child froze while applying the cached catalog. Daemon stopped running. |
| This PR | "did not exit within 20 seconds of the run timeout, killing it", then the next scheduled run applied a catalog normally. | Same. |
In both unpatched cases the child ignored SIGTERM, and runtimeout never fired. The child had reached a Kernel#fork of its own. Ruby joins its timer thread before forking, and that thread was stuck on the inherited lock while exiting, so the main thread waited forever with the GVL held. That is a second way to the same wedge, earlier than the hang at exit in @bootc's dump, and it means the parent-side deadline here is the only thing that recovers the agent.
One small note, not blocking: with default settings the deadline is 72 minutes (runtimeout plus the connect and read timeouts), so a wedged child costs about two missed runs. That seems a fair conservative choice.
Short description
The daemon forks a child for each catalog run and waited on it with a bare
Process.waitpid2. The child enforcesruntimeouton its own run, but nothing guaranteed that it would actually exit afterwards: a stranded thread or hung subprocess left the parent blocked forever and the agent stopped checking in until the service was restarted.Poll the child with
WNOHANGand, onceruntimeoutplus a grace period for sending the report has elapsed, SIGKILL it and reap it so the daemon carries on with its schedule. Aruntimeoutof 0 keeps the old blocking behaviour.Fixes part of #485
Generated by Claude Code
Checklist
I have:
Signed-off-byannotation to each of my commitsGenerated-byorAssisted-byannotations to each of my commits created with the help of an AI agent