Skip to content

fix(asap-tools): keep ProcessMonitor samples when a monitored process dies - #741

Merged
milindsrivastava1997 merged 3 commits into
mainfrom
740-processmonitor-loses-all-samples-when-one-monitored-process-dies
Sep 29, 2026
Merged

milindsrivastava1997 merged 3 commits into
mainfrom
740-processmonitor-loses-all-samples-when-one-monitored-process-dies

Conversation

@milindsrivastava1997

Copy link
Copy Markdown
Contributor

Closes #740

Problem

If any monitored process exited mid-run, psutil.NoSuchProcess escaped the sampler loop, the sampler never sent its data, and remote_monitor.py skipped writing monitor_output.json — losing samples for every process. The run also continued unnoticed (the query client swallows per-query errors and runs its full schedule).

Changes

  • process_monitor.py
    • Seed pid exits → roll back the partial sampling round (keeps all series equal-length), set exited_at_sample: N on the dead pid's entry, and send the data.
    • Child pid exits → stop sampling only that child; monitor keeps running.
  • remote_monitor.py
    • Container client mode: wait loop checks control_pipe.poll(0) and stops the client early when the sampler reports an exit.
    • timed mode: time.sleep → control_pipe.poll(time_to_run) so it also ends early.
    • Logs an error when writing a flagged monitor_output.json.
    • Bare-metal client blocks in start(), so it doesn't stop early, but data is now preserved.
  • compare_costs.py: modes with a flagged pid are skipped with an error; --allow_partial_runs includes them with a warning.
  • Flag key lives in constants.PROCESS_MONITOR_EXITED_AT_SAMPLE_KEY (per-pid field, so existing consumers that iterate keys as pids are unaffected).

Testing

  • New tests/test_process_monitor.py: seed death (real processes; checks alignment, rollback, and flag) and child death (mocked; monitor keeps sampling). Both fail on the previous code.
  • pytest tests in asap-tools/experiments: 18 passed.
  • Not yet exercised in a full e2e experiment.

🤖 Generated with Claude Code

… dies

A monitored pid exiting mid-run crashed the sampler, so remote_monitor wrote
no monitor_output.json and every pid's samples were lost.

- Seed pid exits: roll back the partial round, mark the entry with
  exited_at_sample, and send the aligned data.
- Child pid exits: stop sampling only that child.
- remote_monitor stops the container client / timed wait early when the
  sampler reports an exit, and logs it when writing the output.
- compare_costs skips flagged modes unless --allow_partial_runs.

Closes #740

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
process_monitor depends on psutil, but CI installs only
experiments/requirements.txt, so test_process_monitor failed to import.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ses-all-samples-when-one-monitored-process-dies
@milindsrivastava1997
milindsrivastava1997 merged commit 6864d57 into main Sep 29, 2026
10 checks passed
@milindsrivastava1997
milindsrivastava1997 deleted the 740-processmonitor-loses-all-samples-when-one-monitored-process-dies branch September 29, 2026 02:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ProcessMonitor loses all samples when one monitored process dies

1 participant