test: keep the TSan job green on a future teardown report and a fan-out read - #687
Open
bburda wants to merge 3 commits into
Open
test: keep the TSan job green on a future teardown report and a fan-out read#687bburda wants to merge 3 commits into
bburda wants to merge 3 commits into
Conversation
The future shared state behind a GenericClient response is freed by whichever thread drops the last reference to it. When that is a gateway executor thread, ~_State_base and the _Result::_M_destroy() it runs happen inside librclcpp, which carries no TSan instrumentation: the operator delete is intercepted in the allocator, but the acquire on the shared_ptr refcount that orders it after the caller's future::get() is not, so TSan pairs the free with the caller's read of _Result::_M_error. Anchor the entry on the libstdc++ frame that performs that read. It is the only frame of the pair that is reliably symbolized: librclcpp is stripped, so the freeing frame carries no symbol name for a race: pattern to match.
… reason An aggregator answers a fanned-out listing 200 with no `partial` until a discovery pass has marked the peer healthy and published it as a contributor to the entity. Peer health does not imply the second: the two are set at opposite ends of the same pass, with the peer metadata fetch and its whole budget in between. Reading why a peer dropped out therefore waits for the fan-out to be in the answer rather than for health alone. The reason and shape assertions stay as they were, so a wrong reason or a reshaped failed_peers still fails; only the moment of reading moves.
DISCOVERY_TIMEOUT already carries the sanitizer time scale, so multiplying it by the scale again squared it: one wait became 540 s under a scale of 3 instead of 180 s. setUpClass runs three of those waits back to back, so the class asked for more wall clock than this test's 300 s ctest budget grants even after the sanitizer jobs stretch it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Summary
The
sanitizer-tsanjob onmainis red on two tests that share nothing but the job.test_cancel_outcomespasses all 20 cases; the process exits non-zero because TSan reports one race. The last reference to aGenericClientfuture's shared state is dropped by the rclcpp executor thread, insidelibrclcpp, which is not instrumented. TSan sees theoperator deletethere but not the acquire on theshared_ptrrefcount that orders it after ourfuture::get(), so it pairs the free with the caller's earlier read of the result. Our use of the API is correct and the reference is held for the wholeget().tsan_suppressions.txtgets one more line in the existing block for this class of report, anchored on the libstdc++ read-side frame, because the freeing frame is inside a stripped library and prints as<null>. Reproduced under TSan with a standalone reproducer against the real suppressions file: reported 5 of 5 runs before the line, 0 of 21 after.test_peer_failure_reasonsfailed on an assertion, not a report. Between the aggregator marking a peer online and publishing its contributor map for an entity, a read answers 200 with no fan-out at all, which is the documented purpose of the contributor map. The test's readiness gate waited for the first signal and read the failure reason from a single response, so it asserted a transient state; the window is about 0.9 s without instrumentation and wider under TSan._fan_out_failurenow waits until the answer carries the fan-out before reading the reason, with every original assertion kept. With the readiness gate reduced to "the HTTP server answers", the old helper fails 5 of 5 runs with the exact CI message and the new one passes 5 of 5. The file'sTIMEOUTalso appliedget_time_scale()on top of the already scaledDISCOVERY_TIMEOUT; it now scales once, so the longest single wait under TSan is 180 s inside the test's 900 s ctest budget instead of 540 s.Issue
No issue. Two independent failures of one CI job, each with its own commit.
Type
Testing
test_cancel_outcomesandtest_peer_failure_reasonson jazzy, plain build: 10 of 10 and 10 of 10, plus the aggregation sweep (ctest -R "peer|aggregat", 11 of 11).test_1still fails the fixed test ('timeout' != 'unreachable').sanitizer-tsanjob of this pull request is the only place the suppression is verified against the real gateway under TSan.Checklist