Repository navigation
Dropped webhooks leave QA and CR stamps off the board #502
Description
Some QA and CR comments never show up on the board. Pulldasher only hears about a new comment, push, or close when GitHub sends it a webhook, and some of those webhooks are not getting through. GitHub does not resend a failed delivery, and the only thing that re-reads a pull later is a restart. A restart only covers the repos in the config, so stamps in any other repo stay missing until someone clicks refresh on the card.
Reported on ifixit#64761: the QA comment came after the last push and never counted.
What the board missed on 2026-09-29
- ifixit#64895: the board stopped updating it at 10:28Z and still shows the old head as a draft. It missed a push at 12:18Z, ready for review, and a CR comment at 12:51Z.
- Pushes: #64893, #64521, #64894 and #64885 still show their old head after pushes between 11:01Z and 12:58Z. #64886, pushed three seconds after #64885, came through, so single events are being dropped.
- Repos missing from the config: 8 open pulls in fixbot, DocHarvestor, fixhub-app, ci-actions and expo have a QA comment after their last push that the board ignores.
- Why they stay missing: the startup refresh only walks
config.repos(utils.js#L79, fromapp.js#L98). Other repos reach the board only through webhooks.
Why deliveries fail is still open. It needs the webhook's delivery log or the production container logs.
It isn't the QA pattern, the push check, or comment edits
- The production config uses the same regex as
config.example.js#L5-L8, and it matches every missed comment when run in Node. - Every missed stamp is newer than its pull's head commit, so the check that drops stamps older than the last push doesn't apply.
- The refresh on edit (
githubHooks.js#L158) works: edited QA stamps on ifixit#64779, #64780 and #64782 through #64785 registered.
Ways to fix it
A re-read costs 8 to 15 REST calls per pull (#467). The board holds 151 open pulls from configured repos, 316 in all.
| # | Approach | Fixes missed events | Missing repos | Delay | GitHub calls | New code |
|---|---|---|---|---|---|---|
| 1 | Refresh the card, or restart | When someone notices | Refresh only | Until noticed | 1,200 to 2,300 per restart | None |
| 2 | Add every repo to config.repos |
At restart | Yes | Next restart | 2,500 to 4,700 per restart | Config only, a line per new repo |
| 3 | Startup refresh every hour | Yes | No | Up to 1 hour | 1,200 to 2,300 an hour | A timer |
| 4 | Per-repo list of recently updated pulls every 10 min | Yes | No | Up to 10 min | 156 an hour, plus re-reads | A timer and a list call per repo |
| 5 | Option 4 with conditional requests | Yes | No | Up to 10 min | Re-reads only | Option 4 plus an ETag store |
| 6 | One org-wide search every 10 min | Yes | Yes | Up to 10 min | 6 search calls an hour, plus re-reads | A timer and one search call |
| 7 | Resend failed deliveries on a schedule | Only failed deliveries | Yes | Next run | A few per run | Script, workflow, org owner token |
| 8 | Fix why deliveries fail | That cause only | n/a | n/a | None | Unknown |
I'd go with 6. It needs the least new code of 4 to 6, covers every repo, and search returns closed pulls, so it covers #501's case too. The open question is 5 or 6. Option 5 avoids search, but it only covers configured repos and has to store each list's ETag.
Notes on the options
- Re-reads: 111 org pulls changed in the 24 hours before this was written, so options 4 to 6 spend up to 900 to 1,700 calls a day on re-reads. Comment edits count as updates: fixhub-app#21, ci-actions#15 and expo#1332 each show an
updated_atequal to their QA edit. - Options 2 and 3: both make the startup refresh heavier or more frequent, and Reduce GitHub API consumption to avoid rate-limit outages #467 traced a drained hourly quota to that refresh. Option 3 also lists open pulls only (
git-manager.js#L165-L171), so a pull whose close was dropped stays stuck, the case Fix pulls stuck open after a missed close webhook #501 fixes. - Option 5: an unchanged list comes back 304, which doesn't count against the rate limit.
- Option 6: search has its own limit of 30 calls a minute. Its indexing delay isn't documented, so each search should overlap the previous one by a few minutes.
- Option 7: GitHub's redelivery script needs an org owner token. It misses events Pulldasher answered 200 to but failed to save, and only deliveries from the last 3 days can be resent.
- Option 8: deploys and restarts still drop events, so it needs one of 4 to 6 next to it.
QA comments the board ignores in repos missing from the config
All posted after the pull's last push and matching the QA pattern, as of 2026-09-29. Clicking refresh on each card should clear them.
| Pull | QA comment |
|---|---|
| fixbot#3194 | 2026-08-17 20:13Z |
| fixbot#3215 | 2026-08-17 19:52Z |
| fixbot#3121 | 2026-09-01 18:38Z |
| DocHarvestor#236 | 2026-08-17 18:30Z |
| DocHarvestor#242 | 2026-08-28 03:52Z |
| fixhub-app#21 | edited 2026-09-24 19:11Z |
| ci-actions#15 | edited 2026-09-22 16:47Z |
| expo#1322 | edited 2026-09-24 20:06Z |
Found by comparing signoffs.qa.by from /api/v1/pulls with each pull's comments and head commit date from the GitHub API.
Activity
probablyian commentedon Sep 29, 2026
@jarstelfox any input/preference on how to handle this?
jarstelfox commentedon Sep 29, 2026
Nope!
Catch up on webhooks missed while the server was down
probablyian commentedon Sep 30, 2026
The dropped webhooks weren't random. All 71 stuck rows on the board come from three windows when webhooks weren't getting through. Two of those were outages of the server that runs Pulldasher.
One case doesn't fit. The QA comment on iFixit/ifixit#64761 went up at 03:53 on 09-29, 9 minutes after its last push and 7 hours before the outage. The org webhook's delivery log keeps 3 days, so until about 10-02 an org owner can see what Pulldasher answered. @jarstelfox, could you check that delivery, and redeliver the failed ones from 10:50 to 14:55? That also clears the 13 rows from that window.
#503 already recovers from an outage Pulldasher stays running through, because a failed search keeps its window for the next sweep. db5e626 adds restarts: the first sweep after boot reaches back to the newest pull update in the DB, up to 24 hours.
How the windows were found
/api/v1/pullsthat GitHub shows closed, or with a different head commit. A lost close is dated by its close time, a lost push by when CI started on the new head.Sweep: reach back an hour, skip pulls already re-read
probablyian commentedon Oct 2, 2026
Investigation happening in slack: https://ifixit.slack.com/archives/C6YD8UBS7/p1790970090546029