feat: integration builder POC [CM-1372] - #4463
Conversation
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
PR SummaryHigh Risk Overview A 30s dispatcher claims due units, admits runs using Redis token-pool budget probes, and starts The new Reviewed by Cursor Bugbot for commit a39d4ec. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
|
mbani01
changed the title
There was a problem hiding this comment.
Pull request overview
Adds a proof-of-concept persistence and scheduling layer for integration sync units.
Changes:
- Defines sync-unit types and statuses.
- Adds claiming, rescheduling, and run-recording queries.
- Creates the sync-unit table and due-work index.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
services/libs/data-access-layer/src/integrationBuilder/types.ts |
Defines sync-unit data contracts. |
services/libs/data-access-layer/src/integrationBuilder/syncUnits.ts |
Implements sync-unit database operations. |
backend/src/database/migrations/V1786442761__createSyncUnitsTable.sql |
Adds sync-unit storage and indexing. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
![github-license-compliance[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 9 changed files in this pull request and generated no new comments.
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
Suppressed comments (3)
services/libs/data-access-layer/src/integrationBuilder/syncUnits.ts:48
- Soft-deleting an integration does not deactivate these rows:
IntegrationRepository.destroyuses the paranoid integration model, while this claim only checks the sync-unit status. As a result, disconnected integrations remain claimable and continue syncing indefinitely. Filter candidates to integrations whosedeletedAtis null (and separately decommission their units if retention requires it).
WHERE status = 'active'
AND "nextRunAt" <= now()
AND ("lockedAt" IS NULL OR "lockedAt" < now() - $(leaseMinutes) * interval '1 minute')
services/libs/data-access-layer/src/integrationBuilder/syncUnits.ts:42
- This lease can expire and be claimed by a second worker, but the returned
lockedAtis not used as an ownership token byrescheduleUnit,recordRunSuccess, orrecordRunFailure; all three update byidalone. A slow first worker can therefore overwrite the newer run's watermark/counters and even clear its lock. Pass the claimed lease value (or a generated claim token) to every completion update, include it in theWHEREclause, and reject a zero-row update.
SET "lockedAt" = now(), "updatedAt" = now()
services/libs/data-access-layer/src/integrationBuilder/syncUnits.ts:9
- These new data-access functions are not exported from the package entry point:
@crowd/data-access-layerresolves tosrc/index.ts, which has nointegrationBuilderexport, and this directory has no index module. Consumers therefore cannot use the normal package API and must rely on an internal/src/...deep import. Add anintegrationBuilder/index.tsbarrel and export it from the root index.
export async function upsertSyncUnits(qx: QueryExecutor, units: SyncUnitUpsert[]): Promise<number> {
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 9 changed files in this pull request and generated 1 comment.
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
Suppressed comments (2)
services/libs/data-access-layer/src/integrationBuilder/syncUnits.ts:26
- This only inserts/renames discovered units. A channel or sync omitted by a later discovery remains
activeand continues being scheduled, while a previouslydecommissionedunit that is rediscovered remains decommissioned. Reconcile the complete discovered set transactionally: decommission missing units and reactivate rediscovered ones (while preserving intentionally paused/dead-letter units).
ON CONFLICT ("integrationId", "channelId", "syncName")
DO UPDATE SET "channelName" = EXCLUDED."channelName", "updatedAt" = now()`,
services/libs/data-access-layer/src/integrationBuilder/syncUnits.ts:53
- Filtering soft-deleted integrations only at claim time leaves their units
activewith permanently overduenextRunAtvalues. Those rows stay at the front ofix_sync_units_due, so every scheduler poll must scan past an ever-growing set of unclaimable units. Decommission sync units as part of integration deletion (or add equivalent cleanup) so they leave the partial due index.
AND EXISTS (
SELECT 1
FROM public.integrations i
WHERE i.id = su2."integrationId" AND i."deletedAt" IS NULL
)
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
Suppressed comments (5)
services/libs/integration-builder/src/credentials.ts:31
- These lines document a future implementation change rather than a required invariant. Remove them; the credential-loading function and environment variable names are self-explanatory.
// POC only: secrets come from env; secret-manager adoption (OCI Vault) swaps this
// body without touching callers — getCredential stays the single entry point
services/libs/integration-builder/package.json:19
zodis not used anywhere in the new package, but adding it also introduces a separate Zod 3 installation in the lockfile. Remove the dependency until validation is implemented.
"zod": "^3.22.0"
services/libs/integration-builder/src/types.ts:14
- These POC/future-design notes describe the current change rather than a non-obvious invariant. Remove them; the fixed
kindtype already makes the temporary single-variant constraint clear.
// POC only: single variant; becomes a discriminated union (token, oauth2, ...)
// as more connectors land
services/libs/integration-builder/src/credentials.ts:20
- This change note only restates the temporary switch design and does not document an invariant. Remove it rather than retaining POC commentary in the implementation.
This issue also appears on line 30 of the same file.
// POC scope: GitHub only; each migrated connector adds its platform case here
services/libs/integration-builder/package.json:16
@crowd/commonis not imported anywhere in this new package. Remove the unused dependency so the package declares only its actual runtime requirements.
This issue also appears on line 19 of the same file.
"@crowd/common": "workspace:*",
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 12 changed files in this pull request and generated no new comments.
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
Suppressed comments (5)
services/libs/integration-builder/src/credentials.ts:12
- This lookup discards the integration-specific identity and returns the same global app credential for every GitHub integration. GitHub integrations are scoped by an installation ID (
integrationIdentifier), whilegithub-nangointegrations use mapped connection IDs; becauseManifest.discoverreceives only this credential, it cannot restrict discovery to the requested integration and can associate another installation's channels with it. Include the relevant installation/connection identity in the credential (or pass the integration identity into discovery) and handle the two platform credential models separately.
const integration: { platform: string } | null = await qx.selectOneOrNone(
`SELECT platform
FROM integrations
WHERE id = $(integrationId) AND "deletedAt" IS NULL`,
services/libs/integration-builder/src/credentials.ts:31
- This comment describes the implementation and an unticketed future replacement rather than an allowed invariant or external quirk. Remove it; the helper name and environment-variable reads are self-explanatory.
// POC only: secrets come from env; secret-manager adoption (OCI Vault) swaps this
// body without touching callers — getCredential stays the single entry point
services/libs/integration-builder/src/types.ts:15
- These POC/future-design notes do not document an external quirk, invariant, constraint, legacy complexity, or ticketed TODO. Remove them and let the credential type express the currently supported variant.
// POC only: single variant; becomes a discriminated union (token, oauth2, ...)
// as more connectors land
services/libs/integration-builder/src/credentials.ts:21
- This scope note describes the current implementation and future work without a ticket, which is not an allowed code-comment case. Remove it; the switch already makes the supported platforms clear.
This issue also appears on line 30 of the same file.
// POC scope: GitHub only; each migrated connector adds its platform case here
services/libs/data-access-layer/src/integrationBuilder/syncUnits.ts:40
- The new atomic-claim behavior has no database integration test. Add coverage that runs concurrent claims and verifies an ID is returned once, while active/due, deleted-integration, and expired-lease filtering behave as intended; comparable data-access SQL is exercised in
services/libs/data-access-layer/src/packages/*.integration.test.ts.
export async function claimDueUnits(qx: QueryExecutor, limit: number): Promise<ISyncUnit[]> {
return qx.select(
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 62 out of 64 changed files in this pull request and generated 3 comments.
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
Suppressed comments (7)
Previously missed (4) — in code that hasn't changed since the last review.
services/libs/connectors/src/connectors/github/mappers/prComment.ts:18
- This stores the pull request URL instead of the comment permalink even though
PrCommentNode.urlis available. Consumers will be taken to the PR root rather than the specific comment; the legacy mapper uses the comment URL (services/libs/integrations/src/integrations/github/processData.ts:712).
services/libs/connectors/src/connectors/github/mappers/prComment.ts:20 - This preserves GitHub's uppercase state, while the existing GitHub pipeline normalizes PR states to lowercase (
services/libs/integrations/src/integrations/github/processData.ts:443,478,523,739). Mixed values such asOPENandopenwill split downstream state-based analytics.
services/libs/connectors/src/connectors/github/mappers/reviewThreadComment.ts:24 - This preserves GitHub's uppercase state, while the existing GitHub pipeline normalizes PR states to lowercase (
services/libs/integrations/src/integrations/github/processData.ts:443,478,523,739). Mixed values such asOPENandopenwill split downstream state-based analytics.
services/libs/connectors/src/connectors/github/mappers/issue.ts:25 - Selecting only the first
ClosedEventdrops later closures when an issue was reopened and closed again. The legacy pipeline iterates every returned close event (services/libs/integrations/src/integrations/github/processStream.ts:1002-1026), and the new query also caps the timeline at 10, so this loses valid lifecycle activities. Paginate the close-event connection and map every event.
services/libs/connectors/src/connectors/github/paging.ts:7
- An unbounded page loop conflicts with the sync activity's 30-minute
startToCloseTimeout. On a large repository the activity can time out before returning, and non-rate-limit failures only callrecordRunFailure, so the in-memory committed watermark is lost and the next run restarts the same backfill. Bound work per run (including child pagination) so the workflow can persist progress before the timeout.
services/apps/connectors_worker/src/workflows/dispatcher.ts:22 - Starting the workflow before writing the cadence creates a race with
executeSync: a fast rate-limit path can persist its providerresumeAt, after which this reschedule overwrites it with the normal cadence. That can retry before reset or delay recovery. Establish the next cadence before dispatch with rollback/fencing on start failure, or make these updates conditional on the run being dispatched.
await activity.startRun(unit)
await activity.reschedule(unit.id, unit.platform, unit.syncName)
services/libs/connectors/src/pool/tokenPool.ts:174
- The budget check and decrement are separate Redis operations. Concurrent sync units can all read
remaining = 1, each decrement it, and all issue a provider request, overshooting the shared limit by the concurrency level. Reserve budget atomically (for example with a Lua script or transaction that checks and decrements together) before returning the token.
| newSince = pullRequests[0].updatedAt | ||
| } | ||
|
|
||
| const fresh = pullRequests.filter((pr) => new Date(pr.updatedAt) > sinceDate) |
There was a problem hiding this comment.
Confirmed, and deliberately not fixed in this PR. prWalk.ts compares with a strict > against a second-granularity timestamp, and the same page also satisfies reachedSince and ends the walk, so a PR updated inside the watermark's second is lost permanently rather than retried on the next run. discussions.ts already uses >=, so this is an inconsistency inside our own code rather than a judgement call.
Held back by policy, not by doubt: the fix changes the set of emitted records (the boundary items are re-emitted every run and deduped downstream by sourceId), and every change that affects emitted record values or counts is batched into one reviewed change validated against a full comparison run rather than patched piecemeal. Tracked in the POC backlog as item 13e. Leaving this thread open so the finding stays visible.
| const { pageInfo, nodes } = data.repository.forks | ||
| const forks = nodes | ||
| .filter((node): node is ForkNode => node !== null) | ||
| .filter((node) => !since || node.createdAt > since) |
There was a problem hiding this comment.
Confirmed, same class as the prWalk boundary thread: forks.ts filters createdAt > since while the stored cursor advances past a fork sharing that timestamp, so it is never emitted.
Deferred by the same policy — it changes the emitted record set, so it lands with the other record-value fixes in one reviewed change validated against a comparison run. Tracked in the POC backlog as item 13e; leaving this thread open.
| type: 'username', | ||
| verified: true, | ||
| value: user.login, | ||
| sourceId: user.id?.toString() ?? '', |
There was a problem hiding this comment.
Confirmed. ORGANIZATION_FIELDS and BOT_FIELDS select databaseId, not id, so this branch produces a verified identity with an empty sourceId for organization-owned forks. The same root cause also empties attributes.sourceId.github for bot actors, which no review flagged. Fix is to prefer databaseId and fall back to the GraphQL id.
Deferred by policy rather than doubt: it changes values on emitted records, and all record-value fixes are batched into one reviewed change validated against a comparison run. Tracked in the POC backlog as item 13d; leaving this thread open.
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
…ifest Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
| body: comment.bodyText || '', | ||
| url: comment.url, | ||
| attributes: { isAnswer: discussion.isAnswered ?? false }, | ||
| member: toMember(discussion.author), |
There was a problem hiding this comment.
Comments attributed to discussion author
High Severity
toDiscussionCommentActivity sets member from discussion.author instead of the comment author. REPLY_FIELDS also omits author, so comments and replies cannot be attributed correctly. Every discussion comment would land on the thread starter, corrupting member activity data in the sink.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit e918730. Configure here.
There was a problem hiding this comment.
Confirmed, and the highest-impact finding in this review pass: every discussion comment and reply is currently attributed to the thread starter. You are also right that REPLY_FIELDS does not select author, so the GraphQL query has to change together with the mapper. The legacy contract (processStream.ts:1162-1213) resolves each comment's and reply's own author, including bot and deleted-member handling.
Deliberately not fixed in this PR: it changes values on emitted activity records, and all record-value fixes are batched into one reviewed change validated against a full comparison run rather than patched piecemeal. Tracked in the POC backlog as item 13a. Leaving this thread open so it stays visible.
| : GITHUB_GRID[GithubActivityType.DISCUSSION_COMMENT].score, | ||
| body: comment.bodyText || '', | ||
| url: comment.url, | ||
| attributes: { isAnswer: discussion.isAnswered ?? false }, |
There was a problem hiding this comment.
Answer flag uses discussion state
Medium Severity
Score and isAnswer are derived from discussion.isAnswered rather than the comment's own isAnswer. REPLY_FIELDS does not request isAnswer, so the accepted answer cannot be identified. Every top-level comment on an answered thread would get the answer score boost and isAnswer: true.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit e918730. Configure here.
There was a problem hiding this comment.
Confirmed. The mapper reads discussion.isAnswered, so every top-level comment on an answered thread gets isAnswer: true and the +2 score bonus; the legacy contract (processData.ts:797-801) uses the comment's own isAnswer, which REPLY_FIELDS does not select. Same query change as the comment-attribution thread.
Deferred by the same policy — it changes values on emitted records and lands with the other record-value fixes in one reviewed change validated against a comparison run. Tracked in the POC backlog as item 13b; leaving this thread open.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 68 out of 70 changed files in this pull request and generated 3 comments.
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
Suppressed comments (9)
Previously missed (5) — in code that hasn't changed since the last review.
services/apps/connectors_worker/src/workflows/dispatcher.ts:15
claimDueinherits the three-attempt activity retry policy, but claiming commits locks while returning the selected IDs. If the database commits and the activity response is lost, a retry claims a different batch and leaves the first batch unavailable for the five-minute lease. Make the claim idempotent with a claim owner/token, or give this activity a single-attempt policy.
const units = await activity.claimDue(CLAIM_LIMIT)
services/libs/connectors/src/connectors/github/mappers/issue.ts:26
- Selecting one close event only when the issue is currently closed drops valid closure activities for reopened issues and collapses repeated close/reopen cycles. The existing processor emits every
ClosedEvent(services/libs/integrations/src/integrations/github/processStream.ts:1002-1024); iterate all closure events and ensure the query retrieves the full relevant history.
services/libs/connectors/src/connectors/github/mappers/prComment.ts:18 - Use the comment permalink here. The query already returns
comment.url; storing the pull request URL means every comment activity links to the PR top instead of the specific comment (the existing processor uses the comment URL atservices/libs/integrations/src/integrations/github/processData.ts:712).
This issue also appears on line 20 of the same file.
services/libs/connectors/src/connectors/github/mappers/reviewThreadComment.ts:24
- Normalize this pull-request state to lowercase. The established GitHub mapping uses lowercase at
services/libs/integrations/src/integrations/github/processData.ts:739; this currently emits the GraphQL enum value in uppercase.
services/libs/connectors/src/pool/tokenPool.ts:200 - Once a pool contains tokens but all are parked or quarantined, returning
trueadmits work that cannot acquire a token. Keep the initial empty-pool bootstrap behavior, but returnfalsewhen state exists and no token is healthy.
services/libs/connectors/src/connectors/github/mappers/prComment.ts:20
- Normalize this pull-request state to lowercase. The established GitHub mapping does this at
services/libs/integrations/src/integrations/github/processData.ts:739, and the other new PR mapper already follows that format; leaving this value asOPEN/CLOSED/MERGEDcreates inconsistent activity attributes.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:49 discussion.isAnsweredonly says the discussion has an accepted answer; it does not identify this comment. As written, every comment and reply in an answered discussion is marked as an answer, and every top-level comment receives the +2 score. Select the comment-levelisAnswerfield and use it for both the attribute and score, as the existing processor does atprocessData.ts:797-802.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:50- This attributes every discussion comment and reply to the discussion opener.
REPLY_FIELDScurrently omits each comment's author, but the existing integration maps the comment/reply author (processStream.ts:1162-1213). Include author fields in the GraphQL response and mapcomment.authorhere.
services/libs/connectors/src/connectors/github/paging.ts:7 - This removes any bound on provider pagination even though the sync activity has a 30-minute timeout. A large backfill can time out before
recordRunSuccess, and non-rate-limit failures do not persist the in-memory committed watermark, so the next run starts from the same place and may never finish. Use bounded, resumable runs whose progress is durably committed before the activity timeout.
| type: GithubActivityType.DISCUSSION_COMMENT, | ||
| timestamp: comment.createdAt ?? DEFAULT_TIMESTAMP, | ||
| sourceId: comment.id, | ||
| sourceParentId: discussion.id, |
There was a problem hiding this comment.
Confirmed, and thanks for the legacy reference. The contract is: a top-level comment takes sourceParentId = discussion.id (processData.ts:789) and a reply takes the parent comment's id (processStream.ts:1204-1211), with a reply carrying no isAnswer attribute at all. The two other lines you point at are the sibling findings on author attribution and the answer flag.
Deferred by the same policy — it changes values on emitted records and lands with the other record-value fixes in one reviewed change validated against a comparison run. Tracked in the POC backlog as item 13c; leaving this thread open.
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 68 out of 70 changed files in this pull request and generated 2 comments.
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
Suppressed comments (17)
Previously missed (8) — in code that hasn't changed since the last review.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:22
- This converts a boolean GraphQL field into the string values
"true"/"false". Existing GitHub activities storecategory.isAnswerableas a boolean (services/libs/integrations/src/integrations/github/processData.ts:1384-1391), so preserve the boolean type for downstream filters.
This issue also appears in the following locations of the same file:
- line 44
- line 50
services/libs/connectors/src/connectors/github/mappers/fork.ts:18
isIndirectForkis emitted as a string, whereas the established GitHub activity contract stores it as a boolean (services/libs/integrations/src/integrations/github/processData.ts:356-360). This type mismatch will break boolean filtering.
services/libs/connectors/src/connectors/github/mappers/member.ts:145- For normal GitHub users this discards the fetched profile name and always displays the login. The existing mapper prefers the trimmed name and falls back to login (
services/libs/integrations/src/integrations/github/processData.ts:123), so retain that user-facing data.
This issue also appears on line 146 of the same file.
services/libs/connectors/src/connectors/github/mappers/prComment.ts:18
- This points a pull-request comment activity at the pull request rather than the comment itself. The established mapper uses the comment URL (
services/libs/integrations/src/integrations/github/processData.ts:707-716), which is needed to navigate directly to the activity.
services/libs/connectors/src/connectors/github/mappers/prComment.ts:20 - GitHub GraphQL returns pull-request states as uppercase enum values, but this activity contract normalizes them to lowercase (
services/libs/integrations/src/integrations/github/processData.ts:739). Emitting the raw value creates inconsistent state facets.
services/libs/connectors/src/connectors/github/mappers/reviewThreadComment.ts:24 - GitHub GraphQL returns pull-request states as uppercase enum values, but review-thread comment activities are established to use lowercase state (
services/libs/integrations/src/integrations/github/processData.ts:739). Normalize this value before emission.
services/libs/connectors/src/pool/tokenPool.ts:200 - When tokens exist but all are parked or quarantined, this returns
true, so the dispatcher admits work that has no usable token. Distinguish an unseeded pool (which may proceed to seed) from a known unhealthy pool, and defer the latter.
services/apps/connectors_worker/src/activities/syncRunActivities.ts:86 - This new worker depends directly on a class under the legacy
old/DAL path.CLAUDE.md:40-43requires new database code to use QueryExecutor-backed plain functions rather than reintroducing class-based repositories. Expose the result-publishing operation as a functional DAL API and injectqxhere.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:45
discussion.isAnsweredonly means that the discussion has some selected answer; it does not mean this particular top-level comment is the answer. This gives every top-level comment in an answered discussion the answer bonus (and line 49 also marks all of them as answers). Fetch and use the comment-levelisAnswerfield instead, as the existing processor does atprocessData.ts:797-802.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:42- Replies are linked directly to the discussion, even though their parent activity is the comment they reply to. The existing pipeline sets a reply's
sourceParentIdto the parent comment ID (processStream.ts:1188-1213); pass that ID into this mapper whenisReplyis true so activity relationships remain intact.
services/libs/connectors/src/connectors/github/mappers/member.ts:136 - For an Organization actor,
ORGANIZATION_FIELDSprovidesdatabaseIdbut notid; this branch therefore emits an empty identitysourceId. Use the available database ID first so organization-authored activities do not create identities without a stable source key.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:50 - Every discussion comment and reply is attributed to the discussion author.
REPLY_FIELDSdoes not fetch an author, while the existing ingestion path mapsrecord.author/reply.author(services/libs/integrations/src/integrations/github/processStream.ts:1162-1213). Fetch the comment author and pass it totoMemberto avoid corrupting member attribution.
services/libs/connectors/src/pool/tokenPool.ts:227 - A freshly minted installation token has a new value, so this branch replaces the entire state and clears
parkedUntil/quarantined. BecauseseedGithubTokensruns at every sync, rate-limit and quarantine decisions are effectively undone by the next unit. Update the token value while preserving its state metadata.
services/apps/connectors_worker/src/activities/dispatcherActivities.ts:29 - Each unit independently checks the same pool snapshot, but no budget is reserved. For example, 100 units sharing one integration can all observe 50 remaining requests and all be admitted despite the 50-request estimate. Group admission by pool and atomically reserve/decrement the estimate so concurrent checks cannot over-admit.
const headrooms = await mapWithConcurrency(units, BUDGET_PROBE_CONCURRENCY, (unit) => {
const manifest = findManifest(unit.platform)
const pool = createTokenPool(svc.redis, unit.platform, unit.integrationId, {
probeBudget: manifest?.probeBudget,
})
return pool.hasHeadroom(DEFAULT_RUN_ESTIMATE)
services/libs/connectors/src/connectors/github/paging.ts:7
- Runs are unbounded here, but the Temporal activity has a 30-minute start-to-close timeout and
commitWatermarkis only persisted when the activity returns successfully (or explicitly handles a rate limit). A sufficiently large repository will repeatedly time out and restart without durable progress. Use a finite work bound and continuation watermark semantics for every sync.
services/libs/connectors/src/connectors/github/mappers/pullRequest.ts:45 - Team review requests are always dropped: the query returns
name/idforTeam, but this guard requireslogin. The existing GitHub ingestion expands team requests into activities for each team member (processStream.ts:599-626); fetch the team members and emit those review requests rather than silently skipping them.
services/libs/connectors/src/connectors/github/mappers/member.ts:150 - The mapper fetches
user.emailbut emits only the username identity. The established GitHub processor adds a verified email identity when present (services/libs/integrations/src/integrations/github/processData.ts:146-153); omitting it removes a key member-matching and deduplication signal.
| if (!item.id) { | ||
| return null | ||
| } | ||
| const timestamp = item.submittedAt ?? DEFAULT_TIMESTAMP |
There was a problem hiding this comment.
Confirmed. A pending review falls back to DEFAULT_TIMESTAMP and that epoch value is then baked into the generated gen-PRR_ sourceId, so the record is permanent. The legacy guard (processStream.ts:631) is the right behaviour: skip reviews with no submittedAt. Low probability under an app token, since pending reviews are visible only to their author, but wrong when it happens.
Deferred by the same policy — it changes the emitted record set and lands with the other record-value fixes in one reviewed change validated against a comparison run. Tracked in the POC backlog as item 13f; leaving this thread open.
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
…s only Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 68 out of 70 changed files in this pull request and generated 1 comment.
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
Suppressed comments (14)
Previously missed (4) — in code that hasn't changed since the last review.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:22
isAnswerableis a boolean in both GraphQL and the existing activity contract, but this converts it to the strings"true"/"false". That changes attribute typing for downstream consumers.
This issue also appears in the following locations of the same file:
- line 43
- line 50
services/libs/connectors/src/connectors/github/mappers/fork.ts:20
isForkByOrgis only emitted when the direct parent is itself a fork. A direct fork owned by an organization hasisInOrganization = trueandparent.isFork = false, so it incorrectly gets no organization marker; build the organization and indirect-fork attributes independently.
services/libs/connectors/src/connectors/github/mappers/prComment.ts:18- This links pull-request comment activities to the PR page even though the fetched comment has its own URL. Consumers therefore cannot navigate to the specific comment, unlike the existing ingestion mapping.
This issue also appears on line 20 of the same file.
services/libs/connectors/src/connectors/github/mappers/reviewThreadComment.ts:24
- GraphQL returns uppercase states, while GitHub activity attributes elsewhere are normalized to lowercase. Emitting
OPEN/CLOSEDhere creates inconsistent values for the same attribute.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:42
- Replies are attached to the discussion instead of their parent comment. The existing ingestion contract uses the top-level comment ID as
sourceParentIdfor replies, so this breaks reply threading; pass the parent comment ID into the mapper while keeping the discussion ID only for top-level comments.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:46 - This tests the discussion-wide
isAnsweredflag, so every top-level comment in an answered discussion receives the answer bonus. GitHub exposesisAnswerper comment; query and map that field so only the accepted answer gets the extra score.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:50 - Every discussion comment and reply is attributed to the discussion author. The comment query currently omits authors entirely, so replies by other users will create activities for the wrong member; fetch each comment/reply author and map that author here.
services/libs/connectors/src/connectors/github/mappers/pullRequest.ts:46 - The GraphQL query explicitly returns teams, but this guard drops every team review request because teams do not have
login. This loses review-request activities that the existing ingestion path expands to team members; either query/map team members or represent team requests explicitly.
services/libs/connectors/src/connectors/github/mappers/prComment.ts:20 - GraphQL returns uppercase states, while GitHub activity attributes elsewhere are normalized to lowercase. Emitting
OPEN/CLOSEDhere creates inconsistent values for the same attribute.
services/libs/connectors/src/connectors/github/mappers/issue.ts:26 - Closed activities are emitted only when the issue is currently closed, so an issue that was closed and later reopened loses its historical close event. Iterate over the fetched
ClosedEventnodes regardless of current state, as the existing ingestion path does.
services/libs/connectors/src/pool/tokenPool.ts:227 - Replacing a minted token currently discards
parkedUntilandquarantined, contradicting the stated invariant that quarantined tokens are never revived automatically. It also immediately revives a rate-limited installation whenever another unit mints a fresh token value, even though the installation shares the same budget.
services/libs/connectors/src/pool/tokenPool.ts:173 - The remaining-budget check and decrement are separate Redis operations. Concurrent syncs can both observe
remaining = 1, both decrement, and both use the token, oversubscribing the provider budget; perform the positive check, decrement, and selection atomically (for example with Lua).
services/apps/connectors_worker/src/activities/dispatcherActivities.ts:30 - Each unit checks the same shared snapshot independently without reserving its estimated cost. For example, 100 units all pass against 50 remaining requests and are launched together, so this admission stage does not enforce the aggregate budget; atomically reserve estimates per pool while admitting units.
return pool.hasHeadroom(DEFAULT_RUN_ESTIMATE)
services/libs/connectors/src/connectors/github/paging.ts:7
- An infinite page cap makes large backfills run until the activity's 30-minute timeout. Because committed watermarks are only persisted after the sync returns (or handles a ConnectorError), a timeout loses all in-memory progress and the next attempt restarts the same backfill indefinitely; use bounded runs with resumable persisted cursors.
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 70 out of 72 changed files in this pull request and generated 3 comments.
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
Suppressed comments (11)
Previously missed (7) — in code that hasn't changed since the last review.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:46
discussion.isAnsweredonly says that the discussion has an accepted answer, so this awards the answer bonus to every top-level comment whenever any answer exists. Query and map each comment'sisAnswerfield instead, matching the existing mapping inprocessData.ts:787-802.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:50- Every discussion comment and reply is attributed to the discussion author.
REPLY_FIELDSdoes not fetch an author, but existing ingestion maps each comment/reply's own author (processStream.ts:1162-1213). Fetch the author on comment nodes and mapcomment.author.
services/libs/connectors/src/connectors/github/mappers/member.ts:145 - Ordinary users always get their login as
displayNameeven though the query fetchesname; this regresses the existing GitHub mapping, which prefers the trimmed profile name (processData.ts:123). Preserve that behavior so member display names do not lose profile data.
services/libs/connectors/src/connectors/github/mappers/prComment.ts:18 - This links the activity to the pull request rather than the comment itself. The existing GitHub mapper uses the comment URL (
processData.ts:707-716), andcomment.urlis already available here, so consumers otherwise land on the wrong object.
services/libs/connectors/src/connectors/github/mappers/prComment.ts:20 - GraphQL returns
OPEN,CLOSED, orMERGED, but existing GitHub activity state attributes are normalized to lowercase. Keeping this uppercase creates inconsistent analytics values for PR comments.
services/libs/connectors/src/connectors/github/mappers/pullRequest.ts:143 - Backfilled closed and merged PRs are emitted with
state: 'open'on their opened activity. The existing mapper records the PR's current normalized state (processData.ts:443), so this hard-coded value produces inaccurate state attributes.
services/libs/connectors/src/connectors/github/mappers/reviewThreadComment.ts:24 - GraphQL returns uppercase pull-request states, while the existing review-comment mapping normalizes them (
processData.ts:739). This introduces inconsistent state values for the same activity type.
services/libs/connectors/src/pool/tokenPool.ts:227
- Each run mints a new installation-token value before seeding it. Replacing changed values with
{ value }clearsparkedUntilandquarantined, so the next run immediately revives a rate-limited or rejected installation despite the stated quarantine behavior. Preserve the token state when refreshing its value.
services/libs/connectors/src/pool/tokenPool.ts:173 - The budget check and decrement are separate Redis operations. Concurrent sync activities can all observe positive
remainingand then decrement it below zero while using the same token, exceeding the provider budget. Token selection/reservation and the LRU update need one atomic Redis operation (for example, Lua).
services/libs/connectors/src/connectors/github/mappers/discussion.ts:42 - Replies currently receive the discussion ID as their parent, flattening the reply hierarchy. The established GitHub mapping uses the parent comment ID for replies (
processStream.ts:1188-1211). Pass the parent comment ID when mapping nested replies while retaining the discussion ID for top-level comments.
services/libs/connectors/src/connectors/github/mappers/member.ts:136 - Organization actors enter this fallback branch, but the GraphQL organization fragment fetches
databaseId, notid, so their required identitysourceIdbecomes an empty string. Use the available database ID before falling back to a node ID.
| cursor = pageInfo.endCursor | ||
| } | ||
|
|
||
| return { complete: false } |
There was a problem hiding this comment.
Confirmed, and a good catch — this is worse than a resume inefficiency. On budget expiry runIncremental returns without committing anything at all: the DESC cursor is dropped and newSince is never persisted either. So the next run restarts from the newest PR with the same old since, re-walks the same newest pages, and burns the same budget; older fresh PRs are never reached and their activities are never emitted. runBackfill already has the right shape (commit since + cursor per page), so the fix is to do the same here and only advance since to newSince once the walk actually reaches since.
Deferred by policy, not by doubt: it changes which records get emitted, and every change affecting emitted record values or counts is batched into one reviewed change validated against a full comparison run rather than patched piecemeal. Tracked in the POC backlog as item 13h; leaving this thread open so the finding stays visible.
| } | ||
| } | ||
|
|
||
| while (ctx.hasRunBudget()) { |
There was a problem hiding this comment.
Child pagination ignores run budget
Medium Severity
hasRunBudget is only consulted at the outer page loop. Unbounded child walks (issue comment drain, per-PR commits, discussion replies) can exceed the 10-minute buffer, trip the 60-minute activity timeout, and skip recordRunSuccess, dropping every in-memory watermark from that run.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit accd746. Configure here.
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
…ease Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
| const states = await readStates() | ||
| const healthy = [...states.entries()].filter(([, state]) => isHealthy(state, nowMs)) | ||
| if (healthy.length === 0) { | ||
| return states.size === 0 |
There was a problem hiding this comment.
Quarantined tokens stall all syncs
Medium Severity
hasHeadroom now returns false when the pool has tokens but none are healthy. A single 401 quarantines the installation token permanently, and seed now keeps that flag across remints. Admission never happens, so executeSync never remints and recordRunFailure never dead-letters; units loop on shortDeferRunAt forever.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit cb814c1. Configure here.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 70 out of 72 changed files in this pull request and generated 1 comment.
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
Suppressed comments (22)
Previously missed (9) — in code that hasn't changed since the last review.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:22
category.isAnswerableis emitted as a boolean by the existing GitHub mapper (processData.ts:958-966), but this converts it to a string. Preserve the boolean type to avoid breaking consumers of the category attributes.
This issue also appears in the following locations of the same file:
- line 43
- line 50
services/libs/connectors/src/connectors/github/mappers/fork.ts:18
isIndirectForkis a boolean in the existing GitHub activity contract (processData.ts:349-360), but this converts it to the string"true". Keep the boolean type so downstream filters and aggregations continue to work.
services/libs/connectors/src/connectors/github/mappers/issue.ts:20- Backfilled issue-opened activities should carry the issue's current normalized state, as the existing mapper does (
processData.ts:651-662). Hard-codingopenreports closed issues as open in this activity's attributes.
This issue also appears on line 44 of the same file.
services/libs/connectors/src/connectors/github/mappers/member.ts:145
- Normal GitHub users always get their login as
displayName, despite the query providingname. The existing mapper prefers the trimmed profile name (processData.ts:123), so this degrades member display names for nearly every user with a configured name.
services/libs/connectors/src/connectors/github/mappers/prComment.ts:18 - The activity URL should point to the comment, not the pull request.
comment.urlis already queried, and the existing pipeline emits it (processData.ts:700-717); using the PR URL prevents consumers from linking to the specific comment.
This issue also appears on line 20 of the same file.
services/libs/connectors/src/connectors/github/mappers/pullRequest.ts:143
- Backfilled PR-opened activities should carry the PR's current normalized state, as the existing mapper does (
processData.ts:433-449). Hard-codingopenreports merged and closed PRs as open in this activity's attributes.
services/libs/connectors/src/connectors/github/syncs/discussions.ts:142 - When the run budget expires before reaching the previous watermark or end of the collection, no cursor or progress is committed. The next run starts from page one and repeats the same expensive comment/reply expansion, so large discussion backfills can never complete. Persist resumable backfill/incremental cursor state before returning incomplete.
services/apps/connectors_worker/src/main.ts:36 - This comment only describes the immediately following development-only branch, so it is disallowed by the repository's self-explanatory-code policy.
services/libs/connectors/src/connectors/github/mappers/reviewThreadComment.ts:24 - Pull-request state attributes are normalized to lowercase throughout the existing GitHub mapping (for example
processData.ts:739andmappers/pullRequest.ts:39). Emitting GraphQL's uppercase state here creates a second representation for the same field.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:42
- Replies are parented to the discussion instead of the comment they reply to. The existing GitHub pipeline passes the parent comment ID (
processStream.ts:1204-1211), so this flattens reply threads and changessourceParentIdsemantics. Pass the parent comment ID into this mapper for replies.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:46 discussion.isAnsweredonly indicates that the discussion has an accepted answer; it does not identify which comment is the answer. This awards the answer bonus to every top-level comment and marks all of them as answers. Query and map each comment'sisAnswerfield, as the existing pipeline does inprocessData.ts:787-802.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:50- Every discussion comment and reply is attributed to the discussion author. The GraphQL comment fragment currently omits
author, even though the existing pipeline maps each record's own author (processStream.ts:1162-1213). Fetch the comment/reply author and pass it totoMemberhere.
services/libs/connectors/src/connectors/github/prWalk.ts:92 - Incremental walks do not commit
cursorornewSincewhen the run budget expires. A backlog large enough to span multiple runs therefore restarts at the newest PR on every run and never reaches the old watermark. Store resumable incremental state before returningcomplete: false.
services/libs/connectors/src/connectors/github/mappers/issue.ts:26 - This emits at most one close event and only when the issue is currently closed. Reopened issues lose all historical close activities, while repeatedly closed issues lose every close except the first returned event; the existing stream iterates every
CLOSED_EVENT(processStream.ts:1003-1024). Map all close events and paginate the timeline connection rather than usingfind.
services/libs/connectors/src/connectors/github/mappers/issue.ts:44 - When the closer account is unavailable, this attributes the close to the issue author. A null actor represents a missing/deleted closer, and the existing pipeline maps that case to the ghost member; using the opener corrupts attribution.
services/libs/connectors/src/connectors/github/mappers/prComment.ts:20 - Pull-request state attributes are normalized to lowercase throughout the existing GitHub mapping (for example
processData.ts:739andmappers/pullRequest.ts:39). Emitting GraphQL's uppercase state here creates a second representation for the same field.
services/libs/connectors/src/connectors/github/mappers/member.ts:127 - Organization actors enter this generic branch, but
ORGANIZATION_FIELDSprovidesdatabaseIdrather thanid, yielding an empty identitysourceId; the branch also omits the establishedisOrganizationmember attribute. Forks/comments authored by organizations will therefore be ingested as ordinary members with incomplete identity metadata. Add an explicit Organization mapping and schema support.
services/libs/connectors/src/pool/tokenPool.ts:173 - The positive-budget check and this decrement are not atomic. Concurrent sync runs can all read the same positive
remainingvalue, then each decrement it and obtain the token, driving the bucket below zero and exceeding the provider quota. Reserve capacity with a conditional Redis Lua/transaction operation and rotate when the reservation fails.
services/libs/connectors/src/types.ts:19 - This forward-looking implementation note violates the repository rule against comments describing future changes. Remove it or convert it to a ticketed TODO if the follow-up must be tracked.
services/libs/connectors/src/credentials.ts:17 - This comment only restates the current POC scope and future implementation plan. The repository permits future-work comments only as ticketed TODOs.
services/libs/connectors/src/credentials.ts:28 - This is an unticketed future implementation note, which the repository comment policy disallows. Track the vault migration with a
TODO(CM-XXX)or remove the note.
pnpm-lock.yaml:3581 - Adding the two workspace importers also re-resolves unrelated existing dependencies across the monorepo (for example, downgrading every
esbuild0.28.2 snapshot to 0.28.0 and changing AWS/debug/follow-redirects entries). This expands the rollout beyond the connector POC and can alter other services. Regenerate the lockfile with the repository's pinned pnpm version while preserving unrelated resolutions, or revert the unrelated chunks.
| async seed(tokenId: string, value: string): Promise<void> { | ||
| const json = await redis.hGet(tokensKey, tokenId) | ||
| const state = json ? (JSON.parse(json) as ITokenState) : null | ||
| const next = state ? { ...state, value } : { value } |
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 5 total unresolved issues (including 4 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 91b39f8. Configure here.
| alreadyRunning, | ||
| failed, | ||
| durationMs: Date.now() - startedAt, | ||
| }) |
There was a problem hiding this comment.
Dispatch summary activity never registered
Medium Severity
The dispatcher now calls logDispatchSummary at the end of every tick, but that function is not re-exported from activities.ts. The worker only registers activities from that barrel, so Temporal cannot find the activity and the dispatcher workflow fails after it has already claimed, started, and deferred units.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 91b39f8. Configure here.
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 71 out of 73 changed files in this pull request and generated 2 comments.
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
Suppressed comments (19)
Previously missed (9) — in code that hasn't changed since the last review.
services/libs/connectors/src/concurrency.ts:7
- Invalid limits such as
0, negatives, or non-integers silently create too few/no workers and can return an unprocessed sparse result. This exported utility should reject invalid concurrency values, as the existing packages-worker equivalent does.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:46 discussion.isAnsweredsays only that the discussion has an accepted answer, so this marks and boosts every top-level comment as the answer. Fetch each comment'sisAnswerfield and use that value instead.
This issue also appears on line 50 of the same file.
services/libs/connectors/src/connectors/github/mappers/prComment.ts:20
- GitHub GraphQL returns PR state as uppercase, while existing GitHub activity attributes normalize it to lowercase. Leaving this as
OPEN/CLOSED/MERGEDbreaks consumers expecting the established lowercase values.
services/libs/connectors/src/connectors/github/mappers/reviewThreadComment.ts:24 - Normalize the GraphQL PR state to lowercase. The existing GitHub mapping uses lowercase state attributes, so emitting uppercase values here creates inconsistent activity data.
services/libs/connectors/src/connectors/github/syncs/discussions.ts:145 - A budget-limited walk returns without saving
cursor, so the next run starts from the newest discussions again. On repositories whose fresh discussion/comment fan-out exceeds one run budget, the sync can repeatedly process the same prefix and never reach the watermark. Store resumable cursor state and only promotenewestUpdatedAtafter the walk completes.
services/libs/connectors/src/credentials.ts:11 - This duplicates the existing
fetchIntegrationByIdDAL query, which already filters deleted integrations and returnsplatform. Reuse that function so integration lookup semantics remain centralized in the data-access layer.
pnpm-lock.yaml:6909 - The new workspace importers also downgrade the existing shared
tsx/esbuild snapshot from 0.28.2 to 0.28.0 and introduce unrelated AWS/debug lockfile rewrites. This broadens the PR's dependency impact beyond the connector POC; regenerate with the repository's pinned pnpm version and retain only resolutions required by the two new workspaces.
services/apps/connectors_worker/src/main.ts:37 - This comment just describes the following development-only branch. Remove it;
IS_DEV_ENVanddummyConnectoralready express the behavior.
services/libs/connectors/src/pool/tokenPool.ts:22 - This POC/future-platform note is an architectural plan rather than an allowed external quirk, invariant, concurrency constraint, legacy explanation, or ticketed TODO. Remove it or reference a concrete
TODO(CM-XXX)if follow-up work is required.
services/libs/connectors/src/pool/tokenPool.ts:228
- Refreshing a token preserves
quarantined: true. After one expired installation token receives a 401, every later run mints a valid replacement under the same ID butacquire()permanently rejects it, eventually dead-lettering the unit. Clear quarantine when seeding a replacement token.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:50 - Every discussion comment and reply is attributed to the discussion author, because the comment GraphQL fragments do not fetch an author. This corrupts member attribution whenever another user comments. Fetch each comment/reply author and map that author here, matching the existing GitHub ingestion behavior.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:42 - Replies currently use the discussion as
sourceParentId, flattening the thread hierarchy. The existing GitHub pipeline parents replies to the top-level comment, so pass the comment ID fromcollectRepliesand use it here.
services/libs/connectors/src/connectors/github/prWalk.ts:92 - If the run budget expires before the descending walk reaches the old watermark, no cursor or new boundary is committed. The next run restarts at page one and reprocesses the same newest PRs, so a sufficiently large backlog can prevent older fresh PRs from ever being reached. Persist resumable incremental state while keeping the original
sinceboundary stable.
services/libs/connectors/src/connectors/github/syncs/issues.ts:37 - This advances
sinceafter each page but deliberately discards the page cursor. If the budget expires with more pages, restarting from the timestamp boundary can skip or repeatedly revisit issues sharing thatupdatedAt. Checkpoint the fixed query boundary plus cursor for partial runs, and advancesinceonly after pagination completes.
services/libs/connectors/src/connectors/github/syncs/issueComments.ts:118 - The partial-run checkpoint advances
sincewhile droppingcursor. When multiple issues at the page boundary share anupdatedAt, the next run can skip comments for remaining issues or repeatedly process the first page. Preserve the original query boundary and page cursor until the walk completes.
services/apps/connectors_worker/src/activities/dispatcherActivities.ts:28 - This check does not reserve the estimated budget. When several due units share an integration, each sees the same remaining snapshot and all can be admitted (for example, 100 units can each pass against the same 50 requests), so execution can overrun the provider limit. Group by pool and atomically reserve/release budget during admission.
const pool = createTokenPool(svc.redis, unit.platform, unit.integrationId, {
probeBudget: manifest?.probeBudget,
})
return pool.hasHeadroom(DEFAULT_RUN_ESTIMATE)
services/libs/connectors/src/types.ts:19
- This POC/future-plan note does not document an external quirk, invariant, concurrency constraint, legacy complexity, or ticketed TODO. Remove it; the current single literal type is self-explanatory, and future variants should be introduced when implemented.
services/libs/connectors/src/credentials.ts:17 - This comment only restates the current switch scope and a future implementation plan, which is outside the repository's allowed comment cases. The cases themselves make the supported platforms clear.
services/libs/connectors/src/credentials.ts:28 - This speculative secret-manager plan is not a ticketed TODO or another allowed comment case. Remove it, or track the migration with a
TODO(CM-XXX):if it is actionable work.
| const CLAIM_LIMIT = 100 | ||
|
|
||
| export async function dispatcher(): Promise<void> { | ||
| const startedAt = Date.now() |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 71 out of 73 changed files in this pull request and generated no new comments.
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
Suppressed comments (11)
Previously missed (4) — in code that hasn't changed since the last review.
services/libs/connectors/src/connectors/github/graphql/issues.ts:162
- The mapper uses
.find, sofirst: 10supplies the earliest close event. A currently closed issue that was reopened and closed again is therefore emitted with the old closure's timestamp and actor. Requestlast: 1so the activity represents the current closure.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:22 - This converts a boolean GraphQL field into a string, changing the activity attribute contract from the existing boolean representation (
processData.ts:1384-1391). Preserve the boolean so consumers do not receive"true"/"false".
This issue also appears in the following locations of the same file:
- line 44
- line 50
services/libs/connectors/src/connectors/github/mappers/prComment.ts:20
- GitHub returns PR states in uppercase, while the established activity contract normalizes them to lowercase (
processData.ts:443,478,523,559,595,630,739). Without normalization, comment activities diverge from all other PR activity attributes.
services/libs/connectors/src/connectors/github/mappers/reviewThreadComment.ts:24 - GitHub returns PR states in uppercase, while the established review-comment mapping lowercases this field (
processData.ts:739). Preserve that contract to avoid mixed state values in analytics.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:50
- This assigns every comment and reply to the discussion opener, because the comment queries do not fetch an author and this mapper always uses
discussion.author. Existing GitHub ingestion maps each comment/reply to its own author (processStream.ts:1162-1213); add author fields toREPLY_FIELDSand mapcomment.authorhere to avoid corrupting member attribution.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:46 Discussion.isAnsweredonly says that the discussion has an accepted answer; it does not identify this comment as that answer. On an answered discussion this gives every top-level comment the answer bonus andisAnswer: true. Query and map each comment'sisAnswerfield instead.
services/libs/connectors/src/connectors/github/mappers/discussion.ts:42- Replies currently use the discussion ID as their parent, flattening the thread. The existing ingestion contract uses the containing comment ID for reply
sourceParentId(processData.ts:804-814,processStream.ts:1188-1213); pass the parent comment ID into this mapper for replies.
services/libs/connectors/src/connectors/github/syncs/issues.ts:37 - This discards the page cursor while advancing
sinceto the last item processed. If the run budget expires between pages, the next run starts from page one at a timestamp boundary; equal-timestamp issues can then be repeated indefinitely or skipped depending onfilterBy.sincesemantics. Resume withwatermark.cursor, checkpoint the unchanged query-startsinceplus the next cursor on intermediate pages, and only advancesincewhen the walk completes.
services/libs/connectors/src/connectors/github/syncs/issueComments.ts:118 - The partial-run checkpoint drops the issue-page cursor but advances
since. When the budget ends mid-walk, restarting at that timestamp can repeat or skip issues sharing the boundary timestamp, which also loses their comments. Persistwatermark.cursorwith the original query-startsinceuntil all issue pages finish, then commit the newsincewith a null cursor.
services/libs/connectors/src/pool/tokenPool.ts:173 - The positive-budget check and decrement are separate Redis operations. Concurrent sync runs can all observe the same positive
remainingvalue, then each decrement and acquire the token, driving the counter below zero and sending a burst beyond the provider budget. Use an atomic Lua/WATCH operation that decrements only whenremaining > 0, and try the next token when it fails.
services/libs/connectors/src/connectors/github/mappers/member.ts:136 - Organization actors enter this fallback branch, but the GraphQL organization fragment selects
databaseId, notid, so organization-owned fork activities receive an empty identitysourceId. Prefer the selected database ID before falling back to the global ID.
WIP