Skip to content

Mutiple small improvements on export get #607

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
guillermau wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Mutiple small improvements on export get #607

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
493c3cd
feat: add possibility to add agent in source config
guillermau Oct 8, 2026
c495b7e
fix: doesn't handle case similar software
guillermau Oct 12, 2026
a9f9011
feat: improve zenodo calls
guillermau Oct 12, 2026
0b0f941
feat: handle running out of call credit on github api
guillermau Oct 12, 2026
5c6856f
feat: import allow file of ids
guillermau Oct 12, 2026
005a3ec
feat: add a default version inside user agent header
guillermau Oct 12, 2026
46697be
fix: proper id in zenodo identifier
guillermau Oct 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
282 changes: 133 additions & 149 deletions api/src/core/adapters/GitHub/api/repo.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
// SPDX-License-Identifier: MIT

import { Octokit } from "@octokit/rest";
import { Endpoints } from "@octokit/types";
import { Endpoints, RequestParameters } from "@octokit/types";

export namespace GitHubAPI {
export type Commit = Endpoints["GET /repos/{owner}/{repo}/commits/{ref}"]["response"]["data"];
Expand All @@ -16,6 +16,9 @@ export namespace GitHubAPI {
export type SearchRepositoriesItem = Endpoints["GET /search/repositories"]["response"]["data"]["items"][0];
}

const GITHUB_API_API_TIMEOUT = 60 * 1000;
const GITHUB_API_LOG = "[GitHubAdapter]";

const parseURL = (repoUrl: string | URL): { repo: string; owner: string } => {
let repoUrlObj = typeof repoUrl === "string" ? URL.parse(repoUrl) : repoUrl;
if (!repoUrlObj) throw Error("The provided URL could be parsed");
Expand All @@ -35,205 +38,186 @@ const parseURL = (repoUrl: string | URL): { repo: string; owner: string } => {
export const gitHubEndpointMaker = (params?: { auth?: string }) => {
const octokit = new Octokit(params?.auth ? { auth: params.auth } : {});

const apiCall = async (url: string, options: RequestParameters) => {
try {
const resultApi = await octokit.request(url, options);

if (resultApi.status === 429 || resultApi.status > 500) {
console.debug(`${GITHUB_API_LOG} Got ${resultApi.status} let's take a break for 60s`);
await new Promise(resolve => setTimeout(resolve, GITHUB_API_API_TIMEOUT));
return apiCall(url, options);
}

const remainingRequests = parseInt(resultApi.headers["x-ratelimit-remaining"] || "0");
if (remainingRequests < 50) {
const now = new Date().valueOf();
const resetTime = parseInt(resultApi.headers["x-ratelimit-reset"] || "0") * 1000; // Convert to ms
const timeToWait = resetTime - now;
console.debug(
`${GITHUB_API_LOG} Got ${resultApi.status} let's take a break for ${Math.round(timeToWait / 1000)}s`
);
await new Promise(resolve => setTimeout(resolve, timeToWait));
return apiCall(url, options);
}

return resultApi.data;
} catch (error) {
console.error(error);
return undefined;
}
};

return {
repo: {
issues: {
getLastClosedIssue: async (params: { repoUrl: string | URL }) => {
try {
const { owner, repo } = parseURL(params.repoUrl);
const resIssues = await octokit.request("GET /repos/{owner}/{repo}/issues", {
owner,
repo,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc",
state: "closed"
});

return resIssues.data[0];
} catch (error) {
return undefined;
}
}
},
commits: {
getBySha: async (params: {
repoUrl: string | URL;
commit_sha: string;
}): Promise<GitHubAPI.Commit | undefined> => {
try {
const { owner, repo } = parseURL(params.repoUrl);
const resCommit = await octokit.request("GET /repos/{owner}/{repo}/commits/{ref}", {
owner,
repo,
ref: params.commit_sha,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc"
});
return resCommit.data;
} catch (error) {
return undefined;
}
},
getLastCommit: async (params: { repoUrl: string | URL }): Promise<GitHubAPI.Commit | undefined> => {
try {
const { owner, repo } = parseURL(params.repoUrl);
const resCommit = await octokit.request("GET /repos/{owner}/{repo}/commits", {
owner,
repo,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc"
});
return resCommit.data[0];
} catch (error) {
return undefined;
}
}
},
mergeRequests: {
getLast: async (params: { repoUrl: string | URL }) => {
try {
const { owner, repo } = parseURL(params.repoUrl);
const resPull = await octokit.request("GET /repos/{owner}/{repo}/pulls", {
owner,
repo,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc",
state: "closed"
});

return resPull.data[0];
} catch (error) {
return undefined;
}
}
},
get: async (params: { repoUrl: string | URL }): Promise<GitHubAPI.Repo | undefined> => {
try {
const { owner, repo } = parseURL(params.repoUrl);
const resPull = await octokit.request("GET /repos/{owner}/{repo}", {
const url = "GET /repos/{owner}/{repo}/issues";
return apiCall(url, {
owner,
repo,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
}
},
direction: "desc",
state: "closed"
});

return resPull.data;
} catch (error) {
return undefined;
}
},
getContributors: async (params: { repoUrl: string | URL }): Promise<GitHubAPI.Contributors | undefined> => {
try {
commits: {
getBySha: async (params: {
repoUrl: string | URL;
commit_sha: string;
}): Promise<GitHubAPI.Commit | undefined> => {
const { owner, repo } = parseURL(params.repoUrl);
const resPull = await octokit.request("GET /repos/{owner}/{repo}/contributors", {
const url = "GET /repos/{owner}/{repo}/commits/{ref}";
const data = await apiCall(url, {
owner,
repo,
ref: params.commit_sha,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc"
});

return resPull.data;
} catch (error) {
return undefined;
}
},
getLanguages: async (params: { repoUrl: string | URL }): Promise<GitHubAPI.Languages | undefined> => {
try {
return data[0];
},
getLastCommit: async (params: { repoUrl: string | URL }): Promise<GitHubAPI.Commit | undefined> => {
const { owner, repo } = parseURL(params.repoUrl);
const resPull = await octokit.request("GET /repos/{owner}/{repo}/languages", {
const url = "GET /repos/{owner}/{repo}/commits";
const data = await apiCall(url, {
owner,
repo,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc"
});

return resPull.data;
} catch (error) {
return undefined;
return data[0];
}
},
getTags: async (params: { repoUrl: string | URL }): Promise<GitHubAPI.Tags | undefined> => {
try {
mergeRequests: {
getLast: async (params: { repoUrl: string | URL }) => {
const { owner, repo } = parseURL(params.repoUrl);
const resPull = await octokit.request("GET /repos/{owner}/{repo}/tags", {
const url = "GET /repos/{owner}/{repo}/pulls";
const data = await apiCall(url, {
owner,
repo,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc"
direction: "desc",
state: "closed"
});

return resPull.data;
} catch (error) {
return undefined;
return data[0];
}
},
get: async (params: { repoUrl: string | URL }): Promise<GitHubAPI.Repo | undefined> => {
const { owner, repo } = parseURL(params.repoUrl);
const url = "GET /repos/{owner}/{repo}";
return apiCall(url, {
owner,
repo,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
}
});
},
getContributors: async (params: { repoUrl: string | URL }): Promise<GitHubAPI.Contributors | undefined> => {
const { owner, repo } = parseURL(params.repoUrl);
const url = "GET /repos/{owner}/{repo}/contributors";
return apiCall(url, {
owner,
repo,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc"
});
},
getLanguages: async (params: { repoUrl: string | URL }): Promise<GitHubAPI.Languages | undefined> => {
const { owner, repo } = parseURL(params.repoUrl);
const url = "GET /repos/{owner}/{repo}/languages";
return apiCall(url, {
owner,
repo,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc"
});
},
getTags: async (params: { repoUrl: string | URL }): Promise<GitHubAPI.Tags | undefined> => {
const { owner, repo } = parseURL(params.repoUrl);
const url = "GET /repos/{owner}/{repo}/tags";
return apiCall(url, {
owner,
repo,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc"
});
}
},
users: {
getById: async (account_id: number): Promise<GitHubAPI.User | undefined> => {
try {
const resPull = await octokit.request("GET /user/{account_id}", {
account_id,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc",
state: "closed"
});

return resPull.data;
} catch (error) {
return undefined;
}
const url = "GET /user/{account_id}";
return apiCall(url, {
account_id,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc",
state: "closed"
});
},
getByUsername: async (username: string): Promise<GitHubAPI.User | undefined> => {
try {
const resPull = await octokit.request("GET /users/{username}", {
username,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc",
state: "closed"
});

return resPull.data;
} catch (error) {
return undefined;
}
const url = "GET /user/{username}";
return apiCall(url, {
username,
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc",
state: "closed"
});
}
},
search: {
repo: {
searchByName: async (name: string): Promise<GitHubAPI.SearchRepositories | undefined> => {
try {
const resPull = await octokit.request(`GET /search/repositories?q=${name} in:name`, {
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc"
});
const url = `GET /search/repositories?q=${name} in:name`;
const data = await apiCall(url, {
headers: {
"X-GitHub-Api-Version": "2022-11-28"
},
direction: "desc"
});

if (resPull.data.total_count === 0) return undefined;
if (data.total_count === 0) return undefined;

return resPull.data;
} catch (error) {
return undefined;
}
return data;
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion api/src/core/adapters/GitHub/getExternalData.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ export const getGitHubSoftwareExternalData: GetSoftwareExternal = memoize(
const versionCommitSha = repoTags?.[0]?.commit?.sha;
const lastVersionCommit = versionCommitSha
? await gitHubApi.repo.commits.getBySha({
repoUrl: externalId,
repoUrl,
commit_sha: versionCommitSha
})
: undefined;
Expand Down
1 change: 1 addition & 0 deletions api/src/core/adapters/dbApi/kysely/createPgSoftwareRepository.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -644,6 +644,7 @@ export const createPgSoftwareRepository = (db: Kysely<Database>): SoftwareReposi
externalId
})
)
.onConflict(oc => oc.columns(["sourceSlug", "externalId"]).doUpdateSet({ softwareId }))
.execute();
} else {
await trx
Expand Down
6 changes: 6 additions & 0 deletions api/src/core/adapters/dbApi/kysely/kysely.database.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,10 +208,16 @@ type SimilarExternalSoftwareExternalDataTable = {
sourceSlug: string;
};

export type BotAgent = {
"User-Agent": string;
from?: string;
};

export type SourceConfig = {
queryTimeout?: number;
auth?: string;
rateLimitRetryDuration?: number;
agent?: BotAgent;
};

type SourcesTable = {
Expand Down
4 changes: 2 additions & 2 deletions api/src/core/adapters/doiResolver.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,8 @@ export const resolveDOIIdentifier = async (doiIdentifier: SchemaIdentifier): Pro

if (record) {
return identifersUtils.makeZenodoIdentifer({
zenodoId: record.id.toString(),
url: urlValue.data.value
zenodoId: record.conceptrecid.toString(),
url: record.links.parent
});
}
}
Expand Down
Loading
Loading