diff --git a/README.md b/README.md index ba9d0f4..eae2af5 100644 --- a/README.md +++ b/README.md @@ -49,10 +49,10 @@ Requires [Bun](https://bun.sh) at the release named in the `packageManager` fiel ```sh bun install -bun run --filter geo-ip-service dev +bun run --filter api-service dev ``` -`dev` restarts the service when a source file changes. Every service listens on `PORT`, default 3000, writes JSON log lines at `LOG_LEVEL`, default `info`, answers `GET /healthz`, and shuts down cleanly on `SIGTERM` or `SIGINT`. See [packages/service-kit](packages/service-kit/) for each of these. To run two services side by side, give each its own port: +`dev` restarts the service when a source file changes. A service that needs more configuration lists it in its own README, for example [geo-ip-service](apps/geo-ip-service/#running-locally). Every service listens on `PORT`, default 3000, writes JSON log lines at `LOG_LEVEL`, default `info`, answers `GET /healthz`, and shuts down cleanly on `SIGTERM` or `SIGINT`. See [packages/service-kit](packages/service-kit/) for each of these. To run two services side by side, give each its own port: ```sh PORT=3001 bun run --filter api-service dev diff --git a/apps/geo-ip-service/README.md b/apps/geo-ip-service/README.md new file mode 100644 index 0000000..e9d66be --- /dev/null +++ b/apps/geo-ip-service/README.md @@ -0,0 +1,128 @@ +# geo-ip-service + +Resolves IP addresses to location and network data from the MaxMind GeoLite2 City and ASN databases. It is an internal service with no authentication, meant to be reached only by other services, never from the public edge. Services in this repository call it through [geo-ip-client](../../packages/geo-ip-client/). + +## Running locally + +The service reads the databases from a directory. To run it without a MaxMind license, copy the test databases under the names the service expects: + +```sh +mkdir -p /tmp/geoip +cp apps/geo-ip-service/test-data/GeoLite2-City-Test.mmdb /tmp/geoip/GeoLite2-City.mmdb +cp apps/geo-ip-service/test-data/GeoLite2-ASN-Test.mmdb /tmp/geoip/GeoLite2-ASN.mmdb +GEOIP_DATA_DIR=/tmp/geoip bun run --filter geo-ip-service dev + +curl localhost:3000/lookup/89.160.20.112 +curl localhost:3000/lookup -H 'content-type: application/json' -d '{"addresses":["89.160.20.112","2001:480::1"],"h3Resolution":4}' +``` + +The test databases hold only a few made-up networks; [test-data](test-data/) says where to find the list. + +## Configuration + +| Variable | Default | Meaning | +| -------- | ------- | ------- | +| `GEOIP_DATA_DIR` | required | directory holding `GeoLite2-City.mmdb` and `GeoLite2-ASN.mmdb` | +| `MAX_BATCH_SIZE` | `10000` | most addresses accepted in one batch lookup, the same as the client's default batch size | +| `PORT`, `LOG_LEVEL` | | see [service-kit](../../packages/service-kit/#environment) | + +## Databases + +Both editions are loaded at startup. A file must hold the edition its name says, which is checked against the database's own metadata, so a City file in the ASN slot is refused. + +- Without the City edition the service is `down`: `/healthz` answers `503` and lookups answer `503`. +- Without the ASN edition the service is `degraded`: lookups answer, and results have no `asNumber` or `asName`. +- When loading an edition fails and an earlier copy of it is loaded, the earlier copy keeps serving and the failure is reported in `/healthz` and `/metrics`. + +## API + +All bodies are JSON. Errors are `{ "error": "" }`. + +### `POST /lookup` + +Looks up a batch of addresses. + +```json +{ "addresses": ["89.160.20.112", "2001:480::1", "10.0.0.1", "not-an-address"], "h3Resolution": 4 } +``` + +`addresses` is required. `h3Resolution` is optional, an integer from 0 to 15; when given, results with coordinates also carry the H3 cell holding them at that resolution and the cell's center. + +The answer has one result per requested address, in the same order, duplicates included: + +```json +{ + "results": [ + { + "address": "89.160.20.112", + "status": "found", + "location": { + "coordinates": { "latitude": 58.4167, "longitude": 15.6167 }, + "countryCode": "SE", + "countryName": "Sweden", + "regionName": "Östergötland County", + "cityName": "Linköping", + "timeZone": "Europe/Stockholm", + "asNumber": 29518, + "asName": "Bredband2 AB", + "h3": { "resolution": 4, "cell": "8408b3dffffffff", "center": { "latitude": 58.40578421882332, "longitude": 15.778067337833107 } } + } + }, + { "address": "10.0.0.1", "status": "not-found" }, + { "address": "not-an-address", "status": "invalid" } + ] +} +``` + +| Status | Meaning | +| ------ | ------- | +| `found` | at least one edition has a record for the address; `location` holds what the records have | +| `not-found` | a valid IPv4 or IPv6 address that neither edition has a record for | +| `invalid` | not an IPv4 or IPv6 address | + +Every `location` field is optional and present only when the databases have it: + +| Field | Source | +| ----- | ------ | +| `coordinates` | City: latitude and longitude | +| `countryCode`, `countryName` | City: country ISO code and English name | +| `regionName` | City: English name of the first subdivision | +| `cityName` | City: English city name | +| `timeZone` | City: time zone | +| `asNumber`, `asName` | ASN: autonomous system number and organization | +| `h3` | computed from `coordinates` when `h3Resolution` is given | + +An IPv4-mapped IPv6 address such as `::ffff:89.160.20.112` gives the same result as the IPv4 address. + +| HTTP status | When | +| ----------- | ---- | +| `200` | the batch was answered, whatever the status of each address | +| `400` | the body is not JSON, `addresses` is missing or holds something other than strings, or `h3Resolution` is out of range | +| `413` | `addresses` holds more than `MAX_BATCH_SIZE` entries | +| `503` | the City edition is not loaded | + +### `GET /lookup/:address` + +Looks up one address, with `h3Resolution` as an optional query parameter: `GET /lookup/89.160.20.112?h3Resolution=4`. The body is a single result in the shape above. The HTTP status follows the result: `200` for `found`, `404` for `not-found`, `400` for `invalid` or a bad `h3Resolution`, and `503` when the City edition is not loaded. An IPv6 address can be written as is or percent-encoded. + +### `GET /healthz` + +The [service-kit health report](../../packages/service-kit/#health) with one check per edition, holding its `buildDate`, its `loadedAt` time and the `lastError` of its last load, if that failed. + +### `GET /metrics` + +Metrics in the Prometheus text format: + +| Metric | Type | Labels | Meaning | +| ------ | ---- | ------ | ------- | +| `geoip_database_loaded` | gauge | `edition` | `1` when the edition is loaded | +| `geoip_database_load_failed` | gauge | `edition` | `1` when the last attempt to load the edition failed | +| `geoip_database_build_timestamp_seconds` | gauge | `edition` | build time of the loaded edition, for alerting on its age | +| `geoip_database_load_timestamp_seconds` | gauge | `edition` | time the loaded edition was loaded | +| `geoip_lookups_total` | counter | `status` | addresses looked up, by result | +| `geoip_requests_total` | counter | `route`, `code` | lookup requests by route (`batch` or `single`) and HTTP status | +| `geoip_batch_duration_seconds` | summary | | time to answer a batch, from receiving it to the encoded answer | + +## Tests + +The tests run against MaxMind's test databases in [test-data](test-data/), so they need no license key or network access. diff --git a/apps/geo-ip-service/package.json b/apps/geo-ip-service/package.json index b51c9cb..f218ce8 100644 --- a/apps/geo-ip-service/package.json +++ b/apps/geo-ip-service/package.json @@ -7,6 +7,9 @@ "start": "bun src/main.ts" }, "dependencies": { - "@anyone-protocol/service-kit": "workspace:*" + "@anyone-protocol/geo-ip-client": "workspace:*", + "@anyone-protocol/service-kit": "workspace:*", + "h3-js": "4.5.0", + "maxmind": "5.0.7" } } diff --git a/apps/geo-ip-service/src/app.test.ts b/apps/geo-ip-service/src/app.test.ts index b5dd780..6a2b006 100644 --- a/apps/geo-ip-service/src/app.test.ts +++ b/apps/geo-ip-service/src/app.test.ts @@ -1,29 +1,222 @@ -import { afterAll, beforeAll, describe, expect, it } from 'bun:test'; +import { GeoIpClient, type GeoLocation } from '@anyone-protocol/geo-ip-client'; +import { afterEach, describe, expect, it } from 'bun:test'; +import { join } from 'node:path'; import { startServer } from './app'; +import { ASN, CITY, type Edition, GeoDatabases } from './databases'; -let server: ReturnType; +const TEST_DATA = join(import.meta.dir, '..', 'test-data'); +const TEST_FILES: Record = { + [CITY]: join(TEST_DATA, 'GeoLite2-City-Test.mmdb'), + [ASN]: join(TEST_DATA, 'GeoLite2-ASN-Test.mmdb'), +}; -beforeAll(() => { - server = startServer(0); +const LINKOPING: GeoLocation = { + coordinates: { latitude: 58.4167, longitude: 15.6167 }, + countryCode: 'SE', + countryName: 'Sweden', + regionName: 'Östergötland County', + cityName: 'Linköping', + timeZone: 'Europe/Stockholm', + asNumber: 29518, + asName: 'Bredband2 AB', +}; + +const LINKOPING_H3 = { + resolution: 4, + cell: '8408b3dffffffff', + center: { latitude: 58.40578421882332, longitude: 15.778067337833107 }, +}; + +const SAN_DIEGO: GeoLocation = { + coordinates: { latitude: 32.7203, longitude: -117.1552 }, + countryCode: 'US', + countryName: 'United States', + regionName: 'California', + cityName: 'San Diego', + timeZone: 'America/Los_Angeles', +}; + +const TELSTRA: GeoLocation = { asNumber: 1221, asName: 'Telstra Pty Ltd' }; + +let server: ReturnType | undefined; + +async function serve({ editions = [CITY, ASN], maxBatchSize = 10 }: { editions?: Edition[]; maxBatchSize?: number } = {}) { + const databases = new GeoDatabases(); + for (const edition of editions) { + await databases.load(edition, TEST_FILES[edition]); + } + server = startServer({ port: 0, databases, maxBatchSize }); + return server; +} + +function post(body: unknown) { + return fetch(new URL('/lookup', server?.url), { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: typeof body === 'string' ? body : JSON.stringify(body), + }); +} + +function get(path: string) { + return fetch(new URL(path, server?.url)); +} + +afterEach(async () => { + await server?.stop(true); + server = undefined; +}); + +describe('POST /lookup', () => { + it('answers every address in order, reporting unknown and invalid ones per item', async () => { + await serve(); + + const response = await post({ addresses: ['89.160.20.112', '2001:480::1', '1.128.0.1', '10.0.0.1', 'not-an-address', '89.160.20.112'] }); + + expect(response.status).toBe(200); + expect(await response.json()).toEqual({ + results: [ + { address: '89.160.20.112', status: 'found', location: LINKOPING }, + { address: '2001:480::1', status: 'found', location: SAN_DIEGO }, + { address: '1.128.0.1', status: 'found', location: TELSTRA }, + { address: '10.0.0.1', status: 'not-found' }, + { address: 'not-an-address', status: 'invalid' }, + { address: '89.160.20.112', status: 'found', location: LINKOPING }, + ], + }); + }); + + it('adds the H3 cell and its center at the requested resolution to addresses with coordinates', async () => { + await serve(); + + const response = await post({ addresses: ['89.160.20.112', '1.128.0.1'], h3Resolution: 4 }); + + expect(await response.json()).toEqual({ + results: [ + { address: '89.160.20.112', status: 'found', location: { ...LINKOPING, h3: LINKOPING_H3 } }, + { address: '1.128.0.1', status: 'found', location: TELSTRA }, + ], + }); + }); + + it.each([ + ['a body that is not JSON', '{"addresses": [', 400], + ['a body without addresses', { h3Resolution: 4 }, 400], + ['addresses that are not all strings', { addresses: ['89.160.20.112', 7] }, 400], + ['an H3 resolution above 15', { addresses: ['89.160.20.112'], h3Resolution: 16 }, 400], + ['an H3 resolution that is not an integer', { addresses: ['89.160.20.112'], h3Resolution: 4.5 }, 400], + ['more addresses than the batch limit', { addresses: Array.from({ length: 11 }, () => '89.160.20.112') }, 413], + ])('rejects %s', async (_, body, status) => { + await serve(); + + const response = await post(body); + + expect(response.status).toBe(status); + expect(await response.json()).toEqual({ error: expect.any(String) }); + }); }); -afterAll(async () => { - await server.stop(); +describe('GET /lookup/:address', () => { + it.each([ + ['an IPv4 address', '89.160.20.112', 200, { status: 'found', location: LINKOPING }], + ['an IPv6 address', '2001:480::1', 200, { status: 'found', location: SAN_DIEGO }], + ['an encoded IPv6 address', encodeURIComponent('2001:480::1'), 200, { status: 'found', location: SAN_DIEGO }], + ['an address in no database', '10.0.0.1', 404, { status: 'not-found' }], + ['something that is not an address', 'not-an-address', 400, { status: 'invalid' }], + ])('answers %s', async (_, path, status, body) => { + await serve(); + + const response = await get(`/lookup/${path}`); + + expect(response.status).toBe(status); + expect(await response.json()).toEqual({ address: decodeURIComponent(path), ...body }); + }); + + it('adds the H3 cell when a resolution is given', async () => { + await serve(); + + expect(await (await get('/lookup/89.160.20.112?h3Resolution=4')).json()).toMatchObject({ location: { h3: LINKOPING_H3 } }); + expect((await get('/lookup/89.160.20.112?h3Resolution=x')).status).toBe(400); + }); +}); + +describe('geo-ip-client', () => { + it('reads the same results the service answers', async () => { + await serve(); + const client = new GeoIpClient({ baseUrl: server!.url.href, h3Resolution: 4 }); + + const results = await client.lookupMany(['89.160.20.112', '10.0.0.1', 'not-an-address']); + + expect(Object.fromEntries(results)).toEqual({ + '89.160.20.112': { status: 'found', location: { ...LINKOPING, h3: LINKOPING_H3 }, stale: false }, + '10.0.0.1': { status: 'not-found', stale: false }, + 'not-an-address': { status: 'invalid', stale: false }, + }); + }); }); describe('GET /healthz', () => { - it('reports the service as ok', async () => { - const response = await fetch(new URL('/healthz', server.url)); + it('reports the build date and load time of each database', async () => { + await serve(); + + const response = await get('/healthz'); expect(response.status).toBe(200); - expect(await response.json()).toMatchObject({ service: 'geo-ip-service', status: 'ok' }); + expect(await response.json()).toMatchObject({ + service: 'geo-ip-service', + status: 'ok', + checks: { + [CITY]: { status: 'ok', buildDate: '2026-10-14T19:30:47.000Z', loadedAt: expect.any(String), lastError: null }, + [ASN]: { status: 'ok', buildDate: '2026-10-14T19:30:47.000Z', loadedAt: expect.any(String), lastError: null }, + }, + }); + }); + + it('is degraded without the ASN database and still answers lookups', async () => { + await serve({ editions: [CITY] }); + + expect(await (await get('/healthz')).json()).toMatchObject({ status: 'degraded', checks: { [ASN]: { status: 'degraded' } } }); + expect((await get('/lookup/89.160.20.112')).status).toBe(200); + }); + + it('is down without the City database, and lookups are unavailable', async () => { + await serve({ editions: [ASN] }); + + expect((await get('/healthz')).status).toBe(503); + expect((await get('/lookup/89.160.20.112')).status).toBe(503); + expect((await post({ addresses: ['89.160.20.112'] })).status).toBe(503); + }); +}); + +describe('GET /metrics', () => { + it('exposes database ages and lookup counts in the Prometheus text format', async () => { + await serve({ editions: [CITY] }); + await post({ addresses: ['89.160.20.112', '10.0.0.1', 'not-an-address'] }); + await post({ addresses: 'none' }); + await get('/lookup/89.160.20.112'); + + const response = await get('/metrics'); + const text = await response.text(); + + expect(response.headers.get('content-type')).toBe('text/plain; version=0.0.4; charset=utf-8'); + expect(text).toContain('geoip_database_loaded{edition="GeoLite2-City"} 1'); + expect(text).toContain('geoip_database_loaded{edition="GeoLite2-ASN"} 0'); + expect(text).toContain('geoip_database_load_failed{edition="GeoLite2-ASN"} 0'); + expect(text).toContain(`geoip_database_build_timestamp_seconds{edition="GeoLite2-City"} ${Date.parse('2026-10-14T19:30:47Z') / 1000}`); + expect(text).toContain('geoip_lookups_total{status="found"} 2'); + expect(text).toContain('geoip_lookups_total{status="not-found"} 1'); + expect(text).toContain('geoip_lookups_total{status="invalid"} 1'); + expect(text).toContain('geoip_requests_total{route="batch",code="200"} 1'); + expect(text).toContain('geoip_requests_total{route="batch",code="400"} 1'); + expect(text).toContain('geoip_requests_total{route="single",code="200"} 1'); + expect(text).toContain('geoip_batch_duration_seconds_count 1'); + expect(text.endsWith('\n')).toBe(true); }); }); describe('an unknown path', () => { it('is not found', async () => { - const response = await fetch(new URL('/missing', server.url)); + await serve(); - expect(response.status).toBe(404); + expect((await get('/missing')).status).toBe(404); }); }); diff --git a/apps/geo-ip-service/src/app.ts b/apps/geo-ip-service/src/app.ts index ed6d9f5..81fdecc 100644 --- a/apps/geo-ip-service/src/app.ts +++ b/apps/geo-ip-service/src/app.ts @@ -1,12 +1,129 @@ -import { createHealthReporter, healthResponse } from '@anyone-protocol/service-kit'; +import { BATCH_LOOKUP_PATH, type BatchLookupResponse, type LookupResult } from '@anyone-protocol/geo-ip-client'; +import { createHealthReporter, healthResponse, type HealthCheck, type HealthStatus } from '@anyone-protocol/service-kit'; +import { ASN, CITY, EDITIONS, type Edition, type GeoDatabases } from './databases'; +import { isH3Resolution, MAX_H3_RESOLUTION } from './location'; +import { Counter, METRICS_CONTENT_TYPE, metricFamily, Summary } from './metrics'; -export function startServer(port: number) { - const health = createHealthReporter('geo-ip-service'); +export interface ServerOptions { + port: number; + databases: GeoDatabases; + maxBatchSize: number; +} + +interface BatchRequest { + addresses: string[]; + h3Resolution?: number; +} + +const STATUS_WITHOUT_EDITION: Record = { [CITY]: 'down', [ASN]: 'degraded' }; + +const SINGLE_LOOKUP_STATUS: Record = { found: 200, 'not-found': 404, invalid: 400 }; + +const H3_RESOLUTION_ERROR = `h3Resolution must be an integer from 0 to ${MAX_H3_RESOLUTION}`; + +function errorResponse(status: number, error: string): Response { + return Response.json({ error }, { status }); +} + +function parseBatch(body: unknown, maxBatchSize: number): BatchRequest | Response { + const { addresses, h3Resolution } = (body ?? {}) as Record; + if (!Array.isArray(addresses) || !addresses.every((address) => typeof address === 'string')) { + return errorResponse(400, 'body must be a JSON object with an addresses array of strings'); + } + if (h3Resolution !== undefined && !isH3Resolution(h3Resolution)) { + return errorResponse(400, H3_RESOLUTION_ERROR); + } + if (addresses.length > maxBatchSize) { + return errorResponse(413, `batch holds ${addresses.length} addresses, the limit is ${maxBatchSize}`); + } + return { addresses, h3Resolution }; +} + +function parseH3Query(url: string): number | undefined | Response { + const raw = new URL(url).searchParams.get('h3Resolution'); + if (raw === null) return undefined; + const resolution = /^\d+$/.test(raw) ? Number(raw) : Number.NaN; + return isH3Resolution(resolution) ? resolution : errorResponse(400, H3_RESOLUTION_ERROR); +} + +function databaseCheck(databases: GeoDatabases, edition: Edition): HealthCheck { + return () => { + const { loaded, buildDate, loadedAt, lastError } = databases.state(edition); + return { status: loaded ? 'ok' : STATUS_WITHOUT_EDITION[edition], buildDate, loadedAt, lastError }; + }; +} + +function createMetrics(databases: GeoDatabases) { + const requests = new Counter('geoip_requests_total', 'Lookup requests by route and HTTP status code'); + const lookups = new Counter('geoip_lookups_total', 'Addresses looked up, by result'); + const batchDuration = new Summary('geoip_batch_duration_seconds', 'Time to answer a batch lookup'); + + const render = () => { + const states = EDITIONS.map((edition) => databases.state(edition)); + const timestamps = (field: 'buildDate' | 'loadedAt') => + states.flatMap((state) => (state[field] ? [{ labels: { edition: state.edition }, value: Date.parse(state[field]) / 1000 }] : [])); + + return [ + metricFamily('geoip_database_loaded', 'gauge', 'Whether the database edition is loaded', states.map((state) => ({ labels: { edition: state.edition }, value: Number(state.loaded) }))), + metricFamily('geoip_database_load_failed', 'gauge', 'Whether the last attempt to load the database edition failed', states.map((state) => ({ labels: { edition: state.edition }, value: Number(state.lastError !== null) }))), + metricFamily('geoip_database_build_timestamp_seconds', 'gauge', 'Build time of the loaded database edition', timestamps('buildDate')), + metricFamily('geoip_database_load_timestamp_seconds', 'gauge', 'Time the loaded database edition was loaded', timestamps('loadedAt')), + requests.render(), + lookups.render(), + batchDuration.render(), + ].join('\n') + '\n'; + }; + + return { requests, lookups, batchDuration, render }; +} + +export function startServer({ port, databases, maxBatchSize }: ServerOptions) { + const health = createHealthReporter('geo-ip-service', { + checks: Object.fromEntries(EDITIONS.map((edition) => [edition, databaseCheck(databases, edition)])), + }); + const metrics = createMetrics(databases); + + const finish = (route: string, response: Response) => { + metrics.requests.inc({ route, code: String(response.status) }); + return response; + }; + const unavailable = () => errorResponse(503, `${CITY} database is not loaded`); return Bun.serve({ port, routes: { '/healthz': { GET: () => healthResponse(health()) }, + '/metrics': { GET: () => new Response(metrics.render(), { headers: { 'content-type': METRICS_CONTENT_TYPE } }) }, + + [BATCH_LOOKUP_PATH]: { + POST: async (request) => { + const started = performance.now(); + if (!databases.ready) return finish('batch', unavailable()); + + const batch = parseBatch(await request.json().catch(() => undefined), maxBatchSize); + if (batch instanceof Response) return finish('batch', batch); + + const results = batch.addresses.map((address) => ({ address, ...databases.lookup(address, batch.h3Resolution) })); + results.forEach(({ status }) => metrics.lookups.inc({ status })); + const response = Response.json({ results } satisfies BatchLookupResponse); + metrics.batchDuration.observe((performance.now() - started) / 1000); + return finish('batch', response); + }, + }, + + '/lookup/:address': { + GET: (request) => { + if (!databases.ready) return finish('single', unavailable()); + + const h3Resolution = parseH3Query(request.url); + if (h3Resolution instanceof Response) return finish('single', h3Resolution); + + const { address } = request.params; + const result = databases.lookup(address, h3Resolution); + metrics.lookups.inc({ status: result.status }); + return finish('single', Response.json({ address, ...result }, { status: SINGLE_LOOKUP_STATUS[result.status] })); + }, + }, }, fetch: () => new Response('Not Found', { status: 404 }), }); diff --git a/apps/geo-ip-service/src/databases.test.ts b/apps/geo-ip-service/src/databases.test.ts new file mode 100644 index 0000000..0507006 --- /dev/null +++ b/apps/geo-ip-service/src/databases.test.ts @@ -0,0 +1,63 @@ +import { describe, expect, it } from 'bun:test'; +import { join } from 'node:path'; +import { ASN, CITY, databasePath, GeoDatabases } from './databases'; + +const TEST_DATA = join(import.meta.dir, '..', 'test-data'); +const CITY_TEST = join(TEST_DATA, 'GeoLite2-City-Test.mmdb'); +const ASN_TEST = join(TEST_DATA, 'GeoLite2-ASN-Test.mmdb'); + +describe('databasePath', () => { + it('names the file after the edition', () => { + expect(databasePath('/var/lib/geoip', CITY)).toBe('/var/lib/geoip/GeoLite2-City.mmdb'); + }); +}); + +describe('GeoDatabases', () => { + it('reports the build date and load time of a loaded edition', async () => { + const databases = new GeoDatabases(() => Date.UTC(2030, 0, 1)); + + expect(await databases.load(CITY, CITY_TEST)).toEqual({ + edition: CITY, + loaded: true, + buildDate: '2026-10-14T19:30:47.000Z', + loadedAt: '2030-01-01T00:00:00.000Z', + lastError: null, + }); + expect(databases.ready).toBe(true); + }); + + it('is not ready until the City edition is loaded', async () => { + const databases = new GeoDatabases(); + + await databases.load(ASN, ASN_TEST); + + expect(databases.ready).toBe(false); + }); + + it.each([ + ['a missing file', join(TEST_DATA, 'missing.mmdb'), 'ENOENT'], + ['a file of another edition', ASN_TEST, 'holds GeoLite2-ASN, expected GeoLite2-City'], + ['a file that is not a database', import.meta.path, ''], + ])('reports %s as not loaded', async (_, path, error) => { + const state = await new GeoDatabases().load(CITY, path); + + expect(state).toMatchObject({ loaded: false, buildDate: null, lastError: expect.stringContaining(error) }); + }); + + it('keeps answering from the loaded edition when a later load fails', async () => { + const databases = new GeoDatabases(); + await databases.load(CITY, CITY_TEST); + + const state = await databases.load(CITY, ASN_TEST); + + expect(state).toMatchObject({ loaded: true, lastError: expect.stringContaining('expected GeoLite2-City') }); + expect(databases.lookup('89.160.20.112')).toMatchObject({ status: 'found', location: { cityName: 'Linköping' } }); + }); + + it('answers an IPv4-mapped IPv6 address like the IPv4 address', async () => { + const databases = new GeoDatabases(); + await databases.load(CITY, CITY_TEST); + + expect(databases.lookup('::ffff:89.160.20.112')).toEqual(databases.lookup('89.160.20.112')); + }); +}); diff --git a/apps/geo-ip-service/src/databases.ts b/apps/geo-ip-service/src/databases.ts new file mode 100644 index 0000000..526d3d6 --- /dev/null +++ b/apps/geo-ip-service/src/databases.ts @@ -0,0 +1,81 @@ +import type { LookupResult } from '@anyone-protocol/geo-ip-client'; +import { readFile } from 'node:fs/promises'; +import { isIP } from 'node:net'; +import { join } from 'node:path'; +import { Reader, type AsnResponse, type CityResponse, type Response } from 'maxmind'; +import { toLocation } from './location'; + +export const CITY = 'GeoLite2-City'; +export const ASN = 'GeoLite2-ASN'; +export const EDITIONS = [CITY, ASN] as const; + +export type Edition = (typeof EDITIONS)[number]; + +export interface DatabaseState { + edition: Edition; + loaded: boolean; + buildDate: string | null; + loadedAt: string | null; + lastError: string | null; +} + +interface LoadedDatabase { + reader: Reader; + buildDate: Date; + loadedAt: Date; +} + +export function databasePath(directory: string, edition: Edition): string { + return join(directory, `${edition}.mmdb`); +} + +export class GeoDatabases { + readonly #loaded = new Map(); + readonly #errors = new Map(); + readonly #now: () => number; + + constructor(now: () => number = Date.now) { + this.#now = now; + } + + get ready(): boolean { + return this.#loaded.has(CITY); + } + + async load(edition: Edition, path: string): Promise { + try { + const reader = new Reader(await readFile(path)); + if (reader.metadata.databaseType !== edition) { + throw new Error(`${path} holds ${reader.metadata.databaseType}, expected ${edition}`); + } + this.#loaded.set(edition, { reader, buildDate: reader.metadata.buildEpoch, loadedAt: new Date(this.#now()) }); + this.#errors.delete(edition); + } catch (error) { + this.#errors.set(edition, error instanceof Error ? error.message : String(error)); + } + return this.state(edition); + } + + state(edition: Edition): DatabaseState { + const database = this.#loaded.get(edition); + return { + edition, + loaded: database !== undefined, + buildDate: database?.buildDate.toISOString() ?? null, + loadedAt: database?.loadedAt.toISOString() ?? null, + lastError: this.#errors.get(edition) ?? null, + }; + } + + lookup(address: string, h3Resolution?: number): LookupResult { + if (isIP(address) === 0) { + return { status: 'invalid' }; + } + const city = (this.#loaded.get(CITY)?.reader.get(address) ?? null) as CityResponse | null; + const asn = (this.#loaded.get(ASN)?.reader.get(address) ?? null) as AsnResponse | null; + if (!city && !asn) { + return { status: 'not-found' }; + } + return { status: 'found', location: toLocation(city, asn, h3Resolution) }; + } +} diff --git a/apps/geo-ip-service/src/location.ts b/apps/geo-ip-service/src/location.ts new file mode 100644 index 0000000..dae2207 --- /dev/null +++ b/apps/geo-ip-service/src/location.ts @@ -0,0 +1,37 @@ +import type { Coordinates, GeoLocation, H3Cell } from '@anyone-protocol/geo-ip-client'; +import { cellToLatLng, latLngToCell } from 'h3-js'; +import type { AsnResponse, CityResponse } from 'maxmind'; + +export const MAX_H3_RESOLUTION = 15; + +export function isH3Resolution(value: unknown): value is number { + return Number.isInteger(value) && (value as number) >= 0 && (value as number) <= MAX_H3_RESOLUTION; +} + +function coordinatesOf(location: CityResponse['location']): Coordinates | undefined { + const latitude = location?.latitude; + const longitude = location?.longitude; + return typeof latitude === 'number' && typeof longitude === 'number' ? { latitude, longitude } : undefined; +} + +function h3CellOf({ latitude, longitude }: Coordinates, resolution: number): H3Cell { + const cell = latLngToCell(latitude, longitude, resolution); + const [centerLatitude, centerLongitude] = cellToLatLng(cell); + return { resolution, cell, center: { latitude: centerLatitude, longitude: centerLongitude } }; +} + +export function toLocation(city: CityResponse | null, asn: AsnResponse | null, h3Resolution?: number): GeoLocation { + const coordinates = coordinatesOf(city?.location); + const location: GeoLocation = { + coordinates, + countryCode: city?.country?.iso_code, + countryName: city?.country?.names.en, + regionName: city?.subdivisions?.[0]?.names.en, + cityName: city?.city?.names.en, + timeZone: city?.location?.time_zone, + asNumber: asn?.autonomous_system_number, + asName: asn?.autonomous_system_organization, + h3: coordinates && h3Resolution !== undefined ? h3CellOf(coordinates, h3Resolution) : undefined, + }; + return Object.fromEntries(Object.entries(location).filter(([, value]) => value !== undefined)); +} diff --git a/apps/geo-ip-service/src/main.ts b/apps/geo-ip-service/src/main.ts index a1dae9d..59047c1 100644 --- a/apps/geo-ip-service/src/main.ts +++ b/apps/geo-ip-service/src/main.ts @@ -1,12 +1,28 @@ -import { commonEnv, createLogger, handleShutdownSignals, handleUncaughtErrors, parseEnv } from '@anyone-protocol/service-kit'; +import { GEO_IP_CLIENT_DEFAULTS } from '@anyone-protocol/geo-ip-client'; +import { commonEnv, createLogger, env, handleShutdownSignals, handleUncaughtErrors, parseEnv } from '@anyone-protocol/service-kit'; import { startServer } from './app'; +import { databasePath, EDITIONS, GeoDatabases } from './databases'; -const config = parseEnv(commonEnv); +const config = parseEnv({ + ...commonEnv, + GEOIP_DATA_DIR: env.string(), + MAX_BATCH_SIZE: env.integer({ default: GEO_IP_CLIENT_DEFAULTS.maxBatchSize }), +}); const logger = createLogger({ service: 'geo-ip-service', level: config.LOG_LEVEL }); handleUncaughtErrors(logger); -const server = startServer(config.PORT); +const databases = new GeoDatabases(); +for (const edition of EDITIONS) { + const state = await databases.load(edition, databasePath(config.GEOIP_DATA_DIR, edition)); + if (state.loaded) { + logger.info('Database loaded', { ...state }); + } else { + logger.error('Database not loaded', { ...state }); + } +} -logger.info('Listening', { port: server.port }); +const server = startServer({ port: config.PORT, databases, maxBatchSize: config.MAX_BATCH_SIZE }); + +logger.info('Listening', { port: server.port, maxBatchSize: config.MAX_BATCH_SIZE }); handleShutdownSignals([['http server', () => server.stop()]], { logger }); diff --git a/apps/geo-ip-service/src/metrics.ts b/apps/geo-ip-service/src/metrics.ts new file mode 100644 index 0000000..a580a78 --- /dev/null +++ b/apps/geo-ip-service/src/metrics.ts @@ -0,0 +1,60 @@ +export type Labels = Record; + +export interface Sample { + labels?: Labels; + value: number; +} + +export const METRICS_CONTENT_TYPE = 'text/plain; version=0.0.4; charset=utf-8'; + +function escapeLabelValue(value: string): string { + return value.replaceAll('\\', '\\\\').replaceAll('"', '\\"').replaceAll('\n', '\\n'); +} + +function labelSet(labels: Labels = {}): string { + const pairs = Object.entries(labels).map(([name, value]) => `${name}="${escapeLabelValue(value)}"`); + return pairs.length > 0 ? `{${pairs.join(',')}}` : ''; +} + +export function metricFamily(name: string, type: 'counter' | 'gauge', help: string, samples: Sample[]): string { + return [`# HELP ${name} ${help}`, `# TYPE ${name} ${type}`, ...samples.map((sample) => `${name}${labelSet(sample.labels)} ${sample.value}`)].join('\n'); +} + +export class Counter { + readonly #samples = new Map(); + + constructor( + readonly name: string, + readonly help: string, + ) {} + + inc(labels: Labels = {}, amount = 1): void { + const key = labelSet(labels); + const sample = this.#samples.get(key) ?? { labels, value: 0 }; + sample.value += amount; + this.#samples.set(key, sample); + } + + render(): string { + return metricFamily(this.name, 'counter', this.help, [...this.#samples.values()]); + } +} + +export class Summary { + #sum = 0; + #count = 0; + + constructor( + readonly name: string, + readonly help: string, + ) {} + + observe(value: number): void { + this.#sum += value; + this.#count += 1; + } + + render(): string { + return [`# HELP ${this.name} ${this.help}`, `# TYPE ${this.name} summary`, `${this.name}_sum ${this.#sum}`, `${this.name}_count ${this.#count}`].join('\n'); + } +} diff --git a/apps/geo-ip-service/test-data/GeoLite2-ASN-Test.mmdb b/apps/geo-ip-service/test-data/GeoLite2-ASN-Test.mmdb new file mode 100644 index 0000000..42a3e87 Binary files /dev/null and b/apps/geo-ip-service/test-data/GeoLite2-ASN-Test.mmdb differ diff --git a/apps/geo-ip-service/test-data/GeoLite2-City-Test.mmdb b/apps/geo-ip-service/test-data/GeoLite2-City-Test.mmdb new file mode 100644 index 0000000..03263a4 Binary files /dev/null and b/apps/geo-ip-service/test-data/GeoLite2-City-Test.mmdb differ diff --git a/apps/geo-ip-service/test-data/LICENSE-MIT b/apps/geo-ip-service/test-data/LICENSE-MIT new file mode 100644 index 0000000..f85e365 --- /dev/null +++ b/apps/geo-ip-service/test-data/LICENSE-MIT @@ -0,0 +1,17 @@ +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/apps/geo-ip-service/test-data/README.md b/apps/geo-ip-service/test-data/README.md new file mode 100644 index 0000000..10aa19b --- /dev/null +++ b/apps/geo-ip-service/test-data/README.md @@ -0,0 +1,16 @@ +# Test databases + +`GeoLite2-City-Test.mmdb` and `GeoLite2-ASN-Test.mmdb` are MaxMind's test databases, taken from the `test-data` directory of [maxmind/MaxMind-DB](https://github.com/maxmind/MaxMind-DB) at commit [`0ee49278e35fd7bdf341ac236edbba20de1aceb9`](https://github.com/maxmind/MaxMind-DB/tree/0ee49278e35fd7bdf341ac236edbba20de1aceb9/test-data). They hold a small set of made-up records. The JSON they were built from, which lists every network in them, is in the [`source-data`](https://github.com/maxmind/MaxMind-DB/tree/0ee49278e35fd7bdf341ac236edbba20de1aceb9/source-data) directory at the same commit. + +The tests use these addresses: + +| Address | City edition | ASN edition | +| ------- | ------------ | ----------- | +| `89.160.20.112` | Linköping, Sweden | 29518, Bredband2 AB | +| `2001:480::1` | San Diego, United States | none | +| `1.128.0.1` | none | 1221, Telstra Pty Ltd | +| `10.0.0.1` | none | none | + +To update the files, download both from a newer commit, replace them here, update the commit above, and adjust the expected values in the tests if the records changed. + +Copyright (c) 2013 - 2026 by MaxMind, Inc. The databases are licensed under the [Apache License, Version 2.0](https://github.com/maxmind/MaxMind-DB/blob/0ee49278e35fd7bdf341ac236edbba20de1aceb9/LICENSE-APACHE) or the MIT License, at your option. They are used here under the MIT License, whose text is in [LICENSE-MIT](LICENSE-MIT). diff --git a/bun.lock b/bun.lock index 8b14f03..cc6ad67 100644 --- a/bun.lock +++ b/bun.lock @@ -18,7 +18,10 @@ "apps/geo-ip-service": { "name": "geo-ip-service", "dependencies": { + "@anyone-protocol/geo-ip-client": "workspace:*", "@anyone-protocol/service-kit": "workspace:*", + "h3-js": "4.5.0", + "maxmind": "5.0.7", }, }, "packages/geo-ip-client": { @@ -83,6 +86,14 @@ "geo-ip-service": ["geo-ip-service@workspace:apps/geo-ip-service"], + "h3-js": ["h3-js@4.5.0", "", {}, "sha512-uKmdPc+DuarnR+XqZxEuWOMw7KzzKROrx3MLeJpFnMOs78S9M5eZ+X5RieS9UcSFQqbeXzbfWuX/W9Pyajinqw=="], + + "maxmind": ["maxmind@5.0.7", "", { "dependencies": { "mmdb-lib": "3.0.3", "tiny-lru": "13.0.0" } }, "sha512-+w637dwfv01MKjkrp4sKDBTEKHLPvWLYb647QTjiz3wG/teSemqudIKNShaS6eqZ7ffxC9oZlQQgIqY0rGojog=="], + + "mmdb-lib": ["mmdb-lib@3.0.3", "", {}, "sha512-xQPoBXcNjjHiOvOraFBKtA++uNWF6aCVHL9dRKFXEov8eI3QJwtgiw3qApsonFT5SpoqsEVISUTg3HIDs2DiXw=="], + + "tiny-lru": ["tiny-lru@13.0.0", "", {}, "sha512-xDHxKKS1FdF0Tv2P+QT7IeSEg74K/8cEDzbv3Tv6UyHHUgBOjOiQiBp818MGj66dhurQus/IBcoAbwIKtSGc6Q=="], + "typescript": ["typescript@7.0.2", "", { "optionalDependencies": { "@typescript/typescript-aix-ppc64": "7.0.2", "@typescript/typescript-darwin-arm64": "7.0.2", "@typescript/typescript-darwin-x64": "7.0.2", "@typescript/typescript-freebsd-arm64": "7.0.2", "@typescript/typescript-freebsd-x64": "7.0.2", "@typescript/typescript-linux-arm": "7.0.2", "@typescript/typescript-linux-arm64": "7.0.2", "@typescript/typescript-linux-loong64": "7.0.2", "@typescript/typescript-linux-mips64el": "7.0.2", "@typescript/typescript-linux-ppc64": "7.0.2", "@typescript/typescript-linux-riscv64": "7.0.2", "@typescript/typescript-linux-s390x": "7.0.2", "@typescript/typescript-linux-x64": "7.0.2", "@typescript/typescript-netbsd-arm64": "7.0.2", "@typescript/typescript-netbsd-x64": "7.0.2", "@typescript/typescript-openbsd-arm64": "7.0.2", "@typescript/typescript-openbsd-x64": "7.0.2", "@typescript/typescript-sunos-x64": "7.0.2", "@typescript/typescript-win32-arm64": "7.0.2", "@typescript/typescript-win32-x64": "7.0.2" }, "bin": { "tsc": "bin/tsc" } }, "sha512-8FYau96o3NKOhbjKi/qNvG/W5jhzxkbdm5sj9AbZ/5T5sWqn3hJgLfGx27sRKZWTvyzCP8dLRBTf5tBTSRVUNA=="], "undici-types": ["undici-types@8.9.0", "", {}, "sha512-KTDyRTYX8sWmKXAikPHHSyc63CRPETMctyjKFupcC6OBLXT3xsN0e9aF7m+mIXutFWpUXuedtowG7iLOzp0kQg=="], diff --git a/packages/geo-ip-client/README.md b/packages/geo-ip-client/README.md index a043ec2..a07b50c 100644 --- a/packages/geo-ip-client/README.md +++ b/packages/geo-ip-client/README.md @@ -28,16 +28,7 @@ for (const [address, result] of results) { `stale` is `true` when the result comes from the cache past its TTL because the service could not be reached. -`location` holds whatever the database has for the address. Every field is optional: - -| Field | Example | -| ----- | ------- | -| `coordinates` | `{ "latitude": -33.86, "longitude": 151.2 }` | -| `countryCode`, `countryName` | `"AU"`, `"Australia"` | -| `regionName`, `cityName` | `"New South Wales"`, `"Sydney"` | -| `timeZone` | `"Australia/Sydney"` | -| `asNumber`, `asName` | `13335`, `"CLOUDFLARENET"` | -| `h3` | `{ "resolution": 4, "cell": "84be0e3ffffffff", "center": { "latitude": -33.7411, "longitude": 151.2858 } }`, present when the client was created with `h3Resolution` and the address has coordinates | +`location` holds whatever the databases have for the address; its fields are listed in the [geo-ip-service API](../../apps/geo-ip-service/#post-lookup). ## Options @@ -85,22 +76,4 @@ Cached results are never dropped for age, only when the cache is over `maxCacheE ## Protocol -The client sends every lookup as `POST /lookup` with a JSON body: - -```json -{ "addresses": ["1.1.1.1", "2001:db8::1", "not-an-address"], "h3Resolution": 4 } -``` - -and expects `200` with one result per address, each the result object above plus its `address`: - -```json -{ - "results": [ - { "address": "1.1.1.1", "status": "found", "location": { "countryCode": "AU" } }, - { "address": "2001:db8::1", "status": "not-found" }, - { "address": "not-an-address", "status": "invalid" } - ] -} -``` - -The request and response types are exported from this package as `BatchLookupRequest` and `BatchLookupResponse`, and the path as `BATCH_LOOKUP_PATH`. +The client sends every lookup as a batch to `POST /lookup`, described in the [geo-ip-service API](../../apps/geo-ip-service/#api). The request and response types are exported from this package as `BatchLookupRequest` and `BatchLookupResponse`, and the path as `BATCH_LOOKUP_PATH`; geo-ip-service builds its answers from the same types.