PaymentLab is a local synthetic payment-testing platform for exercising payment lifecycles, concurrency, idempotent recovery, signed asynchronous webhooks, durable workers, deterministic failures, and chaos scenarios without processing real funds or payment credentials. Its PostgreSQL transaction boundaries, lease fencing, post-commit response-loss simulation, tenant isolation, SDK, and observability tools make failure behavior reproducible and inspectable.
PaymentLab 1.0.0 adds a reproducible benchmark harness, local diagnostics, package validation, and CI definitions to the completed platform. It remains a test system: it is not a payment processor, a PCI product, or an integration with financial rails.
Start with the Windows PowerShell quickstart and architecture. Detailed references cover the merchant API and SDK, chaos scenarios, observability and dashboard, and benchmark methodology and measured v1.0 results. See CHANGELOG.md for the v1.0 capability summary. The Milestone 10 completion report records final validation, packaging, CI design, and acceptance evidence.
PaymentLab never processes real funds or real payment credentials. Only the four predefined synthetic tokens below are accepted. Never supply real PANs, CVVs, bank-account details, or personally identifiable financial information. There are no fields for these values, real-card validation, or integrations with banks or payment providers. Unknown tokens are rejected without echoing them in application errors. Database credentials configure infrastructure only; they are not payment credentials. All money-like movements are fictional.
- Positive integer minor-unit amounts, USD only:
1999means $19.99. - Authorization, full capture, cancellation, and full refund with enforced states.
- Four reproducible synthetic-card outcomes.
- Immutable domain snapshots and ordered event histories.
- PostgreSQL storage using SQLAlchemy 2.x, psycopg 3, and Alembic.
- Atomic payment/event/settlement writes and durable IDs across process restarts.
- Optional idempotency keys with durable original-result replay and conflict detection.
- PostgreSQL locks protecting simultaneous operations across threads and processes.
- Durable, versioned webhook events created atomically with payment transitions.
- HMAC-SHA256 delivery to registered localhost endpoints, with durable attempts.
- Explicit webhook retry, event replay, and a small signature-verifying receiver.
- Durable, validated failure profiles with payment/webhook scope and audit records.
- Pre-commit errors, post-commit response loss, bounded latency, synthetic webhook failures, and duplicate delivery without random behavior.
- Durable webhook jobs enqueued atomically with payment/outbox state.
- Multi-worker
SKIP LOCKEDclaiming, lease recovery, retries, dead letters, and manual requeue with at-least-once execution semantics. - An in-memory repository for deterministic tests and ephemeral sessions.
- A CLI shell with payment, settlement, webhook, failure-profile, and audit inspection.
- Strict scenario validation, constrained interpolation, expected errors, explicit retries, bounded concurrency, built-in assertions, and durable redacted reports.
- Structured text/JSON logs, context-local correlation, durable normalized API observations, bounded SQL metrics, Prometheus text, and a bundled local dashboard.
- Isolated reproducible benchmark workloads, a non-destructive
doctorcommand, distribution build verification, and local GitHub Actions definitions.
There are no partial captures/refunds or external queue services. This is a synthetic settlement ledger, not a full banking or double-entry accounting ledger.
src/paymentlab/
models.py Frozen Payment / PaymentEvent dataclasses and states
processor.py State machine, injected clock, repository calls
cards.py / errors.py Synthetic tokens and domain errors
clock.py Clock protocol, SystemClock, LogicalClock
idempotency.py Canonical requests, fingerprints, operation results
ledger.py Immutable entries and shared settlement mapping
failures.py Profiles, validation, matching hooks, presets, audit model
jobs.py Durable job model, states, limits, and store protocol
worker.py Lease-based worker and retry/dead-letter policy
repository.py Repository protocol and in-memory implementation
webhooks.py Models, canonical payload, URL checks, HMAC helper
webhook_delivery.py Endpoint management and post-commit HTTP delivery
persistence/
database.py PostgreSQL URL validation and engine factory
models.py Separate SQLAlchemy records and schema metadata
postgres.py PostgreSQL repository, outbox, transaction boundaries
results.py Versioned immutable payment response snapshots
scenario_models.py Durable scenario runs and incremental step records
observation_models.py Durable normalized API request observations
scenarios.py Strict YAML model, runner, assertions, redaction
scenario_cli.py Validate/run/list/show scenario commands
observability.py Correlation, logging, request observation service
operations.py Bounded PostgreSQL operational aggregates and metrics
dashboard.py Read-only HTML/JSON/Prometheus operator application
redaction.py Shared recursive credential redaction
benchmark.py Bounded benchmark math, isolation, and workloads
benchmark_cli.py Human-readable and JSON benchmark command
diagnostics.py Non-destructive package/database/migration checks
cli.py Persistent or ephemeral sessions
migrations/ Alembic revisions 0001 through 0008
examples/webhook_receiver.py Local signature-verifying development receiver
tests/ Unit tests and isolated PostgreSQL integration tests
Domain models do not inherit SQLAlchemy models; no sessions or SQL enter the
processor. Existing imports from paymentlab and lifecycle method signatures
remain available. The repository retains save, get, update, and list,
adding next_id, ledger, net_settled, and an atomic operation context.
list returns an immutable tuple in
insertion order. Unknown IDs raise PaymentNotFoundError; duplicate saves raise
DuplicatePaymentError.
Repository writes are trusted internal operations; use the processor for lifecycle changes. Updates now require exactly one appended event and unchanged payment terms and cannot replace or truncate history. Both stores use the same settlement mapping. Earlier payment and ledger snapshots remain unchanged.
| Table | Contents and constraints |
|---|---|
payments |
Unique text ID, positive BIGINT amount, USD currency, valid status, predefined synthetic token, timezone-aware creation time. |
payment_events |
BIGINT identity event ID, payment foreign key, positive sequence number, event type, resulting status, timezone-aware timestamp. Unique (payment_id, sequence_number); event type must match status. |
settlement_ledger |
Unique entry ID, payment foreign key, source event reference, CAPTURE or REFUND, signed BIGINT amount, USD, timezone-aware timestamp. Unique (payment_id, entry_type) for full capture/refund; sign must match type. |
idempotency_records |
Primary-key request key, operation, SHA-256 fingerprint, payment/source-event foreign key, resulting status, timestamp, versioned JSONB result snapshot, and optional recorded insufficient-funds error. Only completed results are inserted. |
webhook_endpoints |
Unique readable ID, localhost URL, synthetic signing secret, enabled flag, creation time, and nullable removal time. Removal is soft so old deliveries retain their endpoint reference. |
webhook_events |
Unique readable ID, payment/event foreign key, event type, immutable versioned JSONB payload, and creation time. Unique (payment_id, event_sequence) enforces one webhook event per committed domain event. |
webhook_deliveries |
Unique readable ID, webhook-event and endpoint foreign keys, positive attempt number, PENDING/SUCCEEDED/FAILED status, HTTP result or error, response excerpt, and attempt time. Unique (webhook_event_id, endpoint_id, attempt_number). |
failure_profiles |
Durable profile ID/name, enabled/exhausted state, target/type, validated JSONB options, ONCE/ALWAYS mode, optional scope, priority, and lifecycle timestamps. |
failure_injections |
Append-only application audit of the selected profile, target, operation/subject IDs, failure type, phase, copied options, and injection time. |
jobs |
Stable job ID/type, webhook event/endpoint and delivery references, lifecycle status, priority, attempt budget, availability, lease owner/generation/expiry, timestamps, and last error. |
merchants |
Synthetic merchant ID, display name, active flag, and timestamps; includes the legacy merch_local owner. |
api_keys |
Merchant reference, safe prefix, SHA-256 digest, name, active flag, and creation/revocation times; no plaintext keys. |
api_rate_limits |
One reusable fixed-window admission counter per API key. |
Payment status and creation time have indexes. Primary keys and composite unique constraints supply payment, event, and ledger lookup indexes without redundant single-column indexes. Events reload in sequence-number order, even with equal timestamps. Ledger entries reference their source events and return in event order.
Each processor operation uses one SQLAlchemy Session.begin() transaction, covering
the state read/check as well as writes. Repository calls inside it reuse the same
session through a context variable, so threads sharing a repository do not share
sessions. Standalone repository writes also have their own transaction. Creation
inserts payment and history together. Updates write status, append the domain event,
settlement entry, webhook event, delivery intents, queue jobs, and any idempotency
result before committing. An exception rolls back the complete operation.
Insufficient funds commits the decline before raising its domain error. Invalid
transitions and processor errors perform no write. Storage failures surface as
sanitized PersistenceError messages.
Consider a capture that commits successfully but whose response is lost. The client retries with the same key. PaymentLab returns the original result and adds no new event or ledger entry. Idempotency provides effectively-once state changes for retried requests. Milestone 5 can inject this precise post-commit response-loss condition. This is not a claim of exactly-once network delivery, and PaymentLab does not expose a network API.
All five state-changing methods accept an optional keyword-only idempotency_key.
The CLI accepts --idempotency-key KEY on create, authorize, capture, cancel,
and refund. Keys are case-sensitive and scoped by merchant, not by operation.
Use 1-128 ASCII letters/digits or ., _, :, -,
starting with a letter or digit. Use opaque synthetic request identifiers, never
sensitive personal or credential data. Keys are retained indefinitely; there is
no expiry or cleanup policy in this milestone.
A canonical JSON request includes a format version, operation, target payment ID,
and meaningful create parameters (integer amount, currency, synthetic card).
Sorted fields and fixed separators feed SHA-256, never Python's randomized
hash(). Defaults such as omitted versus explicit USD produce the same fingerprint.
Invalid input is rejected before execution. Reusing a completed key with a different
operation, payment, amount, or card raises IdempotencyConflict without writing.
Successful responses and committed declines are replayable. Insufficient funds
records the DECLINED snapshot and re-raises InsufficientFundsError on each replay.
Invalid transitions, simulated processor errors, and rolled-back database writes
do not reserve keys. They can be attempted again, with normal validation.
Replay returns the original immutable payment including its history at that
time, even after subsequent transitions. A replayed capture may therefore show
CAPTURED while show displays the current REFUNDED state. A stored JSONB snapshot
and result format version preserve that distinction across process restarts.
Example after authorizing an existing synthetic payment:
capture pay_000001 --idempotency-key order-123-capture
capture pay_000001 --idempotency-key order-123-capture
ledger pay_000001
Both captures return the same snapshot; the ledger has one CAPTURE entry. Restart
the shell and retry the same command to check durable replay. Using that key with
refund fails with a conflict. Use a new key for the refund.
The PostgreSQL repository uses READ COMMITTED and targets the payment row with
SELECT ... FOR UPDATE before reading or validating its current state. The
lock lasts until payment, event, ledger, and idempotency writes commit together.
This makes the state check and effect one serialized operation, without adding a
redundant version column or locking the whole table. Unrelated payments can proceed.
Two captures racing without a shared key produce one CAPTURED payment, one capture
event, and one ledger entry. The losing request reads the newly committed state
and raises InvalidTransitionError. The same applies to different-key requests.
Capture versus cancel allows exactly one winner; the loser fails and no impossible
status/ledger combination persists. Concurrent refunds likewise settle once.
Requests with a key first acquire a PostgreSQL transaction advisory lock derived from the first 64 bits of SHA-256 over a namespaced key. They then check the durable key record before acquiring the target payment row. This also protects creation, when no payment row exists yet. Lock order is always key, then payment. A same-key caller waits for the first transaction and replays its committed response. If the first rolls back, the waiting caller can execute normally. No incomplete key records are committed, and a primary-key constraint independently enforces key uniqueness. Hash collisions only serialize unrelated keys; full-key lookup and request fingerprints still determine replay/conflict behavior.
These are database locks, not a Python-global lock, so separate processes receive the same protection. PostgreSQL releases them on commit, rollback, or connection termination. PostgreSQL's locking documentation describes the row and transaction-advisory lock behavior used here.
Operation scopes bound lock waits to 5 seconds and individual statements to 15
seconds. Lock timeouts, deadlocks, or serialization failures become sanitized
ConcurrentModification errors; retry with the same request/key. There is no
automatic retry loop. Same-key success equivalence assumes normal completion
within those limits; persistent contention can return a retryable error.
The in-memory store implements equivalent replay/conflict semantics and serializes processor operations within that store using an in-process lock. Its contents are ephemeral. It does not prove or provide cross-process safety. All PostgreSQL writers must use the Milestone 3 processor/repository path; raw SQL or older application writers bypass these application guarantees.
FailureEngine implements a small FailureInjector hook contract used at three
boundaries: before a payment transaction, after a newly executed payment transaction
commits, and before a webhook send. NoFailureInjector is the processor and webhook
service default, so code that does not configure failures retains the earlier
behavior. Profiles contain structured data only; they cannot store or execute code.
Client request
|
v
Failure Engine -- BEFORE_COMMIT match? --> abort or delay deterministically
|
v
Payment transaction
+--------------------------------+
| state transition |
| domain event |
| settlement entry |
| webhook event/delivery intent |
| idempotency result |
+--------------------------------+
|
COMMIT
|
v
Failure Engine -- AFTER_COMMIT response loss? --> report injected failure
|
v
Client receives result
Webhook event --> Failure Engine --> latency / timeout / connection error /
synthetic HTTP response / duplicate delivery
--> HTTP dispatcher
A BEFORE_COMMIT processor error or timeout is consumed and audited before the
payment transaction starts. It creates no idempotency result, state transition,
payment event, settlement entry, or webhook event. A retry with the same key can
therefore execute normally. Payment fixed latency also runs at this boundary.
RESPONSE_LOST_AFTER_COMMIT runs only after a newly executed transaction has
committed. The caller receives InjectedResponseLoss, while the payment snapshot,
event, settlement entry, webhook outbox work, and idempotency result remain durable.
Retrying the same request with the same key returns the stored success and does not
consume another after-commit failure or repeat any state change. This models a
central payment-systems ambiguity: the client does not know whether a timeout means
the operation failed or merely that its response disappeared.
Payment profiles support PROCESSOR_ERROR_BEFORE_COMMIT, TIMEOUT_BEFORE_COMMIT,
RESPONSE_LOST_AFTER_COMMIT, and FIXED_LATENCY. They require one operation scope:
create, authorize, capture, cancel, or refund, and may additionally target
a payment ID. Webhook profiles support WEBHOOK_TIMEOUT,
WEBHOOK_CONNECTION_ERROR, WEBHOOK_HTTP_RESPONSE, FIXED_LATENCY, and
DUPLICATE_DELIVERY; they may target an endpoint ID and/or webhook-event ID.
Injected webhook failures create FAILED attempts while leaving the immutable event,
payment state, and settlement untouched. Synthetic HTTP responses accept 400-599.
Latency is a real delay in a normal run and an injected sleeper in tests. It must be an integer from 1 through 5000 milliseconds. Duplicate count must be from 2 through 10. A duplicate profile reserves and sends multiple attempts for the same stored webhook event; it never creates more payment transitions, domain events, settlement entries, or webhook events.
ONCE profiles start with one remaining use and become EXHAUSTED after one matching
hook. Enabling an exhausted profile rearms it for one use. ALWAYS profiles continue
until disabled or removed. Each hook applies at most one profile: highest numeric
priority wins, and the lexically smaller profile ID wins a tie. Priorities are
integers from 0 through 1000. PostgreSQL selects and consumes the winner with
SELECT ... FOR UPDATE, so simultaneous requests cannot both consume one ONCE
profile. The audit row and exhaustion update commit together in a short transaction.
Every applied profile creates a separate FailureInjectionRecord. It identifies
the profile, phase, type, target, operation and available payment/webhook/delivery
IDs, copies the validated options, and records the time. These records are testing
metadata; they do not enter payment history or the settlement ledger. Profiles,
exhaustion, and audit rows persist across PostgreSQL-backed shell restarts. Memory
mode supplies the same deterministic behavior but discards it on exit.
The built-in presets are fail-next-authorization,
lose-next-capture-response, slow-next-capture, timeout-next-webhook,
webhook-503, and duplicate-next-webhook. No preset or profile is random.
A webhook event is the durable, immutable representation of one committed PaymentLab domain event. A delivery attempt records one endpoint-specific send of that event. One event can therefore have many attempts across multiple endpoints or retries without duplicating payment history.
Every committed payment.created, payment.authorized, payment.captured,
payment.canceled, payment.refunded, or payment.declined event gets one webhook
event, even when no endpoint exists. A processor error that leaves the payment in
CREATED creates no transition event. Replaying an idempotent operation returns its
original result without adding a webhook event; conflicting key reuse also adds
nothing. PostgreSQL row locks ensure that a 20-way capture race still produces only
one capture domain event, one CAPTURE ledger entry, and one captured webhook event.
Payment operation
|
v
PostgreSQL transaction
+--------------------------------+
| update payment |
| append domain event |
| append ledger entry |
| append webhook/outbox event |
| reserve enabled-endpoint sends |
| enqueue durable jobs |
| save idempotency result |
+--------------------------------+
|
COMMIT
|
v
Worker claim (short transaction)
|
v
Signed HTTP delivery (no claim transaction held)
|
v
Outcome transaction --> SUCCEEDED / RETRY_SCHEDULED / DEAD_LETTER
The event, PENDING delivery intents, and jobs commit in the payment transaction. HTTP runs only in a worker or an explicitly invoked legacy delivery method, so a slow or unavailable receiver cannot delay, roll back, or corrupt a normal persistent payment command. Queue work survives process shutdown and restart.
The canonical UTF-8 body uses sorted keys and compact separators. Its schema is:
{
"api_version": "2026-09-01",
"created_at": "2026-09-09T12:00:00Z",
"data": {
"payment": {
"amount": 2500,
"currency": "USD",
"id": "pay_000123",
"status": "CAPTURED"
}
},
"id": "wh_evt_000123",
"type": "payment.captured"
}It contains integer minor units and synthetic identifiers, with no card token, credential, ORM state, or current-state lookup. Replay sends the bytes already stored on the event. Replaying a captured event after refund therefore still says CAPTURED and does not change the REFUNDED payment, its ledger, history, or idempotency records.
PaymentLab signs the ASCII timestamp, a period, and the exact raw body with HMAC-SHA256:
HMAC_SHA256(secret, "<timestamp>.<raw_body>")
The POST includes PaymentLab-Webhook-Id, PaymentLab-Webhook-Timestamp, and
PaymentLab-Webhook-Signature: v1=<lowercase hex digest>. Receivers must verify the
unparsed request bytes. paymentlab.webhooks.verify_webhook_signature() validates
the version/shape and uses hmac.compare_digest; it returns false for a modified
body, wrong secret, malformed signature, or mismatched timestamp. No timestamp-age
tolerance is imposed in this local milestone.
Delivery uses httpx, never follows redirects, and has a five-second timeout.
Every reserved send ends as SUCCEEDED for HTTP 2xx or FAILED for non-2xx, connection
errors, and timeouts. Status codes, a response excerpt capped at 512 characters,
or a sanitized bounded error are stored. Payment state is never changed by delivery.
Endpoint URLs must use plain HTTP to localhost, 127.0.0.1, or [::1], include
an explicit port and path, and contain no user information or fragment. HTTPS,
public/other hosts, file:, ftp:, and javascript: are rejected; redirects are
disabled. This restriction makes the feature a local testing receiver rather than
an arbitrary URL fetcher.
Endpoint removal is soft and disables future delivery while retaining historical foreign keys. Normal endpoint listings mask the secret; creation displays it once. Secrets are synthetic and stored as plaintext in PostgreSQL for this milestone. Production encryption, key rotation, access controls, and key management remain outside scope. Avoid putting even synthetic secrets in logs or source control.
An event initially targets only endpoints enabled when it occurs. A later endpoint
does not automatically receive old events. webhook dispatch EVENT_ID sends pending
work and creates attempt 1 only for currently enabled endpoints that have never seen
that event. webhook retry DELIVERY_ID accepts a FAILED delivery and creates the
next attempt for its endpoint. webhook replay EVENT_ID creates a new attempt for
every currently enabled endpoint using the original event and payload. These three
explicit compatibility commands execute immediately and settle their associated
job; normal payment commands leave delivery to workers. None changes the payment.
Events and ledger entries are append-only application data: the repository never updates or deletes them. This is not a tamper-proof audit store; a database owner can modify tables directly. Lifecycle rules live in the processor; database constraints complement those rules rather than replace them.
The queue currently has one real job type, WEBHOOK_DELIVERY. Each initial delivery
intent gets one job in the same transaction that creates the payment event and
webhook event. source_delivery_id is unique, so the same intent cannot acquire two
jobs. Payment idempotency replay and losing concurrent operations create no new
event, delivery, or job. The design can add other job types later without changing
claim or lease mechanics; no placeholder job types are present.
Jobs move through PENDING, RUNNING, RETRY_SCHEDULED, SUCCEEDED, and
DEAD_LETTER. Only PENDING and due RETRY_SCHEDULED jobs, plus RUNNING jobs with an
expired lease, are claimable. SUCCEEDED is terminal. DEAD_LETTER is terminal until
an explicit requeue. Database checks validate status, counters, active lease fields,
completion timestamps, type, and priority.
Each poll uses one short PostgreSQL transaction with SELECT ... FOR UPDATE SKIP LOCKED. Eligible jobs are ordered by descending priority, then ascending
available_at, created_at, and stable job ID. Claiming changes the row to RUNNING,
increments its total attempt and lease-generation counters, stores the worker ID,
and assigns a finite UTC lease. The default lease is 30 seconds; allowed values are
1-300 seconds. Other workers skip a locked claim row and cannot steal an unexpired
lease.
The claim transaction commits before HTTP starts. The worker then prepares a PENDING
delivery attempt, calls the existing signing/failure-aware WebhookService, and
uses another short transaction to record success, retry, or dead letter. Completion
requires the same worker ID and lease generation and a lease that has not expired.
This fencing prevents an old worker from overwriting a result after a newer worker
reclaims the job. If a lease expires after the final allowed attempt, the next poll
dead-letters the job rather than stranding it.
Transient failures—network/timeout results, HTTP 408 or 429, and HTTP 5xx—schedule a durable retry. The default delay is five seconds and the default maximum is three worker attempts. HTTP 4xx other than 408/429 and a disabled/removed endpoint are non-retryable. A retry creates a new numbered delivery attempt for the same event and endpoint while retaining earlier attempts. Exhaustion stores the last bounded error and changes the job to DEAD_LETTER. Manual requeue resets the worker-attempt budget for a new three-attempt cycle but retains all delivery and failure-injection history.
Worker execution is at least once. There is an unavoidable crash window:
worker sends signed HTTP request
receiver accepts it
worker crashes before PaymentLab stores job success
lease expires
another worker reclaims and may send the stable event ID again
PaymentLab does not attempt a distributed transaction with the receiver and does
not claim exactly-once HTTP delivery. Receivers can deduplicate using the stable
PaymentLab-Webhook-Id. Multiple active workers normally claim different jobs;
lease recovery deliberately permits another execution after uncertainty.
Milestone 5 hooks remain on the existing delivery service, so workers honor forced timeouts, connection failures, HTTP 500/503/429 responses, latency, and duplicate delivery. A duplicate-count-three profile makes three HTTP attempts for one event inside one job execution. It does not create three jobs, payment events, webhook events, or settlement entries. PostgreSQL locking still makes ONCE profile consumption safe across competing workers.
PostgreSQL allocates readable IDs from payment_id_sequence, never row counts.
Sequence values survive restarts and may have gaps after failed operations because
PostgreSQL sequences are not rolled back. Direct imports must not invent IDs in
the allocator's namespace. IDs are unique within a database, not globally across
independent databases.
The persistent CLI injects SystemClock, the only location that reads actual UTC
time. Stored timestamps use TIMESTAMPTZ, and connections use UTC. The processor
defaults to LogicalClock for backward-compatible deterministic tests: one second
per committed event, beginning at 2025-01-01T00:00:00+00:00. Tests can inject any
Clock through PaymentProcessor(repository, clock). Naive timestamps are rejected.
Sequence numbers, not wall time, define event order.
The memory backend retains Milestone 1's deterministic IDs and logical time.
Actual timestamps and sequence gaps in persistent runtime are not claimed to be
deterministic. PostgreSQL BIGINT bounds persisted amounts to at most
9223372036854775807 minor units; larger Python integers fail persistence without
partial writes.
| Operation | Required state | Result | Settlement impact |
|---|---|---|---|
| create | — | CREATED | 0 |
| authorize, success | CREATED | AUTHORIZED | 0 |
| authorize, decline/insufficient funds | CREATED | DECLINED | 0 |
| capture | AUTHORIZED | CAPTURED | +amount |
| cancel | AUTHORIZED | CANCELED | 0 |
| refund | CAPTURED | REFUNDED | -amount |
All other transitions raise InvalidTransitionError without changing state,
history, or settlement. CANCELED, REFUNDED, and DECLINED are terminal. Cancellation
from CREATED remains unsupported. Capture and refund apply to the entire amount.
repository.ledger(payment_id) returns immutable entries.
repository.net_settled(payment_id) sums signed durable records in PostgreSQL:
authorization of 2500 settles 0, capture settles 2500, and refund restores net 0.
Balance is reconstructed from ledger records, not inferred from current status.
Ledger entry IDs derive from the payment ID and source event sequence.
History records payment.created, payment.authorized, payment.captured,
payment.canceled, payment.refunded, and payment.declined. This is an internal
state-change trail, not a webhook or attempted-operation log.
| Token | Authorization behavior |
|---|---|
test_card_success |
Returns AUTHORIZED. |
test_card_declined |
Returns DECLINED and records payment.declined. |
test_card_insufficient_funds |
Persists DECLINED and its event, then raises InsufficientFundsError. |
test_card_processing_error |
Raises ProcessorError; remains CREATED with unchanged history. Repeating authorization gives the same error. |
None of the failure cards create settlement entries.
Requires Python 3.11+ and PostgreSQL (verified with 17). From the project root in PowerShell:
py -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -e ".[dev]"
docker compose up -d --wait postgres
$env:PAYMENTLAB_DATABASE_URL="postgresql+psycopg://paymentlab:paymentlab_local_only@localhost:5432/paymentlab"
.\.venv\Scripts\python.exe -m alembic upgrade head
.\.venv\Scripts\paymentlab.exe shellCompose runs only PostgreSQL, binds to loopback, and uses a named volume for
durability. These credentials are intentionally local-development-only.
docker compose down retains its volume; adding -v removes stored payments.
The application itself is not containerized.
Without Docker, install PostgreSQL, create an empty paymentlab database and a
local role with permission to create tables, configure that server's URL, and run
the same migration. No tables require manual creation.
Official Windows installers and binary archives
are available for native development.
On macOS/Linux use python3 -m venv .venv, .venv/bin/python, and
.venv/bin/paymentlab; set configuration with export PAYMENTLAB_DATABASE_URL='...'.
After environment activation, python -m paymentlab and paymentlab are equivalent.
.env.example documents both variables. PaymentLab does not load .env
automatically: set environment variables in your shell. Actual .env files,
virtual environments, caches, and local database artifacts are ignored.
Only postgresql:// and postgresql+psycopg:// URLs are accepted.
Alembic revision 0001 builds the original storage tables and payment sequence;
0002 adds durable idempotency records; 0003 adds webhook endpoint, event, and
delivery tables plus their readable-ID sequences; 0004 adds failure profiles,
injection audit rows, constraints, indexes, and readable-ID sequences; 0005 adds
the durable jobs table and sequence and backfills jobs for existing PENDING webhook
attempts; 0006 adds merchant ownership, hashed test API keys, and rate counters;
0007 adds merchant-scoped scenario runs and incremental step results; 0008
adds normalized durable API request observations.
Earlier migrations are unchanged. upgrade head works from 0007 or on
a fresh empty database. Run migrations explicitly; CLI startup does not create or
alter tables.
python -m alembic current shows the revision. Downgrade to base removes the
schema and its data and is intended only for disposable databases.
Downgrading from 0002 to 0001 retains payment/event/ledger data but permanently
discards replay records. Re-upgrading does not recover those records; do not
downgrade a database whose clients still rely on prior keys.
Downgrading from 0003 to 0002 retains Milestone 1-3 data but permanently removes
all webhook configuration, events, attempts, and webhook sequence state.
Downgrading from 0004 to 0003 retains payments, settlement, idempotency, and
webhooks but permanently removes failure profiles, exhaustion state, and injection
audit records.
Downgrading from 0005 to 0004 retains payments and webhook attempts but removes
queue state. Re-upgrading backfills a PENDING job for each still-pending attempt.
Downgrading from 0007 to 0006 preserves all payments, webhooks, jobs, failures,
merchants, and API keys, but permanently removes scenario execution history.
Downgrading from 0008 to 0007 preserves all prior state but permanently removes
API request observation history.
With PAYMENTLAB_DATABASE_URL set, run paymentlab shell. Alternatively use
paymentlab shell --database-url "postgresql+psycopg://.../paymentlab".
First session against a fresh migrated database:
create --amount 2500 --card test_card_success
authorize pay_000001
capture pay_000001
ledger pay_000001
quit
Net settlement is 2500. Start paymentlab shell again:
list
show pay_000001
ledger pay_000001
refund pay_000001
quit
The payment initially reloads as CAPTURED with its original history. Restart a
third time and run show pay_000001 and ledger pay_000001: status is REFUNDED,
entries are +2500 and -2500, and net settlement is 0. Use the actual ID from
create if the database is not fresh. Creating and quitting before authorization
also persists CREATED across restarts.
Commands: create, authorize, capture, cancel, refund, show, ledger,
list, cards, help, and quit/exit. create accepts --amount, --card,
and optional --currency USD. Enter create --help for syntax. Input can also be
piped to the shell; each successful operation commits immediately, not at exit.
Webhook commands are grouped beneath webhook:
webhook endpoint add --url http://127.0.0.1:9000/webhooks [--secret SECRET]
webhook endpoint list
webhook endpoint disable wh_ep_000001
webhook endpoint enable wh_ep_000001
webhook endpoint remove wh_ep_000001
webhook events [--payment-id pay_000001]
webhook show wh_evt_000001
webhook deliveries [wh_evt_000001]
webhook pending
webhook dispatch wh_evt_000001
webhook retry wh_del_000001
webhook replay wh_evt_000001
Omitting --secret generates a 32-byte URL-safe secret. Copy the value displayed
at creation; subsequent list output reveals only its final four characters. The
event show command prints the exact canonical JSON. Delivery output includes event,
endpoint, attempt number, status, and HTTP status without printing secrets.
Failure commands are grouped beneath failure:
failure profile add --name fail-next-capture --target payment --operation capture --type processor-error-before-commit --mode once
failure profile add --name slow-hook --target webhook --type fixed-latency --latency-ms 250 --endpoint-id wh_ep_000001 --mode always
failure profile add --name synthetic-503 --target webhook --type webhook-http-response --status-code 503
failure profile add --name duplicate-hook --target webhook --type duplicate-delivery --count 3
failure profile list
failure profile show fail_prof_000001
failure profile enable fail_prof_000001
failure profile disable fail_prof_000001
failure profile remove fail_prof_000001
failure preset lose-next-capture-response
failure preset duplicate-next-webhook --priority 200
failure injections
Use --payment-id to narrow a payment profile and --endpoint-id or --event-id
to narrow a webhook profile. --priority defaults to 100. Profile listings show
ENABLED, DISABLED, or EXHAUSTED, plus target, type, mode, scope, priority, and
validated configuration. failure injections distinguishes deliberate failures
from ordinary application or HTTP failures.
Flagship shell flow, using the ID printed by create:
create --amount 2500 --card test_card_success
authorize pay_000001
failure preset lose-next-capture-response
capture pay_000001 --idempotency-key capture-demo-1
show pay_000001
ledger pay_000001
webhook events --payment-id pay_000001
capture pay_000001 --idempotency-key capture-demo-1
failure injections
The first capture command reports an injected response loss even though its transaction committed. The second capture returns the original CAPTURED snapshot. The ledger and webhook event list still contain one capture effect.
Queue inspection is available in the shell:
jobs list
jobs list --status PENDING --type WEBHOOK_DELIVERY
jobs show job_000001
jobs dead
jobs requeue job_000001
Run a PostgreSQL-backed worker from PowerShell. --once polls once and executes at
most one job; --max-jobs N is another finite mode. Without either option the
worker polls continuously until Ctrl+C.
$env:PAYMENTLAB_DATABASE_URL="postgresql+psycopg://paymentlab@127.0.0.1:55432/paymentlab"
.\.venv\Scripts\paymentlab.exe worker run --once --worker-id local-worker-1
.\.venv\Scripts\paymentlab.exe worker run --max-jobs 10 --worker-id local-worker-2
.\.venv\Scripts\paymentlab.exe worker run --worker-id local-worker-3 --poll-interval 1Worker IDs use 1-64 simple ASCII characters. Options also expose
--lease-seconds (default 30), --retry-delay-seconds (default 5), and
--poll-interval (default 1). Output includes worker ID, claimed job ID/type,
terminal or retry outcome, and each delivery result. Ctrl+C ends the polling loop;
an in-flight claim remains recoverable when its finite lease expires.
The repository-local PostgreSQL installation can run the complete demo without Docker. In PowerShell at the project root, start it and migrate the application database:
& '.\.local\postgresql\pgsql\bin\pg_ctl.exe' -D '.local\pgdata' -l '.local\postgres.log' -o '-p 55432 -h 127.0.0.1' -w start
$env:PAYMENTLAB_DATABASE_URL='postgresql+psycopg://paymentlab@127.0.0.1:55432/paymentlab'
.\.venv\Scripts\python.exe -m alembic upgrade headIn a second PowerShell window, start the receiver and leave it running:
$env:PAYMENTLAB_WEBHOOK_SECRET='synthetic_worker_demo_secret_123456'
$env:PAYMENTLAB_WEBHOOK_PORT='19093'
.\.venv\Scripts\python.exe examples\webhook_receiver.pyIn the first window, register that endpoint, create the first event, and inspect the pending job. Use the actual IDs printed if the database is not empty.
@'
webhook endpoint add --url http://127.0.0.1:19093/webhooks --secret synthetic_worker_demo_secret_123456
create --amount 2500 --card test_card_success
jobs list --status PENDING
quit
'@ | .\.venv\Scripts\paymentlab.exe shell
.\.venv\Scripts\paymentlab.exe worker run --once --worker-id demo-worker
@'
webhook deliveries wh_evt_000001
jobs show job_000001
create --amount 2600 --card test_card_success
jobs list --status PENDING
quit
'@ | .\.venv\Scripts\paymentlab.exe shellThe payment command returns before any HTTP request. The first worker prints an HTTP 200 delivery and a SUCCEEDED outcome. The following one-line service call models a crashed worker by claiming the second job for one second and exiting without delivery or completion:
.\.venv\Scripts\python.exe -c "from datetime import datetime,timezone; from paymentlab.persistence.database import create_database_engine; from paymentlab.persistence.postgres import PostgresPaymentRepository; e=create_database_engine(); r=PostgresPaymentRepository(e); print(r.claim_job('demo-crashed-worker',datetime.now(timezone.utc),1)); e.dispose()"
Start-Sleep -Seconds 2
.\.venv\Scripts\paymentlab.exe worker run --once --worker-id demo-recovery-worker
@'
webhook deliveries wh_evt_000002
jobs show job_000002
quit
'@ | .\.venv\Scripts\paymentlab.exe shellThe recovery worker increments the lease generation and attempt count, delivers the same stable event, records HTTP 200, and leaves the job SUCCEEDED. Stop the receiver with Ctrl+C, then stop PostgreSQL from the project root:
& '.\.local\postgresql\pgsql\bin\pg_ctl.exe' -D '.local\pgdata' -m fast -w stopRegister an endpoint in one PowerShell session, copy the one-time synthetic secret, then start the example in another session with that exact value:
$env:PAYMENTLAB_WEBHOOK_SECRET="paste-the-generated-synthetic-secret"
$env:PAYMENTLAB_WEBHOOK_PORT="9000"
.\.venv\Scripts\python.exe examples\webhook_receiver.pyThe receiver binds only to 127.0.0.1, verifies the three PaymentLab headers against
the raw request bytes, prints the event type and payment ID, returns 200 for valid
signatures, and returns 401 for invalid ones. Stop it with Ctrl+C. While it is down,
a payment operation still commits and its job remains pending. A worker attempt
records failure and schedules a durable retry; after the receiver restarts, a due
attempt can succeed. The secret is an environment variable only; the example does
not load .env files.
paymentlab shell --memory explicitly selects ephemeral storage and ignores the
database environment variable. Without a URL, the shell also falls back to memory
for Milestone 1 compatibility. Its interactive banner identifies the storage mode.
paymentlab demo stays deterministic and ephemeral unless given an explicit
--database-url.
Memory shell mode keeps the earlier immediate-delivery behavior for deterministic backward-compatible examples. PostgreSQL shell mode is the durable asynchronous path and never drains the queue after a payment command.
Errors print to stderr and the shell continues. Exit status is 1 if any command failed, otherwise 0; invalid top-level arguments return 2. Normal declines are not command errors; insufficient funds and processor failures are. EOF or Ctrl+C ends input; committed PostgreSQL data survives exit.
With PostgreSQL migrated to 0007, the merchant API running, and the synthetic test
key/base URL set in PAYMENTLAB_API_KEY and PAYMENTLAB_BASE_URL:
.\.venv\Scripts\paymentlab.exe scenarios validate examples\scenarios\happy-path.yaml
.\.venv\Scripts\paymentlab.exe scenarios run examples\scenarios\happy-path.yaml
.\.venv\Scripts\paymentlab.exe scenarios run examples\scenarios\lost-capture-response.yaml --json
.\.venv\Scripts\paymentlab.exe scenarios list-runs
.\.venv\Scripts\paymentlab.exe scenarios show-run scn_run_000001Payment and setup operations cross the public HTTP SDK boundary. Bounded worker and queue controls plus built-in payment, settlement, webhook, idempotency, failure-audit, and queue assertions are merchant scoped. Results and step details persist after each run with a canonical SHA-256 source hash. Reports recursively redact API keys, authorization values, signing secrets, tokens, and password-bearing database URLs. See Chaos scenarios for the exact schema, exit codes, runner boundaries, multi-actor configuration, and seven curated examples.
With PAYMENTLAB_DATABASE_URL set and the database migrated to 0008, run:
.\.venv\Scripts\paymentlab.exe api run --host 127.0.0.1 --port 8000 --log-format json
.\.venv\Scripts\paymentlab.exe dashboard run --host 127.0.0.1 --port 8080 --log-format textOpen http://127.0.0.1:8080/. The separate dashboard is an unauthenticated,
system-wide operator tool that binds only to loopback choices. It is read-only and
does not add administrative methods to merchant /v1 APIs or the SDK. The page
polls every five seconds and uses repository-bundled HTML, CSS, and JavaScript.
Both processes expose cheap /healthz, PostgreSQL-aware /readyz, and durable
Prometheus-compatible /metrics. Dashboard JSON endpoints provide bounded 1-hour,
24-hour, or 7-day SQL aggregates for requests, payments, webhooks, jobs, active
failure profiles/injections, and scenario runs. See
Observability and dashboard for metric names, safe labels,
redaction rules, observation failure policy, retention, and endpoint details.
With the application database URL configured, doctor checks the installed version,
Python, PostgreSQL connectivity, and Alembic current/head state without mutating data:
.\.venv\Scripts\paymentlab.exe doctorBenchmarks use the isolated test database and refuse the normal application database:
$env:PAYMENTLAB_TEST_DATABASE_URL='postgresql+psycopg://paymentlab@127.0.0.1:55432/paymentlab_test'
.\.venv\Scripts\paymentlab.exe benchmark run --iterations 10 --warmup 2 --workload all --output .local\benchmark.jsonResults distinguish in-memory, PostgreSQL, idempotent replay, in-process ASGI, job, and scenario workloads. See Benchmark methodology for timing, isolation, JSON schema, and limitations.
from paymentlab import PaymentProcessor
from paymentlab.clock import SystemClock
from paymentlab.persistence.database import create_database_engine
from paymentlab.persistence.postgres import PostgresPaymentRepository
engine = create_database_engine() # PAYMENTLAB_DATABASE_URL
try:
repository = PostgresPaymentRepository(engine)
processor = PaymentProcessor(repository, SystemClock())
payment = processor.create(2500, "test_card_success")
processor.authorize(payment.id)
processor.capture(payment.id)
print(repository.net_settled(payment.id)) # 2500
finally:
engine.dispose()Repositories use short-lived sessions; the caller owns the engine. Domain tests
can use PaymentProcessor(InMemoryPaymentRepository()) as before.
Unit tests need no running database:
.\.venv\Scripts\python.exe -m pytest -m "not integration"Start the repository-local native PostgreSQL instance and use the existing separate test database:
& '.\.local\postgresql\pgsql\bin\pg_ctl.exe' -D '.local\pgdata' -l '.local\postgres.log' -o '-p 55432 -h 127.0.0.1' -w start
$env:PAYMENTLAB_TEST_DATABASE_URL="postgresql+psycopg://paymentlab@127.0.0.1:55432/paymentlab_test"
.\.venv\Scripts\python.exe -m pytest -m integrationFor native PostgreSQL use createdb -h localhost -U paymentlab paymentlab_test
(add your server port). Skip creation if the test database already exists.
Integration tests never use PAYMENTLAB_DATABASE_URL. They require exactly
paymentlab_test on localhost, 127.0.0.1, or ::1, reject URL query parameters
that could override the target, and confirm the connected database name. Wrong
targets fail; absent configuration skips with an explicit reason. A configured
but unreachable database fails rather than silently skipping.
Each integration test creates a uniquely named schema, applies the real Alembic migration, and removes only that schema afterward. Existing application tables are not truncated; no databases are dropped. Schema-name randomness isolates test infrastructure only, never payment outcomes. The test role needs schema-creation permission. Run all tests with the test URL configured:
.\.venv\Scripts\python.exe -m pytestTests retain all 75 Milestone 1 cases and add clock/configuration/ledger tests, persistence and durable IDs, event reconstruction, refund and failure scenarios, database constraints, append-only repository behavior, and migration round-trip and schema comparison. Real database constraints reject final ledger inserts to prove that already-flushed status and event changes roll back together. These constraints exist only in isolated tests, not as runtime failure injection.
Milestone 3 tests exercise all five replay paths against both stores, conflicting requests, snapshots after later transitions, recorded declines, restart replay, and atomic rollback of the idempotency write. Concurrency tests use independent connections and barriers. A controlling transaction holds a payment row (or create key) until PostgreSQL reports every worker waiting on a real lock, then releases them. Bounded waits prevent hangs. Tests include same-key/different-key/no-key captures, capture versus cancel, concurrent refunds, same-key creation, unrelated payment progress, and 20 simultaneous capture attempts. The test-database safety checks remain unchanged.
Milestone 4 tests cover canonical payload bytes, HMAC verification failures, URL
safety, endpoint lifecycle and secret masking, 2xx/500/network classification,
multiple and late endpoints, retry/replay, processor errors, idempotency interaction,
and immutable replay after refund. PostgreSQL tests prove outbox rollback with the
payment/event/ledger/idempotency transaction, event persistence after restart,
pending-attempt recovery, database constraints, 0002 -> 0003 -> 0002, and one
captured webhook event under 20 concurrent captures. HTTP tests use controlled
loopback servers on dynamically allocated ports and make no public network calls.
Milestone 5 tests cover profile validation, all failure types, scopes, ONCE/ALWAYS
lifecycle, deterministic priority, injected sleepers, audit records, and CLI
presets. PostgreSQL tests prove pre-commit rollback semantics, durable lost-response
replay after repository restart, concurrency-safe one-shot consumption, one outcome
under a 20-way capture race, synthetic webhook failures without an HTTP request,
one event with three duplicate attempts, and 0003 -> 0004 -> 0003. Tests use no
random fault selection, public service, real payment processor, or long sleep.
Milestone 6 tests cover atomic job enqueueing, state validation, deterministic
priority/availability ordering, one-job/two-claimer SKIP LOCKED behavior, 20 jobs
across four workers, active and expired leases, fencing stale owners, the external
success crash window, durable retries, dead-letter exhaustion, manual requeue,
graceful finite workers, all Milestone 5 webhook failures through workers, one-shot
failure consumption, duplicate delivery with one job/event, idempotent lost-response
regression, and 0004 -> 0005 -> 0004 -> 0005 migration behavior.
Milestone 8 tests add strict YAML validation, constrained interpolation and redaction,
durable scenario records, expected errors, explicit retry, public API concurrency,
rate limits, tenant isolation, webhook duplicate guarantees, lease recovery,
dead-letter requeue, and the 0006 -> 0007 -> 0006 -> 0007 migration path.
Milestone 9 tests cover adversarial secret redaction, JSON/text logs, concurrent
context isolation, successful and failed request observations, normalized routes,
best-effort observation failure, exact PostgreSQL latency percentiles, bounded time
windows and pagination, every operational domain, Prometheus labels, read-only
dashboard assets/endpoints, readiness failure, and migration 0008 round trips.
All ten planned milestones are complete at version 1.0.0. PaymentLab remains focused on local synthetic testing. PostgreSQL is required for durable features; webhooks are localhost-only and at-least-once; the dashboard trusts its loopback workstation; and benchmarks are machine-specific engineering measurements. There are no real payment rails, merchant processing, PCI claims, exactly-once external delivery guarantees, cloud deployment promises, or production scalability claims.