Skip to content

test: add memory checks and fuzz testing for REST and WebSocket parsers - #35

Merged
SebastianBoehler merged 6 commits into
SebastianBoehler:mainfrom
yluoc:fuzztesting
Oct 11, 2026
Merged

SebastianBoehler merged 6 commits into
SebastianBoehler:mainfrom
yluoc:fuzztesting

Conversation

@yluoc

@yluoc yluoc commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

Summary

The SDK parses REST responses and WebSocket messages it does not control, but the tests only covered inputs we wrote by hand. Nothing checked how the parsers handle malformed or unexpected data, and AddressSanitizer and UBSan never ran in CI.

This PR adds three layers of checks:

  • Sanitizer job (build.yml): builds and runs the offline suite with AddressSanitizer and UBSan on Ubuntu 22.04. test_package_consumer is excluded because it links an uninstrumented external project; the regular matrix still runs it.
  • Fuzz targets (tests/fuzz/):
    • fuzz_websocket_messages covers the market, tick size, and user stream parsers. Accepted book updates must leave every level with a price in (0, 1), a non-negative size, and correct ordering.
    • fuzz_rest_responses covers the order book, open order, order response, cancellation, trade, and market parsers. Accepted order books must have valid levels and no duplicate prices.
    • Rejected input must throw std::exception. A crash, sanitizer report, or broken invariant is a failure.
    • Includes 15 seed inputs taken from existing tests.
    • POLYMARKET_CLIENT_BUILD_FUZZERS=ON (off by default, Clang only) builds libFuzzer targets. Normal test builds compile the same harnesses with a replay driver, so every CI configuration replays the seeds.
  • Fuzz workflow (fuzz.yml): runs each fuzzer for 1 minute on PRs, 15 minutes nightly, or a chosen duration from a manual run. Failing inputs are uploaded as an artifact.

CONTRIBUTING.md documents how to run the fuzzers locally.

Verification

Commands run and their actual results:

  • ctest -LE live -E '^test_package_consumer$' (gcc 13, Debug, ASan + UBSan): 63 of 63 passed, including both seed replays.
  • ctest -L fuzz (Clang 18 fuzz build): 2 of 2 passed.
  • fuzz_websocket_messages -max_total_time=180: about 1.44M inputs, no findings.
  • fuzz_rest_responses -max_total_time=180: about 576K inputs, no findings.
  • fuzz.yml steps run locally in a fresh build (fuzz_rest_responses, 60 s): seed replay passed, about 186K inputs, no findings.
  • A deliberately crashing fuzzer exited with code 77 and wrote crash-* to fuzz-run/artifacts/, the path the upload step collects.
  • clang-format --dry-run --Werror and clang-tidy (21.1.6) on tests/fuzz/*.cpp: clean.

Checks not run and the reason (or none):

  • Prettier and the full scripts/quality.py: the local npx resolves to a Windows npm and fails. CI will run them.
  • macOS replay build: no local macOS machine.
  • Both workflows on GitHub: they first run on this PR, and the nightly fuzz job starts after merge.

Compatibility and evidence

Public API, binary compatibility, or release implications (or none):

None. No public headers or library behavior changed. The new CMake option is off by default, and the replay executables are built only when tests are enabled.

For protocol changes: not applicable.
For live checks: not applicable. All results come from local fixtures and fuzz runs; no live endpoints were used.

@yluoc

yluoc commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

@SebastianBoehler
Our SDK reads data from Polymarket's servers that we don't control. Our normal tests only try messages we thought of ahead of time. Fuzz testing throws millions of random and broken messages at the code to find ones that crash it or quietly corrupt an order book. For a trading SDK, a bad message should give a clean error, never a crash or a wrong price. Fuzzing is the cheapest way to make sure of that.

@SebastianBoehler

Copy link
Copy Markdown
Owner

Thanks Bill, I reviewed the latest head (b74bdf2). All seven CI checks pass, and both seed replays plus the EVM transaction tests passed locally.

I found one build issue to fix before merging: in cmake/PolymarketFuzz.cmake:12-13, the sanitizer link flags are INTERFACE only. With BUILD_SHARED_LIBS=ON and POLYMARKET_CLIENT_BUILD_FUZZERS=ON, the macOS shared library fails to link with unresolved ASan/UBSan symbols, including ___asan_init.

Could you apply the sanitizer link flags to the library itself as well? Changing INTERFACE to PUBLIC, while keeping the existing BUILD_INTERFACE guard, should address this. I confirmed that adding -fsanitize=address,undefined to the generated library link command makes it succeed.

Please add a focused shared-library build check with fuzzers enabled so this configuration stays covered. The rest of the reviewed changes look good.

@SebastianBoehler

Copy link
Copy Markdown
Owner

@yluoc Thanks for the sanitizer link fix in fa19611. I confirmed that the shared library now links locally.

The new macOS shared-library CI job is still failing. It stops while compiling IXWebSocket with Homebrew LLVM 18: <cstddef> cannot find libc++'s <stddef.h>, followed by nullptr_t errors. This happens before the shared-library link or corpus replay.

Could you fix the macOS compiler/SDK/header configuration and rerun the job? The other seven checks pass. We need this new check to complete successfully before merging.

@SebastianBoehler
SebastianBoehler merged commit b37aeb8 into SebastianBoehler:main Oct 11, 2026
7 of 8 checks passed
@yluoc
yluoc deleted the fuzztesting branch October 11, 2026 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants