Testing & Benchmarks¶
CRP ships with a comprehensive test suite covering every SDK level, protocol feature, and product surface - from smoke tests to live LLM verification - so regressions are caught before they reach users.
CRP has 60+ test files and 1,537+ tests covering every module from smoke tests to live LLM verification.
Overview¶
| Category | Files | What It Covers |
|---|---|---|
| Smoke | 1 | Package imports, version, basic types |
| Unit - Core phases | 9 | All 9 SDK phases |
| Unit - Specialized | 14+ | Deep module-level coverage |
| Integration | 1 | Cross-module E2E |
| Production hardening | 1 | Circuit breakers, retries, cleanup |
| Benchmarks | 1 | Performance regression |
| Live LLM | 4+ | Real model verification |
| v5 features | 12+ | Safety surface, STL, horizons, scratch buffer, CSO, SDK levels, Gateway, Comply, Scan |
Quick Start¶
# Install dev dependencies
pip install -e ".[dev]"
# Run smoke tests (fast, ~2 seconds)
python -m pytest tests/test_smoke.py -v
# Run a specific phase
python -m pytest tests/test_phase1.py -v --tb=short
# Run with coverage
python -m pytest tests/test_smoke.py --cov=crp --cov-report=term
Do NOT run all tests at once
The full suite is resource-intensive. Run one file at a time to avoid maxing out system resources. See Running Tests for guidance on pytest-timeout and safe execution.
Sections¶
- Running Tests - How to run each test category, what each file covers, fixtures, and tips
- Benchmarks - Performance results: continuation engine, extraction pipeline, protocol overhead
- Reproduce Benchmarks - Step-by-step guide to reproduce our benchmark results on your own hardware