Skip to content

Testing vfy

Find the suite for your change in the test layers table. Rust tests check the algorithms. Python integration tests exercise the public APIs and compare supported behavior with pinned upstream versions.

Run the required checks

CONTRIBUTING.md lists environment setup and the full pre-PR checklist. Use this page to choose a suite and understand what it checks.

Linux CI is configured to run the Rust tests with cargo-insta and nextest. It rejects pending or unreferenced snapshots:

scripts/cargo-insta-test.sh --locked --workspace --no-default-features

The configured Rust matrix also uses nextest on Linux and macOS. Installed-wheel jobs check CPython and PyO3 behavior. These configurations do not establish that a particular revision passed; check its job results.

tests.support.vfy_project.VfyProject runs temporary projects, Git baselines, subprocesses, and schema-7 reads. It isolates ambient Git, Coverage.py, pytest, and home configuration, and a test can override the environment explicitly. Commands time out after a configurable 120 seconds and keep both streams. Cleanup terminates the POSIX process group, or uses Windows taskkill while the parent is alive. Descendants that leave the group, or outlive their Windows parent, may survive.

Python contracts

The repository typing gate covers Python 3.12–3.15, because version guards select different standard-library APIs, such as sys.monitoring events. Read-only sources/ and semantic fixture inputs are outside the typing gate. Fixtures keep their behavioral harnesses.

Native signatures are checked by tests/repository/test_native_typing.py. tests/typing/consumer.py has positive type assertions and named expected diagnostics for invalid calls. After you install a built wheel, run uv run --no-sync python scripts/check_wheel_typing.py to repeat the consumer checks outside the checkout. The wheel lanes run this check on their installed interpreters.

Unexpected warnings and strict xfail successes fail vfy's own pytest suite. Child projects set their own warning and xfail policy when they test those behaviors on purpose.

Test layout

Rust tests live beside their owning modules. Run installed-package contracts with uv run pytest -q tests/integration and repository checks with uv run pytest -q tests/repository. Shared process helpers live in tests/support/; reviewed source inputs live in tests/fixtures/.

Test layers

Change Owning checks Focused command
Rust algorithms, ordering, state, persistence, and merge atomicity Tests beside the owning module; property tests check algebra and round trips scripts/cargo-test.sh --locked --workspace --no-default-features
Report aggregation, filtering, escaping, and rendering Rust report tests and reviewed snapshots scripts/cargo-insta-test.sh --locked -p vfy-core
Python source analysis Literate analysis fixtures and direct Rust harness cargo test --locked -p vfy-core analysis
Semantic collection and reports Reviewed semantic fixtures and public integration tests uv run pytest -q tests/integration/test_semantic_fixtures.py tests/integration/test_semantic.py
Coverage.py API and schema-7 compatibility API, data, and analysis differentials uv run pytest -q tests/integration/test_api.py tests/integration/test_data_compat.py tests/integration/test_analysis_compat.py
Native pytest execution, workers, and lifecycle Native pytest testing Use the focused command on that page
Public CLI, pytest-cov, selection, and watch Integration tests through VfyProject uv run pytest -q tests/integration
Diagnostics and invalid configuration Streams, exit statuses, and absence of work uv run pytest -q tests/integration/test_cli_diagnostics.py
Built-in asyncio Differential tests against pytest-asyncio 1.4.0 uv run pytest -q tests/integration/test_asyncio.py
Built-in HTML and metadata Reports, hooks, redaction, and publication uv run pytest -q tests/integration/test_pytest_html.py
Installed artifacts Fast Linux wheel lane and full platform wheel matrix CI wheel-fast or wheel job
Repository health Locked licenses, advisories, unused code, and workflow security CI supply-chain jobs
Performance Parity-gated committed harness and retained samples Benchmark commands

Focused development loop

Use the table above for the smallest owning suite. For a CLI iteration, narrow it with uv run pytest -q tests/integration/test_cli.py -k vfy_test. Before handoff, run the full pre-PR checklist. -k is only an iteration aid; release validation uses complete suites.

CI change classification

scripts/ci_change_plan.py classifies each change conservatively:

  • Documentation-only changes skip the platform jobs.
  • Ordinary code and test changes keep the Linux wheel lane.
  • Collector, PyO3, dependency, build, workflow, and unknown changes run the full platform matrix.
  • Updates to main always run the full matrix.

tests/repository/test_workflows.py checks that every workflow job that does not call a reusable workflow declares timeout-minutes.

The CI summary gate requires the change classification and every selected job to succeed. Missing, cancelled, or skipped required jobs fail it.

Every non-documentation tier also requires the Linux CPython 3.12 native adoption subset. It reuses the Linux wheel that the job built. Documentation-only changes skip that job explicitly, and the summary requires that skipped status.

The same required job is configured to compare the pinned iniconfig, itsdangerous, and h11 suites on CPython 3.13 through stock pytest and vfy test --engine=native. The independent route uses the same wheel in a second environment without pytest. The real-package corpus gate checks ordered JUnit identities and outcomes, exit status, and outcome counts; it rejects missing, stale, empty, or incompatible evidence. It retains commands, dependency versions, artifact hashes, reports, and transcripts under target/native-adoption/independent/. This check does not compare full report phases, diagnostics, warnings, or coverage. A configured gate does not establish that a particular wheel passed it.

Measuring CI resource use

When you change CI tiering, keep before-and-after completed-run samples. Each sample records the commit, lane, wall-clock duration, and billed runner minutes. Store them in a JSON file:

{
  "samples": [
    {
      "revision": "<commit>",
      "lane": "quality",
      "duration_seconds": 123.4,
      "runner_minutes": 18.75
    }
  ]
}

Summarize each file with scripts/ci_metrics.py:

uv run python scripts/ci_metrics.py ci-metrics/before.json --label before
uv run python scripts/ci_metrics.py ci-metrics/after.json --label after

Keep the raw records with the measurement note. Report an improvement only when both sample sets use the same workflow revision window and runner classes. Otherwise, describe the result as an observation, not a speed claim.

Review discipline

Use the testing instructions to choose the owning layer, an independent oracle, and the smallest regression fixture. Performance changes also follow the measurement instructions.

Semantic coverage in depth

Semantic changes need independent checks of evaluation, CPython event mapping, persistence, and public reports. Ordinary line and branch differentials must still pass with decisions disabled.

Rust invariants

crates/vfy-core/src/semantic/logic/tests.rs and crates/vfy/src/semantic/tests.rs cover interpreter-free rules: tri-state short-circuit evaluation, not polarity, masking MC/DC pairs, stable source analysis, and pragma exclusions. Reviewed truth tables and generated logic trees (at most four conditions and four levels of nesting) compare the Rust evaluator with an independent reference oracle. Every observation pair is checked for each condition. The oracle checks behavior. It does not copy the evaluator's implementation.

Literate fixtures

tests/fixtures/semantic/*.md is the review surface for supported Python syntax. Each fixture has one executable Python block and one expected JSON block. tests/integration/test_semantic_fixtures.py runs each source through the real collector and compares normalized definitions and observation vectors. Expected files omit temporary paths and source digests, and keep the kind, source line, expression, logic tree, vector, and outcome. The fixtures cover short-circuit and chained conditions, control-flow forms, tri-state observations, exclusions, exceptioning truth tests, generator suspension, and atomic-condition source ranges. They check that collection does not repeat truth tests, fabricate outcomes, or retain incomplete vectors.

The Rust evidence API is tested end to end. Each reported pair is checked against the same Rust proof used for the aggregate percentage, including context filtering and invalid short-circuit rows. Python report tests check that they adapt Rust rows. They do not carry a second MC/DC algorithm.

Adding a syntax form means adding a fixture with a hand-reviewed truth table. Do not add another large string to a Python test. A fixture failure is a semantic regression until the implementation or the language contract is reviewed. Never regenerate expected JSON blindly.

Public-contract tests

The Python suite keeps fixtures focused and tests each boundary separately:

  • schema-7 compatibility and atomic semantic persistence;
  • shard combine, path aliases, source-digest conflicts, append replacement, stale data, contexts, and optimization-level behavior;
  • CLI, pytest options, JSON shape, terminal output, and threshold exit codes;
  • Coverage.py and pytest-cov differential behavior for ordinary line and branch modes, which must not change when decisions are disabled;
  • mutation-testing boundaries: verified passing baselines, stale coverage data, imported aliases, nested and decorated callables, fixture setup, call, and teardown, async and generator owners, xfail semantics, timeouts, fallback, and resumable per-mutation cache entries.

Release validation

Release publication requires the complete Python suite against installed ABI3 wheels on Linux CPython 3.12–3.15, and on macOS ARM64 CPython 3.12, 3.14, and 3.15. The same commit must pass the Rust workspace tests on Linux.

The Linux CPython 3.12 wheel checks are configured to run pytest 9.1.0, 9.1.1, and 9.2.0. Each version runs the native version-boundary suite without xdist, then installs pytest-xdist 3.8.0 for the broader native suite's stock-pytest comparisons. They also run pinned pytest 9.1.1 with the pytest-xdist 3.6.1 minimum for direct-pytest coverage integration. The built-in asyncio check runs without pytest-asyncio installed.

Native engine release criteria

Native vfy test is experimental. Removing that label requires the supported test workflows to pass against stock pytest on Linux and macOS, on every advertised CPython version and supported pytest release. The native installed-wheel matrix covers CPython 3.12–3.15. Verify installation and execution on every advertised platform and interpreter. Unsupported workflows must fail with an actionable diagnostic, without a stock pytest retry. Add compatibility proof before widening the pytest range.

The version-boundary suite also checks locked pytest 9.1.1 without xdist; the version-check method lists the complete matrix. Corpus and real-project comparisons use 9.1.1.

Area Required proof
Test behavior Rust invariants, plus public VfyProject differentials against pinned pytest for success, failure, teardown, interruption, and repeated sessions
Plan and bindings Rust typed-plan validation, plus public tests for dynamic fixtures, binding resolution, warnings, report locations, and exception translation
Installed artifacts Clean-wheel vfy test runs on every advertised platform and Python version
Existing projects Pinned suites run natively. Collection, outcomes, diagnostics, coverage, and requested reports are compared. Exclusions are recorded.
Failure handling Earlier coverage and reports are kept where promised, resources are released, and the documented exit status is returned. No stock retry.
First use Clean install, then the documented adoption example. Configured options are honored or rejected explicitly.
Performance Parity-gated measurements of the exact candidate artifact, using the committed harness. All samples, workload, versions, and environment are kept.

On the candidate, install a wheel without Rust. Through the public CLI, run a file, node ID, parametrized case, and -k selection. Check successful and failed runs, assertion details, source locations, exit status, help, and quiet output. Record configuration changes, rejected plug-ins, diagnostic differences, and real-project exclusions. Keep user feedback separate from automated results. Each claimed workflow needs public-boundary proof: the corpus comparison covers outer sessions, uses stock pytest for nested pytester projects, and does not check complete captured-output parity. See the adoption scope for unsupported workflows and plug-ins.

For an edit-loop claim, measure one test, one file, affected-test selection, and larger suites. Measure watch from saved edit to displayed result, including debounce, collection, and reporting. Separate first invocation from warm-cache runs, and execution time from total command latency. Compare outcomes and coverage with a fresh full run before interpreting timings. Use the committed benchmark harness; correctness gates alone do not establish performance.

Find comparison methods in Native pytest testing and publication steps in the release checklist.

Ecosystem comparisons

The scheduled ecosystem workflow (.github/workflows/ecosystem.yml) defines pinned attrs, click, and httpx smoke comparisons against Coverage.py and vfy. It compares normalized files, lines, arcs, contexts, and report totals. Its separate recorded-project matrix has these lanes:

Project Python Configured check
Rich 3.12, 3.13 Require the parity comparison to pass
Rich 3.14 Require a known schema-7 rejection
Requests 3.12, 3.13, 3.14 Require the parity comparison to pass
Datasette 3.12 Require a known report-generation or source-parsing rejection

The diagnostic lanes require a nonzero result matching their expected error; a passing diagnostic job does not establish successful parity. Jobs retain reports, databases, resolved dependencies, logs, and semantic differences, and publish no timings. This configured matrix is separate from the historical benchmarks; check the candidate's retained job results for execution evidence.

The workflow does not block releases. A case can become a required CI check after its expected behavior and comparison method are reviewed.

Executable documentation

tests/integration/test_documentation.py copies the source, tests, and config from examples/coverage/ into temporary projects. It runs the coverage, selection, mutation, Python API, and CI examples through VfyProject. test_documented_semantic.py checks the semantic guide’s missing-input example, including the condition vectors and failing and passing gates:

uv run pytest -q tests/integration/test_documentation.py tests/integration/test_documented_semantic.py

Mark an executable block with <!-- doc-test: name --> immediately before its fence. Names are unique within each page. Console blocks have one $ command per line. Tests substitute the installed interpreter for uv run vfy and uv run pytest, so wheel tests exercise the wheel under review. Python blocks are run or written as shown. CI recipes use inline run: commands. The extractor rejects multiline commands rather than skipping them. Job conditions and artifact transport still need a real Actions run.

Keep expected lines, branches, tests, and statuses independently reviewable. Unmarked API fragments are partial examples.

Documentation has two validation boundaries:

  • The strict Zensical build checks the pages under docs/: navigation, links, and anchors.
  • Repository entry points such as README.md, CONTRIBUTING.md, benchmark notes, and changelogs sit outside the site. They need a separate link review.

The build cannot catch clipped tables, unreadable diagrams, or an unbalanced feature grid. Check changed pages at desktop and mobile widths, in both color schemes. The required CI workflow runs these tests for documentation-only changes. Code changes run them in the integration and installed-wheel suites.

The contributor fixture under examples/coverage/ builds vfy from the containing checkout and locks its Python dependencies. The user tutorials use PyPI installation; this fixture validates the local source under review. Run uv lock --project examples/coverage after changing dependency requirements, then uv sync --locked from the example directory. Keep the example inside a reviewed checkout so its relative source path remains valid.

The separate installation smoke copies the fixture into temporary projects and points their vfy source at the checkout under review. It resolves that local source, installs a noneditable wheel in a fresh environment, verifies the recorded source directory and installed package and extension origins, and executes the actual uv run commands from the coverage, selection, mutation, and semantic guides:

uv run python scripts/check_tutorial_install.py

It checks 80% coverage, the failing threshold, 100% after adding the missing test, JSON/XML exports, affected selection, combined coverage, and the full-run fallback. It also checks a surviving mutation becomes protected after a stronger assertion, a missing input brings condition and MC/DC coverage to 100%, and upstream pytest-html and pytest-metadata remain active with vfy coverage. This contributor check requires Linux or macOS, Git, and the source-build prerequisites. The documentation CI job runs it separately from candidate-build tests; passing one does not establish the other.

Measuring Rust execution coverage

Measure Rust unit tests and Python integration tests together. The Python tests load a Rust extension built with -C instrument-coverage, including in child processes. Use a separate build directory, absolute LLVM_PROFILE_FILE paths containing %p and %m, and the LLVM tools that match the pinned Rust compiler. Include the Rust test executables and the extension. Exclude test sources and dependencies.

Compare unit-only and combined reports against the same source-map inventory. Review uncovered lines and regions by behavior, not against a percentage target. Region coverage is not branch coverage. Record the interpreter, platform, build flags, test results, and skipped cases with each measurement. Normal-exit profiles can omit counters from deliberately terminated processes. Measure each CPython version in the supported CI matrix, because collector paths are versioned.

tests/integration/test_collector_edge_cases.py covers branch-event retirement, non-LIFO suspensions, file tracers, cover_pylib, and encoding edges. tests/integration/test_collector_interleaving.py compares threaded calls, and generator instances resumed out of suspension order, with pinned Coverage.py. The same cases independently check semantic observations. Rust merge properties assert the expected context-partitioned union as well as the merge algebra. Semantic persistence tests compare complete stored records and their ordering.