Native pytest testing¶
Use these methods to compare an installed native engine with pinned pytest and real projects. Testing vfy lists suites and release requirements; the CLI reference lists supported options and plug-in restrictions.
vfy test defaults to independent execution and one worker. Use
--engine=pytest-native to exercise the installed-pytest compatibility route.
Internal differential checks use the hidden --engine=pytest override.
Default-route tests omit engine and worker overrides and verify independent
execution records.
Run focused tests¶
After setting up the development environment, run these commands from the repository root:
scripts/cargo-test.sh --locked -p vfy-pytest
cargo test --locked -p vfy-core
.venv/bin/python -m pytest -q \
tests/integration/test_native_*.py \
tests/integration/test_watch_native_workers.py \
tests/integration/test_watch.py::test_native_watch_discards_obsolete_evidence_and_publishes_recovery \
tests/integration/test_pytest_config_projection.py
Rust tests cover engine rules such as report ordering, JUnit identities, and
XML escaping. Python integration tests run temporary projects through
VfyProject and compare public behavior with pinned pytest.
On Linux with CPython 3.12, CI is configured to run each pytest version listed under release validation. A separate first pass uses the declared minimum pytest 8.0.0 and pluggy 1.5.0, checking version boundaries, protocol wrappers, warning filters, and warnings retained after control exceptions. The regular matrix restores pluggy 1.6.0. To reproduce the check in a disposable environment with vfy installed:
bash scripts/check_native_pytest_versions.sh
Find a test¶
Use the suite map to choose a layer. Native public
checks are tests/integration/test_native_*.py; watch and mutation suites have
separate entry points. Typed-plan tests live in vfy-core, and
test_native_dynamic_bindings.py checks bindings and pre-fixture failures
through VfyProject. Recorder and corpus-driver tests are under
tests/repository/.
Cache checks in test_native_cache_selection.py cover --lf, --ff, --nf,
--sw, --sw-skip, and --sw-reset, including stale state, collection errors,
and recovery. Keyboard interrupt preserves prior native cache files.
--cache-show and --cache-clear are unsupported.
Required adoption subset¶
The regular CI workflow checks pinned iniconfig, sortedcontainers, pluggy, colorama, and Rich projects on Linux CPython 3.12. See CI change classification for when this gate is required. It reuses the tier's Linux wheel and build manifest.
validation/pytest_runner/check_native_adoption_matrix.py --required-subset
compares stock pytest with explicit vfy test --engine=pytest-native using one and two
workers. All five projects require complete ordered case and collection
identities, matching setup/call/teardown outcomes and exceptions, exit statuses,
warnings, diagnostics, and captured output. Route checkouts use equal-length
paths so path widths do not alter rendered output. The checker binds the
candidate to the supplied wheel, build manifest, and current source revision
and tree, and refuses uncommitted source files. Ignored build and environment
directories remain permitted. Colorama covers its pytest-discoverable unittest
cases; the other projects use their pinned pytest suites.
The pinned Rich workload retains all 981 tests and applies the same reviewed patch to every outcome and coverage checkout:
- The broken-pipe test closes the reader before the demo starts, retains the direct demo precheck, and requires producer status 1.
- The auto-refresh test writes background frames to
StringIObecause Console capture is thread-local. The refresh thread and update loop run; main-thread capture must includeStep 9. - The recursive traceback test raises
RecursionErrorafter 20foocalls. Hidden-frame and truncation assertions remain, and captured output compares literally, including the hidden-frame count.
Preparation verifies the pinned revision and git apply --check. Each route
retains workload-patch.json with the patch path and hash and source hashes
before and after preparation.
Warning comparisons remove exactly one trailing formatter newline from native comparison copies when present; two or more trailing newlines remain literal. Raw records remain unchanged, and the results retain each applied rule and the literal differences. Warning order and multiplicity still have to match.
Rich has six reviewed rules for volatile call-phase stdout at revision
9d8f9a372cc5916fd4781fec207ced7ddac2f08f. They affect comparison copies only;
raw execution records and transcripts remain unchanged. The allowed spans are:
| Volatile value | Exact test scope |
|---|---|
| Render duration, two spans | test_console.py::test_brokenpipeerror |
| SWIG object address, one span | test_inspect.py::test_inspect_swig_edge_case |
| LRU wrapper address, one span | test_inspect.py::test_qualname_in_slots |
| Log clock label or equivalent padding, two spans | test_logging.py::test_exception |
| Log formatter timestamp, two spans | test_logging.py::test_exception |
| Log clock label, one span | test_logging.py::test_exception_with_extra_lines |
The rules check each span's surrounding text and occurrence count; missing or extra spans fail the comparison. Every other captured or diagnostic literal remains in the comparison. These adoption rules do not change the upstream corpus's literal accounting contract.
Coverage runs use stock pytest with vfy's plug-in and native execution with one and two workers. Every route must exit with status 0, report the complete ordered case inventory, match the stock coverage run's case outcomes, and retain nonempty schema-7 data and JSON reports. Pinned Coverage.py 7.15.4 reanalyzes each route's own data and must reproduce its normalized report. All three routes use vfy's collector: Coverage.py is a report-analysis oracle, not an independent collection oracle. For the pinned sortedcontainers workload, the harness adds an autouse fixture that seeds each of its three random stress tests to 0 and restores the previous random state. This gives each test the same inputs across worker partitions; the tracked upstream files remain unchanged. The harness records the fixture hash.
| Project | Coverage compared across routes |
|---|---|
| iniconfig, sortedcontainers, colorama | Exact files, lines, arcs, contexts, and normalized JSON reports |
| pluggy | Exact aggregate files, lines, arcs, and normalized JSON reports. Context differences remain in the artifacts: stock pytest itself executes the measured Pluggy framework, while native execution has different framework calls. |
| Rich | Matching source-file inventories and the same-run report check above. Thread scheduling and import or worker execution state can change hits and contexts, which remain in the artifacts. |
This untimed subset does not establish broad plug-in, platform, or performance compatibility.
CI retains route transcripts, reports, execution records, coverage data, and
failure diagnostics under target/native-adoption/, including on failure.
To reproduce from the repository root, supply a release wheel and its build
manifest created at build time, then run:
.venv/bin/python validation/pytest_runner/check_native_adoption_matrix.py \
--wheel dist/vfy-*.whl \
--build-manifest dist/adoption-build.json \
--output target/native-adoption \
--python-version 3.12 \
--required-subset
Pytest-mock adoption projects¶
The adoption matrix also includes two projects that explicitly load pytest-mock 3.15.1. These optional entries compare complete ordered execution records through stock pytest and native one- and two-worker routes. Their reviewed coverage contracts are:
| Project | Pinned revision | Scope and environment | Coverage contract |
|---|---|---|---|
| Poetry Core | b52c25c | Complete constraints, JSON, packages, pyproject, SPDX, utilities, VCS, version, core version, and factory unit suites. Build and integration suites are outside this entry. | Aggregate lines, arcs, tracers, and reports must match. Serial schema-7 data was also observed to match exactly. |
| pyproject-api 1.9.1 | 651b632 | Full suite; --cov-config=/dev/null isolates coverage from upstream covdefaults settings. The report oracle uses the same explicit configuration. |
Exact schema-7 data and reports. |
Poetry Core initializes its grammar and marker parser caches lazily in each worker. Parallel workers can attribute that initialization to different first-use tests. Its reviewed aggregate-only contract retains those context differences in the report while requiring identical measured lines, arcs, tracers, and normalized reports. Every route's report is also checked against Coverage.py using that route's own database.
Comparison copies normalize known checkout roots in parametrized test IDs and coverage context labels. Captured output also normalizes the requested pytest temporary roots, including each native worker's root. Raw execution records and coverage databases remain unchanged.
Use the wheel and build manifest described above:
.venv/bin/python validation/pytest_runner/check_native_adoption_matrix.py \
--wheel dist/vfy-*.whl \
--build-manifest dist/adoption-build.json \
--output target/native-adoption/mock-projects \
--python-version 3.12 \
--project poetry-core --project pyproject-api
Platformdirs and Requests also request pytest-mock in the matrix, but require other runtime plug-ins. Supporting mock fixtures does not establish adoption parity for those suites.
Poetry Core's builder tests share files in their source fixtures. Run its full configured suite with one worker; the automated entry above excludes those builders so its two-worker comparison does not race over the shared files.
Pytest-xdist adoption projects¶
The optional xdist matrix requests the same pytest-xdist 3.8.0 worker counts
(0 and 2) and --dist=loadfile on both routes. Stock pytest uses -n;
native vfy test uses pytest-xdist's --numprocesses. On the vfy CLI,
-n selects vfy's own file-group workers.
The pinned workloads are Requests-cache 1.3.3's complete upstream deployment
unit phase, Black's official no_jupyter phase, and Chardet 7.6.0's complete
configured suite with its pinned external test-data corpus.
The checker compares JUnit identities, outcomes, diagnostic bodies, properties, and captures. Coverage phases compare schema-7 data and normalized reports and check each route's report against Coverage.py 7.15.4. Raw warnings and logs are retained; this checker does not compare the corpus gate's full warning and phase records. Native routes also require Rust engine records with complete test phases and matching worker execution counts. A configured workload is not a claim that the current wheel passed it.
.venv/bin/python validation/pytest_runner/check_native_adoption_matrix.py \
--wheel dist/vfy-*.whl \
--build-manifest dist/adoption-build.json \
--output target/native-adoption/xdist \
--python-version 3.12 \
--xdist-project requests-cache --xdist-project black \
--xdist-workers 0 --xdist-workers 2
Chardet includes an unmarked timing assertion that failed in an instrumented
stock baseline. Use --xdist-project chardet --xdist-phase outcomes to compare its complete
test outcomes separately; retain any coverage-phase failure as distinct evidence.
Selecting --xdist-phase coverage also runs the outcomes baseline and requires
coverage to preserve its test inventory and outcomes on each route.
This optional matrix does not change the required five-project CI subset.
Installed-wheel CI¶
The native pytest workflow is configured for Linux and macOS with Python 3.12–3.15. It installs matching artifacts in separate reference and candidate environments, runs the integration suites and focused differential checkers, then compares the pinned upstream corpus.
The installed-wheel lanes discover every tests/integration/test_native_*.py
suite, including new fixture, reporting, worker, and version regressions.
Watch and mutation suites run separately through their explicit entry points.
The workflow also runs a pinned Rich project check with the candidate vfy
wheel. It executes the 981-case suite through pytest with vfy's pytest
plug-in, then through native vfy test with one and two workers. Every route
uses vfy's coverage collector. The checker compares ordered JUnit case
identities and outcomes across routes, then reanalyzes each route's own schema-7
data with pinned Coverage.py 7.15.4 and compares complete normalized per-file
coverage records and totals. Coverage.py is the report-analysis oracle, not an
independent collector: the check does not claim collection parity or require
threaded runs to observe identical branch hits. It does not compare diagnostic
transcripts or measure performance.
The check then applies a reviewed source-breaking edit in the disposable native
checkout. Native execution must report a real call-phase test failure and exit
with status 1. The checker restores the source, reruns the affected tests, and
requires status 0 and a clean checkout. This verifies the failure-and-repair
sequence through the installed native command. Each edit stage gets an isolated
Python bytecode cache so same-size, same-second edits cannot reuse stale .pyc
files. The check does not test import-cache invalidation.
Reproduce the Rich project check¶
From the repository root, prepare a candidate wheel at
target/native-pytest/dist/ and use the locked Rich dependencies:
uv venv --python 3.12 target/native-pytest/rich-environment
uv pip sync --python target/native-pytest/rich-environment/bin/python \
--require-hashes benchmarks/locks/rich.txt
uv pip install --no-deps \
--python target/native-pytest/rich-environment/bin/python \
target/native-pytest/dist/vfy-*.whl
uv run --no-sync python -c \
'from pathlib import Path; from benchmarks.prepare_projects import checkout_project; from benchmarks.project_manifest import manifest; checkout_project(manifest("rich"), Path("target/native-pytest/upstream-rich"))'
uv run --no-sync python validation/pytest_runner/check_native_project.py \
--upstream-root target/native-pytest/upstream-rich \
--python target/native-pytest/rich-environment/bin/python \
--vfy-wheel target/native-pytest/dist/vfy-*.whl \
--output target/native-pytest/results/native-rich
The checker verifies that the installed package matches the supplied wheel and
that the project checkout matches the pinned revision. It writes route records
and normalized reports under target/native-pytest/results/native-rich/; the
source edit is confined to a fresh checkout and restored before success. CI
runs this check in eight installed-wheel lanes: Ubuntu and macOS with Python
3.12, 3.13, 3.14, and 3.15.
tests/repository/test_native_project.py tests the checker itself. Other
checkers in validation/pytest_runner/ cover failure limits, warnings,
plug-in restrictions, reports, and installation. CI retains their results.
The separate ecosystem workflow compares coverage engines through stock pytest;
this Rich check applies only to the pinned suite and routes above.