Skip to content

Check your pytest suite

Use this guide to check a suite’s configuration, collection, fixtures, and plug-ins before switching CI to vfy test.

What is supported

On Linux and macOS, vfy’s default native engine runs pytest-style tests without installed pytest. Check the compatibility matrix for the test features and plug-ins your suite uses. The optional --engine=pytest-native route uses installed pytest for collection and vfy for execution.

Check your suite

Run from the project root. Add --engine=pytest-native to inspect the native compatibility route instead of the default native engine:

uv run vfy diagnostics
uv run vfy diagnostics --collect -- tests

Diagnostics reports visible configuration, plug-in, and hook conflicts with locations and suggested fixes. It lists which installed plug-ins vfy loads and which it omits. The static scan can include nested conftest.py files outside the chosen test path; --collect checks that path by importing modules and running collection hooks. It does not run test bodies or validate fixture behavior and outcomes. A clean diagnostic result is not a compatibility guarantee. Pass run options after --.

Compare one suite before you change CI

Use the same environment and services for both full runs:

uv run pytest tests -q
uv run vfy test --engine=native --no-cov tests -q

tests limits both commands to that directory. --no-cov runs every collected test without coverage. Do not combine --all with a path or other pytest selector. To check the native compatibility route, install vfy[pytest] and add --engine=pytest-native to the second command.

Compare collection, outcomes, exit status, skips, xfails, warnings, fixture effects, failure messages, and teardown. Reduce differences to a small case and check Compatibility. For coverage, reports, workers, and the edit loop, use Run tests with vfy.

Check plug-in behavior with a test that needs it. The native and pytest-native engines admit different plug-ins, so run vfy diagnostics --collect and the comparison with the same engine. Pass --engine=pytest-native to check pytest collection with vfy execution. For example, supported Hypothesis releases provide --hypothesis-profile and --hypothesis-seed under the independent engine. The pytest-native engine omits Hypothesis's pytest plug-in, so @given tests can run there without those options or the plug-in's checks. Matching passing tests alone does not establish that a plug-in's safeguards are active. See the version and plug-in scope.

Keep test dependencies installed for this comparison. For example, pytest-regressions supplies the data_regression fixture; removing it leaves tests that request the fixture unable to run. A collector plug-in can have a less visible effect: pytest-mypy-plugins collects YAML test cases, so removing it can reduce the collected count without failing the remaining tests. pytest-mypy-plugins requires pytest as the runner.

If --engine=pytest-native rejects a custom pytest.Item or a project pytest_runtest_call hook, compare the default native engine before changing the suite. It supports public custom item execution and call hooks; the optional native route has a narrower item and phase-hook scope. Keep the same test dependencies and compare the collected count and outcomes with stock pytest. If a required plug-in is still omitted or needs pytest internals, use pytest with vfy coverage as shown below. Troubleshooting covers inherited options and missing fixtures.

If your suite depends on AnyIO or another unsupported runtime plug-in, keep pytest as the runner and use vfy for coverage:

uv run pytest -p no:pytest_cov -p vfy.pytest_plugin --cov=src tests

Replace src and tests with your source and test paths. This keeps pytest's installed plug-ins active. Native affected-test selection and mutation testing require a suite whose execution plug-ins vfy supports.

Load supported plug-ins

Native vfy provides coverage, asyncio, and test-results HTML support. It disables general installed plug-in autoloading and loads its supported adapters. It also loads an installed plug-in when a static scan of its module finds only fixtures and hooks that native execution runs unchanged, such as pytest-datadir. The scan follows hook reexports within the same package. Unresolved imports, import cycles, and hook decorators or names that cannot be resolved statically leave a plug-in omitted. Explicitly loading an unsupported runtime plug-in is rejected. vfy does not fall back to pytest. An explicit -p pytest_cov, including one in configured addopts, is replaced by vfy's coverage adapter, and pytest's built-in plug-in names such as -p warnings are accepted. See Compatibility for versions and supported features.

Use pytest-xdist

Installed xdist loads when its options are requested. Explicit loads through -p xdist.plugin, PYTEST_PLUGINS, or a root conftest.py also work. Use --numprocesses for xdist workers; -n controls native workers. See distributed execution and coverage across processes.

Use pytest-mock fixtures

An installed supported pytest-mock autoloads. Disable it with -p no:pytest_mock, PYTEST_DISABLE_PLUGIN_AUTOLOAD, or --disable-plugin-autoload. Explicit -p pytest_mock or -p pytest_mock.plugin, PYTEST_PLUGINS, a root pytest_plugins declaration, and pytest addopts work. The plug-in must come from the interpreter's active installation locations, including enabled user site; a project-local or PYTHONPATH-only copy is rejected. Unsupported versions fail before collection.

The mocker, class_mocker, module_mocker, package_mocker, and session_mocker fixtures keep their scopes and undo patches at teardown, including failures and early stops. Patching, spying, stubbing, and assertion messages come from pytest-mock; each native worker has its own scoped fixtures.

Use pytest-randomly

Native execution loads installed pytest-randomly 5.0.0 and preserves its test ordering and reseeding between setup, call, and teardown. Use --randomly-seed=0 to reproduce an order. The original --randomly-dont-reset-seed and --randomly-dont-reorganize options also apply.

Use pytest-timeout

An installed supported pytest-timeout autoloads. Disable it with -p no:timeout or either autoload opt-out above. As under pytest, -p no:pytest_timeout leaves the timeout entry point enabled. Unsupported versions fail before collection.

Native execution runs pytest-timeout's own timer hooks, so --timeout, timeout in configuration, PYTEST_TIMEOUT, @pytest.mark.timeout, and func_only behave as under pytest, including in native workers. --session-timeout and session_timeout require --workers=1; native file workers reject them before collection because their session stop state is local to each worker. Only the signal method is supported: native output capture would discard the thread method's stack dump before it exits the process. vfy rejects --timeout-method=thread, timeout_method = thread, and markers that request it before the affected timer starts. Embedded pytest runs must start signal timers in the main thread, since pytest-timeout substitutes the thread method outside it.

Use pytest-benchmark

An installed pytest-benchmark 5.3 autoloads. Disable it with -p no:benchmark or either autoload opt-out above. Other versions are omitted, so their benchmark fixture is unavailable natively.

pytest-benchmark's own fixture, options, markers, and reports run unchanged: benchmark(...), benchmark.pedantic(...), @pytest.mark.benchmark, --benchmark-disable, --benchmark-skip, --benchmark-only, --benchmark-json, the results table, and conftest.py implementations of pytest-benchmark's hooks such as pytest_benchmark_update_json. vfy marks the fixture of a test that skips or fails as expected, as pytest-benchmark does from its report hook, so no unused-fixture warning follows. Native execution does not run the plug-in's check of @pytest.mark.benchmark arguments: unknown arguments still fail the test when the fixture is created, but positional arguments are ignored.

Measurement needs one process. As pytest-benchmark does under pytest-xdist, each native worker runs benchmarked functions once without timing them, and vfy prints a note when it used workers. Pass --workers=1 to measure.

Use Platformdirs fixtures

The installed Platformdirs pytest entry point autoloads platformdirs_isolated. Explicit -p platformdirs.pytest_plugin and a root pytest_plugins = ["platformdirs.pytest_plugin"] declaration also work. For a declaration, use vfy test -p no:platformdirs or either autoload opt-out above to prevent duplicate registration through the entry point. Admission checks the reviewed fixture module's contents: a later release with the same module works, but changed contents fail before collection. This adapter provides fixtures; other execution hooks are rejected.

Use pytester

Load pytest's pytester fixture with -p pytester or a root conftest.py declaration such as pytest_plugins = ["pytester"]. The helper plug-in pytester_assertions is also accepted. Nested pytester runs use stock pytest.

Setup and teardown hooks

The native runner supports plain pytest_runtest_setup and pytest_runtest_teardown hooks in conftest.py, including tryfirst and trylast ordering. Register them before tests start. Hooks registered later, and wrappers of these two hooks, are rejected.

pytest_runtest_protocol hooks from conftest.py and admitted plug-ins run around each native test in pluggy's order. Register them before execution starts. Wrappers, including legacy hookwrapper=True hooks, receive item and nextitem and the protocol's result. A plain hook that returns a result would replace pytest's test protocol, so native execution reports it instead of running the test. Wrappers that suppress a protocol exception require --workers=1; native parallel workers reject that recovery because it can omit assigned test reports. These hooks also run with xdist's disabled-worker setting, --numprocesses=0. Distributed xdist runs reject project and compatible third-party protocol hooks; use vfy's native file workers or run those workflows directly with pytest.

pytest still runs pytest_sessionstart and the supported collection hooks in conftest.py: pytest_collection, pytest_collect_file, pytest_ignore_collect, pytest_pycollect_makemodule, pytest_pycollect_makeitem, pytest_generate_tests, pytest_make_parametrize_id, pytest_itemcollected, pytest_deselected, pytest_collection_modifyitems, and pytest_collection_finish. The native runner builds its test list after that.

The test list is fixed after collection. Inserting, removing, or reordering session.items during the run is not supported, and vfy does not catch every such change. Changes to live test functions, argument values, fixture names a test requests, and skip or xfail markers take effect at the matching point in the run. Replacing pytest's fixture definitions during the run is not supported.