Compatibility¶
vfy is Alpha. It supports CPython 3.12 and newer. The configured CI matrix covers 3.12–3.15 on Linux, and 3.12, 3.14, and 3.15 on macOS.
vfy's compatibility tests compare the documented interfaces below with Coverage.py 7.15.4. Compatibility with future upstream releases requires new comparisons. vfy does not need Coverage.py at runtime.
For adoption questions, see the FAQ. CPython 3.11 and older, and PyPy, are unsupported.
Versions¶
| Component | Supported range or target | Reference or configured validation |
|---|---|---|
| CPython | >=3.12 |
CI: Linux 3.12–3.15; macOS 3.12/3.14/3.15 |
| Coverage.py oracle | 7.x schema-7/public API | 7.15.4 |
| Optional pytest support | >=9.1; required by pytest-native/pytest engines |
Linux CPython 3.12 CI: 9.1.0 (floor), 9.1.1, 9.2.0; locked integration and differential oracle: 9.1.1 |
Native vfy test (default) |
Experimental; documented Linux/macOS support; pytest is not required | Recorded validation: Linux; pinned pytest 9.1.1 compatibility reference |
vfy test --engine=pytest-native |
Experimental; Linux and macOS; stable pytest >=9.1 |
Explicit compatibility route; worker and watch boundary tests |
| pluggy | >=1.5,<2 with vfy[pytest] |
CI floor: 1.5.0 with pytest 9.1.0; locked reference: 1.6.0 |
| Hypothesis | Optional; final 6.x releases from 6.152.1 for independent execution | Locked reference: 6.170.5 |
| pytest-xdist | Optional; native execution supports 3.8.0 | Direct pytest minimum lane: 3.6.1; native scheduler, worker, report, and coverage differentials: 3.8.0. With pytest 9.2.0, xdist 3.8.0 loses the group suffix used by --dist=loadgroup |
| pytest-cov | >=7.1,<8 (integration) |
7.1.0 |
| pytest-html | 4.2.0 (integration) |
4.2.0 |
| pytest-metadata | 3.1.1 (integration) |
3.1.1 |
| pytest-asyncio | 1.4.0, 1.4.1, 1.4.1.post1 (independent adapter admission) |
1.4.0 and 1.4.1.post1 differentials |
| pytest-mock | Optional, >=3.13,<4 for native execution |
Scoped fixtures, patch cleanup, and assertion diagnostics compared with stock pytest |
| pytest-randomly | Optional, 5.0.0 for native execution |
Original collection and setup, call, and teardown reseeding hooks; requires the installed upstream module |
| pytest-timeout | Optional, >=2.4,<2.5 for native execution; signal method in the main thread; session timeouts require --workers=1 |
Marker, configuration, command-line, environment, func_only, and serial session timeouts compared with stock pytest; reference 2.4.0 |
| pytest-benchmark | Optional, >=5.3,<5.4 for native execution; measures with --workers=1 |
Fixture, pedantic, marker options, disable, skip and only modes, JSON output, and skipped-test cleanup compared with stock pytest; reference 5.3.1 |
| pytest-codspeed | Optional, 5.0.3 or 5.0.4 for independent execution; inactive fixtures and markers | The independent adapter provides unmeasured fixtures and markers, including the package’s BenchmarkFixture import. Measurement, instrumentation, and other plugin internals require --engine=pytest; --codspeed and CODSPEED_ENV are rejected. |
| Independent replacements for absent timeout and benchmark plug-ins | POSIX main-thread signal timeouts, configuration and runtime marker overrides, func_only, and 0 to disable; unmeasured benchmark calls, group metadata, skip and only selection |
Signal timeout phases and configuration precedence are compared with pytest-timeout 2.4.0, including independent workers. Timers restore the previous signal handler and reject a foreign ITIMER_REAL deadline before mutation. A runtime call-only marker replaces an owned protocol timer; the protocol deadline does not resume after the call. Thread timers, session deadlines, and debugger override options require pytest-timeout. Benchmark calls and pedantic follow pytest-benchmark's disabled mode; explicit --benchmark-disable also permits warmup, sort, and grouping settings. Measurement, profiles, comparisons, and artifacts require the installed plug-in. CodSpeed measurement requires --engine=pytest. |
| Platformdirs | Optional, reviewed fixture module (4.12.0–4.12.3) | Fixture isolation and cleanup compared with stock pytest, including native and xdist workers |
Package metadata declares dependency ranges; the lockfile pins reference versions. A configured CI lane is not evidence that the current revision passed. Recorded results apply only to their stated wheel and revision.
Native pytest limits¶
vfy test --engine=pytest-native requires a stable pytest release from 9.1 onward. Installed pytest handles configuration, collection,
imports, and assertion rewriting on this route. Rust runs tests and fixtures.
See execution engines for the default
selection rule.
Independent affected-test selection supports unfiltered safe coverage runs
using a passing local baseline. Selection binds after project collection hooks;
a changed collected inventory runs the full requested scope. Selected tests
retain collection order. Portable baseline import/export, selection audits,
failure confirmation, and selection under other policies require the installed
pytest routes. See
selection rules and external inputs.
Independent assertion rewriting generates vfy-specific bytecode and temporary
names. Tools that identify pytest rewriting by inspecting its private bytecode
names may not recognize it. The independent pytest module has a public package
file and standard import metadata for consumers that inspect its origin.
The native and pytest-native engines drive test execution from Rust; Python stack inspection therefore does not find pytest execution frames
that would be present with --engine=pytest. For example, a stack-based ignore
rule for _pytest may behave differently. Use --engine=pytest when a tool
requires these pytest implementation details.
The real-package validation harness compares ordered JUnit outcomes, counts, and exit statuses.
Owned independent interpreters bind their native lifetime before activating
application import paths, .pth files, sitecustomize, or usercustomize.
They forward the caller's effective interpreter flags and retain its startup
environment, temporarily withholding PYTHONPATH and warning-category imports
until binding. Warning options resolve before site hooks, which can still change
their filters. The original
path and site policy also applies to nested workers. Optimization, warning
options, bytecode-cache settings, and fixed hash seeds follow that policy.
In owned interpreters, sys.flags.safe_path and sys.flags.no_site remain set
after application paths and site hooks are restored. An explicit caller -I or
-E continues to control environment handling.
The native test map and corpus gate describe the compatibility checks.
Native behavior and limits:
raiseerror()works with pytest-compatible exceptions. Parametrized JUnit cases keep their own properties, including values recorded during fixture teardown.record_xml_attributefollows the selected JUnit family's attribute rules. Independent execution supports ElementTree's registered namespace prefixes, including the default prefix. Malformed names leave existing JUnit XML intact. Saved attribute recorders follow pytest's reporter lifetime. Repeated collection reports for one node share a JUnit testcase and retain every error or skip event. Independent multiworker collection and pytest-xdist can differ in collection-skip counts.- Independent fixture requests expose the collected item's live
pathin function, class, module, and package scopes. Session scope raisesAttributeError. - Finalizers run in reverse registration order. With
--engine=pytest-native, a first-time fixture request during teardown follows the installed pytest version's rules. - Warning filters see marker changes made before the test starts. Failure output includes captured setup and teardown output.
--capture=fd,--capture=sys, and--capture=nowork.--capture=tee-sysis unsupported. Sys capture leaves direct file descriptor writes uncaptured. Disabling capture leaves output uncaptured.- Short summaries keep test IDs and outcome reasons. Native skip summaries list each test ID, not a source location.
- Independent execution requires runtime test reports to retain the active item’s node ID; rewriting report node IDs is unsupported.
- Independent parameter IDs preserve lone surrogates through pytest's default
escaping. With
disable_test_id_escaping_and_forfeit_all_rights_to_community_support = true, surrogate IDs raiseUnicodeEncodeErrorduring collection. - Custom collectors keep their function-generation methods. Package xunit hooks share pytest's owner lifetimes. Faulthandler timeouts and deferred thread exceptions take part in native lifecycles.
--engine=pytest-nativesupports plain conftest setup and teardown hooks and rejects phase hook wrappers.vfy diagnostics --engine=pytest-nativeidentifies literal wrapper declarations andspecnamealiases without importing project code.- The independent engine runs supported conftest collection hooks, including
generated parametrization and per-case deselection callbacks. Installed
pytest handles these hooks with
--engine=pytest-native. See the supported hook list. Hook specifications snapshot warning options at registration; their publicwarn_on_implandwarn_on_impl_argsfields are writable. - During the test protocol, the independent engine calls
pytest_assertrepr_compareimplementations visible to the current test and uses the first non-empty answer before its own rules. Config.inicfgis a read-only mapping of the configuration file's values and raises pytest's deprecation warning; assigning to it is not supported. The legacytmpdirpath object implements the commonly usedpy.path.localmethods, not the full class.
Native execution uses its own executor. Contexts for pytest and pluggy internals can therefore differ. When evaluating a suite, compare outcomes and coverage of the code you own.
Native worker and xdist options, fixture lifetimes, stepwise, and logging limits are documented under Workers.
The supported runtime plug-ins are listed below.
The optional pytest-mock adapter supports the upstream fixture and configuration hooks. See loading instructions. The compatibility script runs pytest-mock 3.13.0, 3.14.0, 3.15.0, 3.16.0, and 3.15.1 against stock pytest. The main differential oracle is 3.15.1.
vfy does not automatically fall back to pytest execution. See the adoption guide for supported projects and the test guide for commands.
Supported interfaces¶
Supported:
vfy coveragecommand workflows;- supported Coverage.py configuration, schema-7 data, shard merging, and contexts;
- include/omit file patterns with literal braces, component wildcards, recursive
directory wildcards, and invalid embedded
**diagnostics; - the documented
CoverageandCoverageDataPython API; - line and branch coverage reports in terminal text, Markdown, HTML, JSON, XML, LCOV, and annotated source;
- supported pytest-cov options under stock pytest (with vfy's coverage plug-in),
native
vfy test, and Python subprocess coverage; - Coverage.py's documented public plug-in protocols.
Not supported: private Coverage.py modules, the private dumps() byte format,
Python runtimes without sys.monitoring, hard process termination, and
undocumented third-party plug-in internals. Output is not byte-identical to
Coverage.py. Semantic decision coverage and mutation testing are vfy
extensions, described below.
Coverage.py data¶
| Behavior | Support |
|---|---|
| Read schema-7 line and branch data | Supported |
| Write schema-7 data that Coverage.py can read | Supported; direct writes preserve line and branch mode |
| Preserve contexts and file-tracer names | Supported |
| Reject tracer conflicts without changing destination data | Rejected before destination mutation; add_file_tracers() is atomic even though Coverage.py 7.15.4 can retain earlier batch entries |
| Merge compatible parallel shards | Supported |
| Reject mixed modes and unsupported schemas | Supported |
Ordered, existence-gated [paths] aliases |
Supported; Coverage.py differential target |
Schema 7 is an internal Coverage.py format, not a stable interchange standard. Supporting another Coverage.py release requires tests against that release. A matching schema number is not enough.
Collection¶
| Behavior | Support |
|---|---|
| Line and branch collection matching Coverage.py's default core | Targets Coverage.py’s default core |
| Multiline branch destinations and active snapshots | Supported |
| Source, include, and omit composition | Supported |
| Threads and interleaved generators | Supported |
[run] dynamic_context |
test_function and none; Coverage.py 7.15.4 differentials cover qualified names, switcher ordering, nesting, generator resumption, exceptions, and overlapping threads |
| Same-line loops, exhausted and early-closed generators | Supported |
| Exceptions | Supported |
| pytest-xdist normal completion | Supported |
| Spawned or exec'd Python subprocesses | Supported on normal shutdown |
Application sitecustomize in measured subprocesses |
Coverage.py 7.15.4 differential for hook identity and absence, single execution, usercustomize ordering, measured imports, startup errors, and unresolvable search entries |
| Fork without exec | Supported on POSIX, with isolated child shards |
| Retry collection after failed monitoring initialization | Supported, including forked children |
| Monitoring slot reuse after stopping | Clears code-local masks, including restored fork frames |
| Retry after failed source or plug-in discovery | Preserves collected measurements; Coverage.py discovers at get_data(), vfy at stop() |
[run] sigterm |
Saves, then exits normally for the signal, on POSIX |
[run] patch = _exit |
Saves before os._exit(), including stopped instances; save errors preserve the requested exit status |
[run] core and [run] disable_warnings |
sysmon, ctrace, and pytrace are accepted as tracer preferences; collection uses sys.monitoring with the branch-arc conventions Coverage.py uses for the requested core. COVERAGE_CORE is read when Coverage is constructed; set_option("run:core", ...) can override it before collection. Any warning names resolve like Coverage.py |
[run] concurrency = multiprocessing |
Coverage.py differential for fork, forkserver, and spawn children on POSIX; greenlet, eventlet, and gevent are rejected |
| SIGKILL and unpatched hard interpreter exits | Unrecoverable by design |
The package installs vfy, not a coverage command. See the
CLI reference for supported coverage
commands. check and mutate are vfy extensions.
pytest-cov¶
vfy's own plug-in implements the pytest-cov 7.1.0 options, default terminal
report, report destinations, cov and no_cover fixtures and marker, per-test
phase contexts, append and erase, threshold exit status, Python subprocesses, and
xdist data handoff.
vfy test loads this plug-in only in its child run, and disables competing
coverage plug-ins there. Ordinary pytest keeps normal entry-point discovery, so
pytest-cov and vfy can be installed together. See
pytest integration.
Analysis and reports¶
Line statements and exclusions target Coverage.py. Branch analysis covers functions, loops, comprehensions, conditional expressions,
match and guards, with, try, partial conditions, excluded branches, and
custom partial expressions. Branch arcs under unexcluded elif and else suites
stay reportable beneath an excluded if header. Cleanup branches in finally
retain explicit destinations inside the suite. Abrupt try and handler exits do
not give cleanup a fallthrough branch; an executable else suite can supply a
static continuation.
- For Python source, runner and saved-data reports use the file inventory recorded during collection, including unexecuted files that pass the filters.
exclude_linesreplaces the default patterns.exclude_alsokeeps them.- Excluding a function or overload does not apply the default ellipsis pattern to adjacent blank or comment lines.
- Source loading honors UTF-8 byte-order marks, encoding declarations, and universal newlines. Supported encodings: UTF-8, ASCII, Latin-1, and Windows-1252. Other codecs, such as Shift-JIS and ISO-8859 variants other than Latin-1, raise an explicit error.
| Format | Status |
|---|---|
| Terminal text | Totals, missing ranges, precision, threshold; display filters preserve totals and API percentages |
| Markdown and total API output | Display filters preserve totals and API percentages |
| JSON | Format 3 line and branch data, plus Rust function and class regions |
| Cobertura XML | Coverage.py-compatible line and branch fields, source roots, package depth |
| LCOV | Coverage.py-compatible line and branch fields, including sanitized filenames and zero-hit versus unknown branch arms |
| HTML and annotations | Coverage.py-compatible documented output; annotation directories distinguish sources with the same basename |
Keep run:relative_files the same during collection and reporting. If relative
data is reported with that option off, vfy keeps the recorded measurements.
Coverage.py's absolute-path lookup can report those lines as unexecuted.
Semantic decision coverage¶
Semantic coverage is an opt-in Alpha extension. Runs without it do not perform semantic analysis; compatible schema-7 readers ignore its additional tables.
| Surface | Contract |
|---|---|
| Collection | vfy coverage run --decisions, vfy test --cov-decisions, Coverage(decisions=True). Decision mode implies branches. |
| Decisions | Boolean control flow, plus named match case selection, exception dispatch, cancellation-aware async, and value selection |
| Evidence | Boolean vectors and masking MC/DC pairs, plus named outcomes attributed to contexts, keyed by source digest and structure |
| Persistence | vfy_* tables in schema-7 files, written atomically with ordinary coverage. Compatible Coverage.py readers ignore them. |
| Distributed runs | xdist, subprocess, and fork shards keep contexts. combine remaps paths, rejects conflicting source digests, and unions identical evidence atomically. |
| Reports | vfy coverage check (terminal and JSON); pytest terminal and JSON; independent decision, condition, and MC/DC thresholds |
| Plug-ins | Dynamic-context plug-ins work. File-tracer plug-ins are rejected in decision mode with an error. |
The report refuses evidence for a changed source digest, so the percentage always applies to the source that ran. Commands, truth tables, exclusions, and the JSON schema are in the semantic decision coverage guide.
Python API¶
The Python names that are public are listed in the
Python API; other vfy modules are
internal. The documented Coverage and CoverageData interfaces target Coverage.py
7.15.4, including constructor and public-method parameter names. Reports can use
active, unsaved data; updates validate before replacing data.
Empty line additions and purged line data retain their collection mode.
Touching a measured file rejects a conflicting plug-in tracer.
Distributions include py.typed and native extension declarations. Consumer
checks cover lazy exports, coverage data, reports, and async decorators.
Missing or malformed measured source raises NoSource or NotPython.
ignore_errors omits those files. If none remain, reporting raises NoDataError.
CoverageData.dumps() uses a vfy-private format. It is not Coverage.py's
serialized representation. It round-trips only through a matching vfy build,
including semantic evidence and contexts.
Plug-ins¶
Supported protocol areas: static and dynamic file tracers; line-range and
line/arc translation; custom file reporters and virtual source; dynamic contexts;
configurers; executable-file discovery; and callable and configuration-module
registration. Default plug-in methods are compared directly with Coverage.py
7.15.4.
Custom-reporter include and omit filters are compared with Coverage.py 7.15.4
through Coverage.report(), including reports with every source excluded.
Custom reporter branch totals use exit_counts(). Single-exit lines do not
count as branches. Custom reporters' code_regions() and arc_description() callbacks are not
consumed. Their LCOV output omits function records and uses numeric descriptions
for exit arcs.
Plug-in compatibility checks¶
The protocol fixtures exercise tracer registration, file and line translation, virtual reporters, dynamic contexts, configuration, and discovery. The third-party package checks compare saved coverage and reports against Coverage.py. Run them with your release wheel and test your own plug-in’s collection and reporting behavior.
Pytest result HTML and metadata¶
The coverage plug-in, vfy.pytest_plugin, leaves upstream pytest-html and
pytest-metadata enabled under stock pytest. Their existing hooks and fixtures
continue to use the upstream implementations.
To use vfy's own test-results HTML and metadata under stock pytest, explicitly
load -p vfy.pytest_reporting alongside the coverage plug-in. This reporting
plug-in replaces pytest-html and pytest-metadata: it blocks their automatic
loading and rejects earlier explicit loads. Native vfy test loads vfy's
reporter automatically.
vfy's reporting plug-in and native vfy test --html support a subset of pytest-html
4.2.0 and pytest-metadata 3.1.1 with pytest 9.1.1: metadata, redaction, CSS, extras
attachments, and the title, summary, and table-header hooks, including native
workers. generate_report_on_test publishes completed rows with one worker.
Parallel runs publish merged results after execution.
Base64 PNG attachments preserve decoded bytes in asset-backed reports and use
data URIs in self-contained reports; image content that is not valid base64
remains an external reference. Generated image asset names and extensions can
differ from pytest-html; use the references in the generated HTML. The image
helpers do not accept pytest-html’s extension= argument.
Not supported: per-test table-row hooks, upstream imports, internal objects, and interactive UI. The output is static HTML. See Pytest result HTML.
Configuration¶
The configuration reference defines the supported Coverage.py configuration. Unknown options fail with an error.
SlipCover¶
vfy does not provide the slipcover executable or import, [tool.slipcover],
JSON input merge, immediate de-instrumentation controls, or Python 3.9–3.11 and
PyPy instrumentation. These are separate interfaces, not aliases.
Test selection¶
Test selection and vfy explain are vfy extensions. Passing selected runs
combine fresh coverage with saved coverage for skipped tests; those tests are not
rerun. For eligible body-only edits, replay also retains global coverage outside
the edited bodies, including unchanged helpers called during collection of a
skipped module. --cov-append runs the full requested suite and appends to
existing data instead of replaying saved results.
See selection rules and limits, watch mode, and CI baseline transfer.
Coverage and JUnit destinations resolved through pytest addopts are staged along
with explicit report options. Publishing into an existing report directory preserves
unrelated files. Obsolete watch runs cannot replace accepted outputs.
Mutation test isolation¶
vfy mutate is a vfy extension. The default native engine requires Linux,
one worker, and serial isolation. With --engine=pytest-native, which follows the
native engine's Linux and macOS limits, a single-threaded POSIX process with
fork() uses file isolation by default: each test file gets its own fork of
the collected session, with session fixtures created and finalized per file. Changes to Python globals
do not carry between files. Choose --test-isolation serial for suites that
share session fixtures or global state across files.
An omitted isolation option selects serial when the caller has additional Python
threads or the host lacks POSIX fork(). Explicit --test-isolation file
requires a single-threaded POSIX process with fork(). The clean baseline and
mutation tests use the same isolation mode. See
mutation test isolation for the
shared worker budget and supported traceback options.
Compatibility evidence¶
Changes to this contract need a differential or upstream-produced fixture, an edge-case test, and documentation of the public behavior. The testing guide maps the checks to their suites.
Built-in asyncio¶
The native engine emits the unset loop-scope warning for explicit asyncio plugin requests or an enabled pytest-asyncio installation. Built-in support alone does not require that setting.
vfy targets pytest-asyncio 1.4.0's public API: strict and auto modes, async
fixtures, loop scopes and factories, event-loop policies, port fixtures, and
Hypothesis integration. The independent adapter admits installed versions 1.4.0,
1.4.1, and 1.4.1.post1. vfy does not ship pytest_asyncio or its private APIs. The async guide owns setup and behavior details;
release validation describes installed-wheel
checks. Direct-pytest coverage integration requires pytest-xdist 3.6.1 or newer;
upstream withdrew 3.6.0 for a
pytest-cov regression.
Native xdist execution targets 3.8.0. Performance has
separate gates.
The owned adapter acquires event-loop policies on demand. A pytest-asyncio compatibility alias activated first retains its autouse policy fixture.