CLI reference¶
The vfy command-line interface: global options, commands, options, defaults,
and exit statuses. Run uv run vfy COMMAND --help in an installed project
environment for command-specific help.
Watch mode, selection policies, CI baseline options, semantic checking, and mutation testing are Alpha.
vfy [GLOBAL_OPTIONS] COMMAND ...
| Global option | Meaning |
|---|---|
-V, --version |
Print vfy VERSION |
--config PATH |
Use this vfy configuration file |
--isolated |
Ignore configuration files and environment-selected config |
--color [auto\|always\|never] |
Colored output (default: auto). --color without a value means always. See environment variables for NO_COLOR and FORCE_COLOR. |
-v, --verbose |
Write the configuration source and effective execution settings to stderr. Repeating the option has no further effect. |
-q, --quiet |
Show only requested output; reports and errors still appear |
--quiet and --verbose cannot be combined. Verbose output does not change
reports or where they go. To control pytest's output, pass pytest arguments,
such as vfy --quiet test -q.
| Command | Use it to |
|---|---|
test |
Run tests with coverage by default, selecting affected tests for safe, unfiltered runs. |
explain |
List the tests that reach PATH:LINE, from saved coverage |
coverage |
Collect, report, and check coverage; manage saved data |
mutate |
Run mutation analysis |
config |
Show or validate resolved configuration |
diagnostics |
Print runtime and integration diagnostics |
cache |
Inspect or clean the project cache |
For config file selection, environment variables, and overrides, see configuration discovery and precedence.
vfy test¶
vfy test [--engine native|pytest-native|pytest]
[--data-file PATH] [--rcfile PATH] [--all] [--watch] [--no-replay]
[--history PATH] [(-n|-j|--workers) (N|auto|logical)]
[--source PATH]
[--native-scheduling file|scope|cases|striped|demand]
[--baseline PATH] [--export-baseline PATH] [--plan-output PATH]
[--plan-only] [--audit-selection] [--confirm-failures]
[TARGET ...] [PYTEST_ARGS ...]
vfy test runs tests with coverage. With --no-cov, it runs them without
collecting coverage. Such runs leave saved coverage untouched, skip coverage
database reads and replay preparation, and still record test outcomes and
durations.
The default native engine selects affected tests for unfiltered safe
coverage runs. Eligible function-body edits use
scoped collection. A test
inventory change found during collection triggers a full run. This engine also
supports --plan-only and
--plan-output. Other independent workflows, including
--no-cov, --cov-append, explicit selectors, watch, and other selection policies, run the full
requested scope. The pytest-native and pytest routes require vfy[pytest].
| Execution condition | Behavior |
|---|---|
Valid passing baseline for current HEAD |
Select tests affected by uncommitted changes using the configured policy (safe by default). A passing selected run merges fresh coverage; a failed selected run preserves the baseline. |
| No changes with a valid baseline | Independent execution runs the requested scope. The pytest-native and pytest routes restore complete coverage without running tests. |
--all |
Run every test in the requested scope; a passing clean-checkout run replaces the baseline. Uses safe selection regardless of configured policy and rejects pytest selection controls such as -k, -m, file/node targets, and PATH:LINE on the command line or in PYTEST_ADDOPTS. Static selectors in configured addopts define the suite instead. |
| No valid baseline or a safety fallback | Run the full requested scope and print a Note: with the reason. |
PATH:LINE target |
With pytest-native or pytest, run the fastest recorded test reaching the one-based source line; explain lists all matching tests. |
balanced or aggressive selection |
With pytest-native or pytest, select by changed line with fewer safeguards; either policy can skip affected tests. |
Selectors supplied through -o addopts=... also count as command-line controls.
A failed selected run prints Passing baseline preserved because the selected run
failed. An unchanged pytest-native or pytest run prints No code changed; complete
coverage restored without running pytest.
Safety fallbacks include --no-cov, pytest selectors or explicit file/node
ID targets, changed invocation or settings, unavailable Git
state, and untracked, deleted, renamed, test, or conftest.py changes. Changing
the interpreter optimization level (-O, -OO, or PYTHONOPTIMIZE) invalidates
both local replay and portable CI baselines. --cov-append runs the full
requested suite and preserves existing coverage. See the complete
fallback conditions
and selection policies.
| Test option | Default and effect |
|---|---|
--engine native\|pytest-native\|pytest |
Defaults to native, regardless of installed pytest. It supports affected-test selection for unfiltered safe coverage runs, plan inspection, selectors, history, watch, and vfy workers. --baseline, --export-baseline, --audit-selection, and --confirm-failures require explicit pytest-native or pytest execution and vfy[pytest]. vfy mutate accepts native (default) or pytest-native. Native mutations require Linux, one worker, and serial isolation. |
--data-file PATH, --rcfile PATH |
Data defaults to .coverage or COVERAGE_FILE. vfy settings come from --config or discovery; --rcfile names the pytest and Coverage.py configuration file for the run. |
--history PATH |
.vfy/test-history.sqlite3, unless configured; stores durations and outcomes. See history storage for SQLite and JSON choices. |
--source PATH |
Repeatable source roots for affected-test analysis. Uses test configuration, then coverage run source. |
--no-replay |
Disable baseline replay; incompatible with baseline import. |
Coverage-enabled vfy test supplies --cov-context=test unless explicitly
configured. Line targets and test reachability in explain need saved per-test contexts;
ordinary line coverage alone cannot identify test node IDs. Pairs of semantic observations
can still appear in explain without per-test contexts.
vfy test options and pytest options can be interleaved, with no -- separator.
Global vfy options such as --config go before test. The independent engine
reads configuration, collects tests, and rewrites asserts itself. The explicit
pytest-native and pytest routes use installed pytest for these operations.
The independent engine's --version or -V response and session header identify
the actual vfy version. Explicit pytest-native and pytest routes retain installed
pytest's version response and header. The independent pytest API exposes its
compatibility version separately for plug-ins.
Unsupported options fail before execution. See
Compatibility. vfy test returns pytest's exit
status.
Pytest 8.2+ argument files (@args.txt) use one argument per line. Nested
paths resolve from the invocation directory. Expansion is limited to 64 levels,
1,024 files, 8 MiB, and 100,000 arguments.
If a selected test ID no longer exists at collection, the run fails before test bodies start and keeps the passing baseline.
JUnit output replaces its target atomically. The target's parent directory must be writable, including when the target is a symlink.
Pytest coverage options¶
vfy test accepts these coverage controls through the selected engine.
Stock pytest uses the explicitly loaded vfy.pytest_plugin. See
collection for plug-in loading and runnable workflows.
vfy test --help summarizes common coverage options and shows an example.
With the optional vfy[pytest] extra installed, get stock pytest and coverage option help with
uv run python -m pytest -p no:pytest_cov -p vfy.pytest_plugin --help.
vfy supports the subset described in Compatibility.
| Option or fixture | What it does |
|---|---|
--cov[=SOURCE] |
Adds a path or an importable package or module to measure. Repeat it for more. --cov= with an empty value disables source filtering. |
--cov-reset |
Clears the sources added earlier on the command line |
--cov-report=TYPE[:DEST] |
Repeatable term, term-missing, annotate, html, xml, json, markdown, markdown-append, or lcov reports |
--cov-report= |
Saves coverage data without writing a report |
--cov-config=PATH |
Uses this Coverage.py-style configuration file (default: .coveragerc) |
--cov-branch |
Measures branch coverage |
--cov-append |
Appends to existing coverage data; disables selected-run replay |
--cov-decisions |
Collects decision, condition, and MC/DC data; implies branch coverage |
--cov-decision-report=TYPE[:DEST] |
term (default) or json[:PATH]; JSON defaults to vfy-decisions.json |
--cov-fail-under-decisions=MIN, --cov-fail-under-conditions=MIN, --cov-fail-under-mcdc=MIN |
Independent semantic percentage gates |
--cov-fail-under=MIN |
Makes pytest exit with status 1 if coverage is below MIN |
--cov-precision=N |
Sets rounding for the threshold and the number of decimals in the terminal report |
--cov-context=test |
Labels each test's nodeid|setup, nodeid|run, and nodeid|teardown phases |
--no-cov-on-fail |
Skips the report when tests fail |
--no-cov |
Turns off coverage for a run that has it configured |
cov fixture |
Gives you the active vfy.Coverage object |
no_cover fixture |
Leaves the test's call phase out of coverage |
@pytest.mark.no_cover |
Leaves the marked test's call phase out of coverage |
Without --cov-report, vfy writes the terminal report. term and term-missing
accept :skip-covered. File reports accept a destination after :, and Markdown goes
to coverage.md by default. markdown is a vfy addition. Machine-readable formats
can differ from pytest-cov in the ways listed in
Compatibility.
Workers¶
-n, -j, and --workers set the number of test workers. Each accepts an
integer from 1 to 256, auto, or logical. Execution defaults to one worker,
except with --engine=pytest, which uses automatic parallelism when pytest-xdist is installed.
-n 0 runs serially. The native engine requires pytest-xdist for this mode;
the independent engine accepts it without pytest-xdist. These aliases also work
after pytest targets and flags.
Each worker has its own fixture lifetimes. With --engine=pytest-native, runs collect
once in the controller and fork their workers when they can, including runs with coverage and the
default --capture=fd. Other runs start fresh workers (see
native worker start).
With --engine=pytest-native and pytest-xdist 3.8.0 installed, pytest's
--numprocesses and --dist use
upstream xdist scheduling, with native execution inside the workers. Combining
xdist with native file-group capacity above one is rejected.
--stepwise requires one native worker and does not load xdist.
--native-scheduling chooses how the shared Rust scheduler assigns tests to
native or independent workers:
| Value | Assignment |
|---|---|
file (native default) |
Each file runs in one worker. Use this when tests in a file share state. |
scope |
Each test class, and each file's module-level functions, runs in one worker. Matches xdist loadscope. |
demand (native parallel default) |
Workers claim another case batch as they finish. |
cases |
Fixed contiguous case ranges. Can split one parametrized family across workers. |
striped |
Cases interleaved across workers. Each worker keeps collection order. |
Native capacity is limited to 256 workers and capped by file groups or cases, depending on the scheduler. Unknown test durations use the upper-middle recorded duration; without history, tests have equal weight.
File and scope scheduling balance workers by the durations in --history.
Each worker initializes and finalizes its own fixtures. demand, cases, and striped
can initialize a file's module, class, package, or session fixtures in several workers;
use file when tests share module state. Independent workers collect in fresh
interpreters before assignment. Every requested independent worker starts
together; a worker left without tests finishes a collect-only session. Serial
independent runs retain collection order. The optional pytest-native engine requires
a collected fork for demand scheduling.
These scheduling options cannot be used with xdist. Native file-group execution with file or live CLI logging requires one worker, and automatic worker selection runs serially for these options. See Workers for capacity and process behavior.
Watch mode¶
--watch runs immediately, then starts a fresh test process after source changes
settle. An obsolete run keeps the last accepted coverage and reports. A failed or
obsolete run needs a full recovery run before selection resumes. --all --watch
always runs the full scope. Watch rejects node IDs, PATH:LINE targets, and the
balanced and aggressive policies. Ctrl-C exits with status 2. See
Rerun on save.
CI selection controls¶
| Option | Behavior |
|---|---|
--confirm-failures |
Rerun failed tests in a fresh serial process. The primary result, coverage, and reports are kept. |
--export-baseline PATH |
Export a passing full run as a portable CI baseline. Requires --all. |
--baseline PATH |
Import a compatible ancestor baseline and select affected tests. An incompatible input causes a full run. |
--plan-output PATH |
Save the finalized selection plan. |
--plan-only |
Build and finalize a plan without running tests. Writes to stdout unless --plan-output is set. |
--audit-selection |
Run the full scope and check whether the plan skipped a failing test. |
--plan-only and --plan-output also work with the independent engine without
installed pytest. The other options require explicit --engine=pytest-native or
--engine=pytest execution with vfy[pytest] installed. These controls support only the
safe policy and cannot be combined with --watch. --audit-selection and
--plan-only are mutually exclusive. Baseline import cannot be combined with
--all or --no-replay. --export-baseline cannot be combined with
--baseline or --plan-only. --cov-append is not supported. See
CI selection.
vfy explain¶
vfy explain [--data-file PATH] [--history PATH] [--json] PATH:LINE
Lists tests that reached a line in saved per-test coverage, without running tests.
Collect test reachability with uv run vfy test --all (which supplies
--cov-context=test). Pairs of semantic observations can be reported without test
contexts. It reports reachability and
recorded durations, not dependency paths.
PATH:LINE is a source path and a one-based line number, such as
src/parser.py:42.
Output starts with Tests reaching PATH:LINE: N. Then come the estimated
durations of the selected and full suites, when recorded, and one line per test,
fastest first, with its recorded duration. The selected estimate describes the
fastest reaching test; the full estimate covers all matching tests. If the line
is part of a decision
recorded with --cov-decisions, a MC/DC observation pairs: N pair(s) block follows,
with one condition N: line per pair of observations. Condition numbers start
at one. For example, the bounds tutorial
records this pair for x >= 0 before the upper-bound test is added:
MC/DC observation pairs: 1 pair(s)
condition 1: test_bounds.py::test_below_lower_bound|run (0-) -> test_bounds.py::test_inside_bounds|run (11)
Context labels can also be application labels rather than test IDs.
--json writes an object with location, tests (each with nodeid and
duration_ms), selection (reasons, affected_files, affected_functions,
and affected_branches), and semantic_evidence. The text output omits
selection.
--data-file PATH: coverage data to read (default:.coverage, or$COVERAGE_FILE). No data file or matching shards is an error.--history PATH: test history with recorded durations (default: thehistorykey under[tool.vfy.test], which defaults to.vfy/test-history.sqlite3).--json: write JSON instead of text.
Loading saved coverage combines discovered shards into the selected data file and removes consumed shards after a successful replacement. It can therefore change saved data even though it runs no tests.
If the line has neither recorded test reachability nor pairs of semantic
observations, the command fails with
no historical test reaches PATH:LINE. See
How vfy selects tests.
vfy coverage¶
vfy coverage run [OPTIONS] TARGET [ARGUMENTS...]
vfy coverage report [OPTIONS]
vfy coverage html [OPTIONS]
vfy coverage json [OPTIONS]
vfy coverage xml [OPTIONS]
vfy coverage lcov [OPTIONS]
vfy coverage annotate [OPTIONS]
vfy coverage combine [OPTIONS] [PATH...]
vfy coverage erase [OPTIONS]
vfy coverage debug data [OPTIONS]
vfy coverage check [--strict] [OPTIONS]
These commands use Coverage.py-compatible data, configuration, output formats,
schema-7 files, path aliases, and report thresholds. Semantic checking is a
vfy extension. vfy does not install or shadow a coverage executable or
package.
Output goes to the command's output option, the configured destination, or the
command's default. - means stdout. Diagnostics and confirmations go to stderr.
coverage run accepts -a/--append, --branch, --decisions,
-p/--parallel-mode, -m/--module, repeatable --source, --include, and
--omit, --context, --data-file, and --rcfile. --decisions implies
--branch. --append loads existing data before the target runs.
--parallel-mode writes a shard. The exit status is the target's.
| Command | Options in addition to --data-file and --rcfile |
|---|---|
report |
--include, --omit, --contexts, --precision, --fail-under, --ignore-errors, -m/--show-missing, --skip-covered/--no-skip-covered, --skip-empty/--no-skip-empty, --sort=COLUMN, --format text / markdown / total |
json |
--include, --omit, --contexts, --precision, --fail-under, --ignore-errors, -o/--output, --pretty-print, --show-contexts |
xml, lcov |
--include, --omit, --contexts, --precision, --fail-under, --ignore-errors, -o/--output |
html |
--include, --omit, --contexts, --precision, --fail-under, --ignore-errors, -d/--directory, --title, --skip-covered/--no-skip-covered, --skip-empty/--no-skip-empty |
annotate |
--include, --omit, --contexts, --precision, --ignore-errors, -d/--directory |
combine |
--keep, --strict, and optional shard files or directories |
erase |
--parallel to also remove discovered shards |
debug data |
No additional options. Prints path, schema (always 7), has_arcs, files (the file count), and the line count of each measured file. |
Report commands load saved coverage by combining discovered shards into the
selected data file and removing consumed shards after successful replacement.
report writes to stdout. combine is transactional and honors configured
[paths] aliases. erase removes the selected data file and, with --parallel,
its shards.
report --skip-covered and --skip-empty hide file rows while preserving
the aggregate total and --fail-under result in each output format.
Sort columns are name, stmts, miss, branch, brpart, and cover.
Use --sort=-cover to list the highest coverage first.
--fail-under compares the total rounded to --precision. At the default
precision of zero, 79.66% passes --fail-under=80. A threshold of 100 requires
every line and branch to be covered.
vfy coverage check¶
vfy coverage check [--strict] [OPTIONS]
Checks decision, atomic-condition, and masking MC/DC evidence from a
decision-enabled run. It accepts --data-file, plus
repeatable --include, --omit, --contexts, --source, and
--exclude-decisions; --format text|json, -o/--output, --pretty-print, and
--history; and the independent thresholds --fail-under-decisions,
--fail-under-conditions, and --fail-under-mcdc.
--history defaults to .vfy/test-history.sqlite3; [tool.vfy.test] history does not apply.
--strict sets each threshold you do not specify to 100%. Explicit thresholds
take precedence. Missing or stale data exits with status 1. An unmet threshold
exits with status 2. The command does not run tests or change source code.
vfy mutate¶
vfy mutate [OPTIONS] [PATH:LINE] [PYTEST_ARGS ...]
Mutation testing applies one supported operation at a time to changed Python
files by default. It first checks that the baseline passes, then runs tests
against each replacement. Callable-body mutations use recorded test reachability.
Import-time mutations run the full requested pytest scope in fresh processes.
Results are saved as they finish by atomically replacing .vfy/mutations.json.
The checkout is never modified.
| Option | Default and effect |
|---|---|
--changed, --all, --base REF |
Changed files by default; choose one scope. --base selects current files changed since the merge base. |
--engine native\|pytest-native |
native by default; pytest-native requires vfy[pytest]. |
--source PATH, --operator NAME |
Repeatable source roots or operator filters. |
--history PATH |
.vfy/test-history.sqlite3, unless configured. |
--max-mutations N |
0 means no limit. |
--timeout SECONDS |
Positive finite mutation timeout limit in seconds; 30 unless configured. The adaptive policy can lower the effective limit; see the timeout calculation. |
--timeout-policy adaptive\|fixed |
adaptive (default) reduces collected-mutation limits using recorded clean-test durations. fixed uses the configured timeout directly. |
-j, --workers N\|auto\|logical |
Independent execution defaults to 1 and requires one worker. Explicit --engine=pytest-native defaults to auto unless configured. |
--test-isolation serial\|file |
Independent execution uses serial and rejects file. With explicit --engine=pytest-native, defaults to file for a single-threaded POSIX process with fork(); otherwise serial. file forks test files under the shared worker limit. The collected backend admits up to three fresh pytest supervisors, each collecting under one slot of that limit, reserving an outer scheduling position for body mutations where capacity permits; see file isolation. |
--no-cache |
Skip the mutation cache. |
--format text\|json, -o/--output PATH, --pretty-print |
Text to stdout by default; - also selects stdout. |
--fail-on-survivor |
Exit 2 when a mutation survives. |
--data-file PATH, --rcfile PATH |
Select coverage data and the pytest and Coverage.py configuration file passed to test runs. vfy settings come from --config. |
An optional PATH:LINE narrows the catalog. Remaining arguments go to pytest.
Fresh-process mutations run the selected engine in a direct child; timeout and
interruption cleanup covers its process group and registered test-file groups. See
file isolation for fixture boundaries
and the shared worker budget.
File isolation creates session fixtures separately for each test file; changes
to Python globals do not carry between files. Select --test-isolation serial
when files need shared session state. An explicit --test-isolation file
requires a single-threaded POSIX process with fork(); only an omitted option
selects serial automatically when these conditions are unavailable. Baseline
and mutation tests use the same isolation mode.
Isolation is resolved at command entry. Threads first started during project
collection can cause file execution to fail its fork-safety checks; they do not
change the isolation mode midrun. Use --test-isolation serial for such suites.
File and node targets, -k, and -m on the command line restrict both the
passing baseline and mutation tests. A restricted baseline does not become a full-suite passing
baseline for vfy test; changing the pytest scope invalidates mutation-cache
results for the earlier scope. --all selects source files, so it can be
combined with these pytest selectors:
vfy mutate --all --source src -q tests/test_sign.py::test_zero_is_nonnegative
--base REF selects whole current Python files changed since the merge base of
REF and HEAD. That includes committed PR changes and staged, unstaged, and
untracked files. It implies --changed and rejects --all and PATH:LINE. A
missing reference or missing common history fails before baseline collection, so
fetch the base history first. JSON scope metadata records both resolved commits
and the selected files. Without --base, changed scope compares against HEAD.
See PR mutation testing.
vfy config¶
vfy config show
vfy config validate
show prints JSON with source (the file used, or null), settings, and
provenance, which records how the file was found. If no [tool.vfy] table
exists, the source is the Coverage.py file found by fallback. validate checks
types and supported values without running tests. It covers selection modes,
workers, positive finite timeouts, timeout policies, check formats, and
percentage ranges.
vfy diagnostics¶
vfy diagnostics [--engine=native|pytest-native] [--json] [--collect] [-- PYTEST_ARGS ...]
Reports the Python version and platform, the Ruff parser and profile, pytest and
xdist availability, pytest-cov coexistence, the configuration source, cache
state, and Rust-build diagnostics. The default native engine works
without installed pytest. Diagnostics resolves the selected engine's test
configuration and arguments without importing project code. Plug-in and hook
support remains unknown until collection runs. JSON records the selected
engine and its test_readiness.
--engine=pytest-native inspects the optional installed-pytest route. It also checks
statically visible plug-in declarations and runtime hooks in conftest files.
Without importing project code, the scan identifies literal wrapper=True and
hookwrapper=True on setup and teardown hooks, including specname aliases.
The installed plug-in inventory shows the native load plan: loaded by an
adapter or because every hook in its module runs unchanged, replaced by a vfy
adapter, or omitted. This is a plan, not proof that
collection or execution succeeded. Native startup also names installed plug-ins
it omits compared with pytest autoloading.
Findings from nested conftest files come from a project-wide scan and may not
apply to the selected test paths. Xdist options and hooks are accepted with
pytest-xdist 3.8.0. Other versions get an installation diagnostic. --json
output is suitable for bug reports.
--collect runs collection with the selected engine in a child process.
Test modules and collection hooks run; test bodies do not. The output
reports the exact collected item count. Collection does not prove that tests
pass, so run vfy test with the same engine to check fixtures and outcomes. Pass
invocation-specific pytest options after -- so diagnostics can inspect them.
Native diagnostics also retains native_readiness in JSON for that route.
vfy cache¶
vfy cache dir
vfy cache clean
dir prints the .vfy path in the current directory. clean removes that directory.
Files outside it, such as the project-root .coverage file and Coverage.py
shards, are not changed. Cleanup rejects a symlink or directory junction at
.vfy and does not follow links inside it. A missing cache is a successful
no-op. Filesystem errors are reported. A partly completed deletion cannot be
rolled back.
Exit statuses¶
| Status | Meaning |
|---|---|
0 |
Success |
1 |
Configuration, data, or filesystem error |
2 |
Argument parsing error, failed coverage or survivor gate, or Ctrl-C in watch mode |
4 |
Test configuration or usage error, including a rejected explicit fork request (see native worker start) |
vfy test uses pytest-compatible exit statuses. Status 4 indicates a test
configuration or usage error from the selected engine. vfy coverage run
returns the target program's exit status.