Skip to content

Benchmarks

Measure vfy with the benchmark harness. Each workload defines the commands and the outputs it compares.

How to read benchmark evidence

Check the engine, Python version, machine, selected tests, coverage settings and worker count before comparing times. The reporting workload compares saved coverage against Coverage.py before sampling. Selection benchmarks compare rerun outcomes and reconstructed coverage with a full run. Mutation benchmarks verify the common mutations and the tests that detect them.

Use release wheels and pin the workload revision and dependency lock. Record all samples from both tools, with their execution order, warmups, median and spread. Elapsed time includes only the stages named in the workload. A process peak and the memory used by an entire worker tree measure different things.

Run a comparison

Follow the setup and report recipe to check parity and collect samples. The native benchmark recipes cover installed-wheel test execution.

README chart workloads and methodology

The chart compares each tool with vfy on the same workload. Time is median elapsed command time. Memory is the median sampled peak proportional set size across the process tree. Each pair starts at zero and uses its own time and memory scale.

Package Workload Comparison
Packaging Full suite of 62,435 tests across four workers vfy with file scheduling and pytest-xdist
Rich Both tools select the same 1 of 981 tests after an edit to Emoji.__repr__ vfy and pytest-testmon
flake8-simplify 69 shared mutations against 226 tests with one mutation worker vfy and mutmut

The Linux ARM64 runs used CPython 3.12.14 and the vfy 0.0.1 release wheel. The Linux container shared its host with the local test suite during sampling. All three vfy commands use --engine=pytest-native: pytest collects the tests and vfy executes them. Packaging checks JUnit outcomes before timing and again for each measured run. The mutation comparison checks the shared mutation outcomes before timing and throughout sampling.

For Rich, each pair starts from fresh full-suite histories and the same edit to rich/emoji.py. The timer covers the affected rerun, including JUnit output from both tools and vfy’s reconstructed coverage report. Full-suite baseline creation is outside the timer. The harness compares selected test outcomes with a full pytest-cov run and checks vfy’s JSON report and aggregate coverage database after the edit. Per-test context attribution differs from the full run. The Rich lock pins pytest-testmon 2.2.0, pytest-cov 7.1.0 and Coverage.py 7.16.2.

Measured medians

Package Tool Time (s) Peak tree PSS (MiB)
Packaging vfy 11.52 583
Packaging pytest-xdist 20.77 1642
Rich vfy 0.39 58
Rich pytest-testmon 0.75 43
flake8-simplify vfy 1.47 69
flake8-simplify mutmut 27.63 136

Each tool ran three warmups. Packaging and Rich contributed 15 measured pairs each and flake8-simplify contributed 30. The order alternated within each workload. The paired samples, confidence intervals, build identity and upstream revisions accompany the chart.

Reproduce the comparisons

Build the release wheel and record the imported extension with the native benchmark setup. Run these commands from the vfy checkout on Linux. Set the paths to your wheel, its build manifest, the upstream checkouts and fresh output directories.

python validation/pytest_runner/benchmark_native_xdist.py \
  --wheel "$VFY_WHEEL" --build-manifest "$VFY_BENCHMARK_BUILD_MANIFEST" \
  --project packaging --python-version 3.12 --workers 4 --native-workers 4 \
  --native-scheduling file --timing-route stock-xdist-4 --timing-route pytest-native-j4 \
  --capture=no --override-upstream-capture --tree-resources \
  --work-root "$PACKAGING_WORK" --output "$PACKAGING_RESULTS" \
  --mode publication --warmups 3 --runs 15

For Rich, prepare the pinned checkout and its locked environment, then run the affected-test comparison. The selection protocol describes the edit and the full-suite checks.

python -m benchmarks prepare projects rich "$RICH_WORK" --python python3.12
python -m benchmarks.benchmark_real_selection --project rich --engine pytest-native \
  --root "$RICH_WORK/project" --python "$RICH_WORK/environment/bin/python" \
  --preparation-manifest "$RICH_WORK/artifacts/manifest.json" --tree-resources \
  --artifacts "$RICH_RESULTS/artifacts" --output "$RICH_RESULTS/result.json" \
  --mode publication --warmups 3 --runs 15

For flake8-simplify, use the resolved dependency lock and install the project from the pinned checkout so its tests can read the distribution metadata. Install mutmut from its pinned checkout, the vfy wheel and the locked dependencies in the same environment. The mutmut checkout is 41353df71526c8309e77521b9342a539dd434479. The mutation setup describes the required inputs.

"$BENCH_PYTHON" -m benchmarks.benchmark_adopter_mutation \
  --root "$FLAKE8_SIMPLIFY" --revision d02de77a453dfcb81b67a0239d76fa1be610e658 \
  --expected-tests 226 --vfy-engine pytest-native --workers 1 \
  --mutmut "$MUTMUT" --mutmut-repository "$MUTMUT_CHECKOUT" \
  --lock "$MUTATION_LOCK" --output "$MUTATION_RESULTS" --warmups 3 --runs 30

Run each comparison with --validate-only and a separate output directory to check correctness before collecting samples.