Skip to content

CI and quality gates

Run the full pytest suite, fail the job below a coverage threshold, and save reports for review. vfy replaces pytest-cov; pytest remains the runner.

Both coverage workflows on this page run from the root of the coverage example, after you add its negative-value test. In that project, run uv add --dev 'vfy[pytest]' to install vfy and pytest, then commit pyproject.toml, uv.lock, the source, and the tests. Supported runners install a prebuilt wheel. Copy the full-suite workflow below into .github/workflows/coverage.yml.

To run the same steps on your machine:

uv run pytest -p no:pytest_cov -p vfy.pytest_plugin --cov=src --cov-branch --cov-report=
uv run vfy coverage report --show-missing --fail-under 100

Run the full suite

name: coverage
on: [push, pull_request]
permissions:
  contents: read
jobs:
  coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
        with:
          python-version: "3.12"
      - name: Install
        run: uv sync --locked
      - name: Collect
        run: uv run pytest -p no:pytest_cov -p vfy.pytest_plugin --cov=src --cov-branch --cov-report=
      - name: Export XML
        if: always()
        run: uv run vfy coverage xml -o coverage.xml
      - name: Enforce threshold
        if: always()
        run: uv run vfy coverage report --show-missing --fail-under 100
      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        if: always()
        with:
          name: coverage
          include-hidden-files: true
          if-no-files-found: warn
          path: |
            .coverage
            coverage.xml

The collection step runs every test. The later steps write coverage.xml, check the 100% coverage threshold, and upload the data. They run even after a test failure; a passing coverage gate does not clear that failure.

If collection fails before writing data, XML export and the threshold check also fail. The upload step warns if no artifact exists.

Split tests across two jobs

This recipe uses the independent engine. First check your suite and compare its outcomes with pytest. Use separate jobs when one runner cannot finish the suite in time. Each job writes a separate coverage file (a shard); a final job combines the shards before checking coverage.

name: coverage
on: [push, pull_request]
permissions:
  contents: read
jobs:
  tests:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        include:
          - id: bounds
            test-file: tests/test_bounds.py
          - id: formatting
            test-file: tests/test_formatting.py
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
        with:
          python-version: "3.12"
      - run: uv sync --locked
      - name: Collect one test file
        env:
          TEST_FILE: ${{ matrix.test-file }}
          COVERAGE_SHARD: .coverage.${{ matrix.id }}
        run: uv run vfy test --data-file "$COVERAGE_SHARD" --cov=src --cov-branch --cov-report= "$TEST_FILE"
      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        if: always()
        with:
          name: coverage-${{ matrix.id }}
          path: .coverage.*
          include-hidden-files: true
          if-no-files-found: error

  report:
    needs: tests
    if: ${{ !cancelled() }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
        with:
          python-version: "3.12"
      - run: uv sync --locked
      - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
        with:
          pattern: coverage-*
          path: shards
          merge-multiple: true
      - name: Combine
        run: uv run vfy coverage combine --strict --keep shards
      - name: Export XML
        if: always()
        run: uv run vfy coverage xml -o coverage.xml
      - name: Enforce threshold
        if: always()
        run: uv run vfy coverage report --show-missing --fail-under 100
      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        if: always()
        with:
          name: combined-coverage
          include-hidden-files: true
          if-no-files-found: warn
          path: |
            .coverage
            coverage.xml
            shards/.coverage.*

Each test-file job writes one .coverage.* shard, and the report job combines them. Neither test job checks the threshold, because a shard holds only part of the data. Make the test jobs and the report job required checks in branch protection. A failed test job still fails the workflow.

--keep keeps the shards after combining, so you can inspect them. All jobs must share the same OS, checkout, and source paths. Don't combine line and branch data, or data from different source revisions. If jobs use different checkout roots or operating systems, set up path aliases first.

Enforce semantic coverage thresholds

After checking compatibility, replace the collection step with this partial example to require decision, condition, and MC/DC (modified condition/decision) coverage. See semantic coverage for what each measure checks. The test run fails with status 1 for an unmet pytest coverage threshold. The separate check exits 2 for an unmet semantic threshold and 1 for missing or stale data.

      - name: Test and measure semantic coverage
        run: >-
          uv run vfy test --all --cov=src --cov-branch --cov-decisions
          --cov-report=term-missing
          --cov-decision-report=json:build/decisions.json
          --cov-fail-under=90 --cov-fail-under-decisions=95
          --cov-fail-under-conditions=90 --cov-fail-under-mcdc=85

      - name: Verify semantic report
        if: always()
        run: >-
          uv run vfy coverage check --format text
          --fail-under-decisions 95
          --fail-under-conditions 90
          --fail-under-mcdc 85

vfy coverage check can also run in a separate job with the uploaded .coverage file and the same source checkout. It reads decision observations from .coverage; build/decisions.json is a report for review. For the model and exit statuses, see semantic coverage.

Schedule mutation testing

Add these steps to a PR or scheduled workflow. Use checkout with fetch-depth: 0, and replace main with the PR target branch. Mutation testing does not edit the checkout; --fail-on-survivor fails the PR step with status 2 when a mutation survives.

      - name: Fetch the mutation comparison branch
        if: github.event_name == 'pull_request'
        run: git fetch origin main
      - name: Collect a passing mutation baseline
        run: >-
          uv run vfy test --all --source src --cov=src
          --cov-context=test --cov-report=
      - name: Mutate PR changes
        if: github.event_name == 'pull_request'
        run: >-
          uv run vfy mutate --base origin/main --fail-on-survivor
          --format json --output build/mutations.json

      # Run this in a nightly or release-review workflow, not on every edit.
      - name: Mutate the complete source tree
        if: github.event_name == 'schedule'
        run: >-
          uv run vfy mutate --all --source src
          --format json --output build/mutations.json

The default native engine runs mutations serially on Linux. To run them in parallel, install vfy[pytest] and add --engine=pytest-native -j auto --tb=native to the vfy mutate commands. Add --engine=pytest-native --tb=native to the vfy test baseline step as well so both commands use the same traceback settings.

The full-suite run in the same job is the passing baseline. Upload build/mutations.json to review survivors and incomplete results later. For result statuses and limits, see Mutation testing.

--changed only sees uncommitted edits, so a clean PR checkout needs --base. The baseline still comes from the current checkout.

Select tests in CI

Install vfy[pytest] for native selection and baseline controls. Use vfy test --engine=pytest-native --all for a complete CI run. Without --all, a restored .vfy/ cache can select a subset of tests, or reuse saved coverage without running any tests. Use selection only when you intend to skip unaffected tests.

Transfer a passing baseline between checkouts with --export-baseline and --baseline. A baseline is the saved per-test coverage from a passing full run. Export it from a clean checkout:

uv run vfy test --engine=pytest-native --all --cov=src --cov-context=test --export-baseline baseline.json

Download that file into the checkout you're testing:

uv run vfy test --engine=pytest-native --cov=src --cov-context=test --baseline baseline.json --plan-output plan.json

Keep the source roots, coverage settings, and dependencies the same. The baseline must come from an ancestor of the commit under test. A common setup exports it from each passing run on your main branch, stores it as a CI artifact, and downloads it in pull request jobs. Don't copy .vfy/ between jobs.

A valid but incompatible baseline makes the run execute the full suite. A malformed file fails the job before any test runs or any output is replaced.

--plan-only writes the plan without running tests and also works with --engine=native without installed pytest; --plan-output saves that plan. Portable baseline controls and --audit-selection require an installed pytest route. --audit-selection runs the full scope and checks whether the selection missed any failures. For all the options, see CI selection controls. For limits that come from outside the code, such as external services, see test selection.

Keep artifacts when a job fails

The coverage recipes above upload .coverage, coverage.xml, and any shards with if: always(), so the files survive a failed test step. Give any other upload step the same condition.

For reproducible performance checks, see the benchmark guide.