GitHub Actions

The project uses several workflows using GitHub Actions to maintain code quality and confirm that the package and website are building correctly. The actions are defined in the .github/workflows directory and currently include:

Continuous integration workflow

The ci.yml workflow runs when a pull request is opened and when new commits are made to an existing pull request. It is the main quality assurance check on new code and runs three jobs:

• code quality assurance (qa): does the code pass all the pre-commit checks.

• code testing (test): do all unit and integration tests in the pytest suite pass.

• documentation building (docs_build): does the documentation build correctly.

If any of those checks fail, you will need to push new commits to the pull request to fix the outstanding issues. The status of code checking for pull requests can be seen at:

ImperialCollegeLondon/virtual_ecosystem

Although GitHub Actions automates these steps for any pushes, pull requests and releases on the repository, you should also perform the same steps locally before submitting code to ensure that your code passes testing. The pre-commit test is automatic but follow the instructions for running pytest and building the documentation.

CI workflow details

name: Test and build

# When does this run - new, reopened or updated PRs, pushes to main or develop and when
# the workflow is called by another workflow, such as the publishing actions.
on:
  pull_request:
    types: [opened, synchronize, reopened]
  push:
    branches: [main, develop]
  workflow_call:

jobs:
  qa:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
        with:
          python-version: "3.12"
      - uses: pre-commit/action@v3.0.1

  test:
    needs: qa
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ ubuntu-latest, macos-latest, windows-latest ]
        python-version: ["3.12", "3.13", "3.14"]

    steps:
    - uses: actions/checkout@v6

    - uses: actions/setup-python@v6
      with:
        python-version: ${{ matrix.python-version }}

    - name: Install Poetry
      run: |
        pipx install poetry --python python${{ matrix.python-version }}
        poetry --version

    - name: Install dependencies
      run: poetry install

    - name: Run unit tests
      run: poetry run pytest --cov-report xml -m "not integration"

    - name: Archive log for error checking on failure
      if: failure()
      uses: actions/upload-artifact@v7
      with:
        name: failed_tests_ve_run_cli_log
        path: ${{ runner.temp }}/log_file.log
        archive: false
        retention-days: 1

    - name: Upload coverage to Codecov
      if: >- 
          ${{ ! (
             github.event.pull_request.user.login == 'dependabot[bot]' || 
             github.event.pull_request.user.login == 'pre-commit-ci[bot]'
            ) && (
             matrix.os == 'ubuntu-latest' && 
             matrix.python-version == '3.12'
            )
          }}
      uses: codecov/codecov-action@v6
      with:
        fail_ci_if_error: true
        token: ${{ secrets.CODECOV_TOKEN }}
        verbose: true
  docs_build:
    needs: qa
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
        with:
          python-version: "3.12"

      - name: Install Poetry
        run: |
          pipx install poetry --python python3.12
          poetry --version
      
      - name: Install dependencies
        run: poetry install

      - name: Build docs using sphinx
        run: |
          cd docs
          poetry run sphinx-build -W --keep-going source build
      
      - name: Archive built docs for error checking on failure
        if: failure()
        uses: actions/upload-artifact@v7
        with:
          name: built-docs
          path: docs/build
          retention-days: 2

Publication workflow

The publish.yaml workflow runs when a release is made on the GitHub site and uses trusted publishing to build the package and publish it on PyPI.

The full workflow setup can be seen below, along with comments, but the basic flow is:

1. When a GitHub release is published, the PyPI publication workflow is triggered.

2. The standard continuous integration tests are run again, just to be sure!

3. If the tests pass, the package is built and the wheel and source code are stored as job artefacts.

4. The built files are automatically added to the release assets.

5. The built files are then also published to the Test PyPI server, which is configured to automatically trust publications from this GitHub repository.

6. As long as all the steps above succeed, the built files are now published to the main PyPI site, which is also configured to trust publications from the repository.

The last step of publication to the main PyPI site can be skipped by including the text test-pypi-only in the title text for the release. This allows pre-release tests and experimentation to be tested without automatically adding them to the official released versions.

Publication workflow details

name: Publishing

on: 
  release:
    types: [published]

jobs:
  # First, run the standard CI checks - for this to work correctly, the workflow needs
  # to inherit the organisation secrets used to authenticate to CodeCov.
  # https://github.com/actions/runner/issues/1413
  ci_checks:
    uses: ./.github/workflows/ci.yml
    secrets: inherit
  
  # Then if the standard CI checks pass run the integration tests
  integration_tests:
    needs: ci_checks
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        # All supported OSs, minimum and maximum python version
        os: [ ubuntu-latest, macos-latest, windows-latest ]
        python-version: ["3.12", "3.14"]
    
    steps:
    - uses: actions/checkout@v6

    - uses: actions/setup-python@v6
      with:
        python-version: ${{ matrix.python-version }}

    - name: Install Poetry
      run: |
        pipx install poetry --python python${{ matrix.python-version }}
        poetry --version

    - name: Install dependencies
      run: poetry install

    - name: Run integration tests
      run: poetry run pytest -m "integration"

    - name: Archive log for error checking on failure
      if: failure()
      uses: actions/upload-artifact@v7
      with:
        name: failed_tests_ve_run_cli_log
        path: ${{ runner.temp }}/log_file.log
        archive: false
        retention-days: 1

  # Next, build the package wheel and source releases and add them to the release assets
  build-wheel:
    needs: integration_tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6

      # Build the package - this could use `poetry build` directly but pyproject.toml 
      # already has the build-system configured to use poetry so `pip` should pick that
      # up automatically.
      - name: Build sdist
        run: |
          python -m pip install --upgrade build
          python -m build

      # Upload the build outputs as job artifacts - these will be two files with x.y.z
      # version numbers:
      # - virtual_ecosystem-x.y.z-py3-none-any.whl
      # - virtual_ecosystem-x.y.z.tar.gz
      - uses: actions/upload-artifact@v7
        with:
          path: dist/virtual_ecosystem*

      # Add the built files to the release assets, alongside the repo archives 
      # automatically added by GitHub. These files should then match exactly to the
      # published files on PyPI.
      - uses: softprops/action-gh-release@v3
        with:
          files: dist/virtual_ecosystem*

  # Now attempt to publish the package to the TestPyPI site, where the virtual_ecosystem
  # project has been configured to allow trusted publishing from this repo and workflow.
  #
  # The skip-existing option allows the publication step to pass even when the release
  # files already exists on PyPI. That suggests something has gone wrong with the
  # release or the build file staging and the release should not be allowed to continue
  # to publish on PyPI.

  publish-TestPyPI:
    needs: build-wheel
    name: Publish virtual_ecosystem to TestPyPI
    runs-on: ubuntu-latest
    permissions:
      id-token: write

    steps:
      # Download the built package files from the job artifacts
      - name: Download sdist artifact
        uses: actions/download-artifact@v8
        with:
          name: artifact
          path: dist
      
      # Information step to show the contents of the job artifacts
      - name: Display structure of downloaded files
        run: ls -R dist
      
      # Use trusted publishing to release the files downloaded into dist to TestPyPI
      - name: Publish package distributions to TestPyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          repository-url: https://test.pypi.org/legacy/
          verbose: true
          # skip-existing: true

  # The final job in the workflow is to publish to the real PyPI as long as the release
  # name does not contain the tag 'test-pypi-only'
  publish-PyPI:
    if: ${{ ! contains(github.event.release.name, 'test-pypi-only')}}
    needs: publish-TestPyPI
    name: Publish virtual_ecosystem to PyPI
    runs-on: ubuntu-latest
    permissions:
      id-token: write

    steps:
      # Download the built package files from the job artifacts
      - name: Download sdist artifact
        uses: actions/download-artifact@v8
        with:
          name: artifact
          path: dist
      
      # Information step to show the contents of the job artifacts
      - name: Display structure of downloaded files
        run: ls -R dist
      
      # Use trusted publishing to release the files downloaded into dist to PyPI
      - name: Publish package distributions to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1

Updates to pre-commit

The Virtual Ecosystem repository is registered with the pre-commit.ci service. This runs and reports the status of the pre-commit suite - which duplicates the ci.yml workflow above - but also adds weekly update checks on the pre-commit hooks used for the project.