The case to use PCOV to generate test coverage results

• 2 min read

Test coverage is a metric which is often overvalued. Developers often assume they have to get to 100% coverage in their applications at all costs.
I think this is the wrong mindset. Not every single line of your application has to be tested. Writing Unit tests to test that a method getEmail() actually returns an email address is in my opinion, wasted time. (For packages, I agree that a high test coverage can be benefitial. It's usually also easier to do.)

However, knowing how much of your application is covered by tests can be a good indicator in which shape your project is. At work™, we recently started generating code coverage for all our projects.

In a particular large project (+1500 tests) the addition of running the test suite with xdebug enabled slowed the CI run down quite a bit. Normally the test suite takes 2 to 3 minutes to run. With xdebug enabled it took ~30 minutes. 😱

We're using GitHub Actions to run the test suite. In particular, we're using the shivammathur/setup-php-Action to create the correct PHP environment.
While figuring out, how we could speed up the CI run, I've stumbled upon the PCOV-option of the Action. PCOV is a driver built for generating code coverage. One of its main claims is, that it's much faster than xdebug. (Note: There will be a differerence in the coverage reporting between PCOV and xdebug (see here)).

Is it faster? Let's try it out.
These are the changes I had to make to our workflow. Instead of using coverage: xdebug we're now using coverage: pcov. In addition, I had to set ini-values: pcov.directory=app. This tells the driver where our application code lives.

name: Setup PHP
uses: shivammathur/setup-php@v2
with:
  php-version: 7.4
  extension: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
 coverage: xdebug
 coverage: pcov
 ini-values: pcov.directory=app

And this is the entire GitHub Actions workflow we're using right now.

name: Tests (with Coverage)

on: push

jobs:
    tests:
        runs-on: ubuntu-latest

        steps:
            - name: Checkout Code
              uses: actions/checkout@v2

            - name: Setup PHP
              uses: shivammathur/setup-php@v2
              with:
                  php-version: 7.4
                  extension: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
                  ini-values: pcov.directory=app
                  coverage: pcov

            - name: Install Composer Dependencies
              run: composer install --prefer-dist --no-interaction --no-suggest

            - name: Copy Environment File
              run: cp .env.example .env

            - name: Generate Application Key
              run: php artisan key:generate

            - name: Execute PHPUnit Tests
              run: vendor/bin/phpunit --coverage-text --coverage-clover=coverage.xml

We've pushed the change in the mentioned work project and the time for running the test suite dropped significantly. From ~30 minutes to the ususal 2 to 3 minutes!

I was so happy to report this to the rest of the team. And I think it's also amazing, how low effort the change was. Actions like setup-php makes working with and tweaking Continous Integration workflows on GitHub so much easier.

Webmentions What are webmentions?

    Likes and more

    Replies

    • Photo of Laravel News Links Laravel News Links mentioned

      Improve runtime when generating code coverage with PCOV stefanzweifel.io/posts/the-case…

      April 9th, 2020

    • Photo of Stefan Zweifel Stefan Zweifel replied

      Currently it's just display in the Actions log. Still evaluating all the different tools… For my personal projects I'm using codecov.io. Pretty easy to integrate: github.com/stefanzweifel/…

      March 30th, 2020

    • Photo of Regis Freyd Regis Freyd replied

      Thanks a lot! And how do you use the code coverage.xml file after this? An external tool? Is the result displayed in the PR too?

      March 29th, 2020

    • Photo of Stefan Zweifel Stefan Zweifel replied

      Yes, I'am aware. Forgot to mention that in the post. Update the post to mention that and link to the docs about that. Thanks 🙂

      March 29th, 2020

    • Photo of Stefan Zweifel Stefan Zweifel replied

      The same thought went through my head too. I should ship more 😅

      March 29th, 2020

    • Photo of Chuck Tomkress Chuck Tomkress replied

      You probably already know but just a heads up that you can get slight differences in coverage % when running it against the same test suite.

      March 29th, 2020

    • Photo of Tom Witkowski Tom Witkowski replied

      Next thing I have to add to all OSS projects ... I want to write real code & features. Don't show me how I can improve the CI all the time. 😂

      March 29th, 2020