Continuous Integration with Travis CI

You did it. You have covered your code with tests and now you can trust you code. Actually, you can't. One day, you will forget to run the tests, deploy to production and get an angry call from your boss later.

This needn't happen if you use contrinuous integration. In this tutorial you will learn how to setup Nette Tester with Travis CI.

How Travis CI Works

Travis CI (“Travis”) is a very popular service for hosted continuous integration. The service is free of charge and offers automated execution of your tests after pushing to your repository. It is also fully integrated with GitHub.

In order to configure Travis, you need to add a file named .travis.yml to the root of your repository. The file consists of sections described later in this tutorial. None of the sections is mandatory, however if we leave out the script section, Travis will attempt to run phpunit.

The sections are executed in following order:

  • before_install
  • install
  • before_script
  • script (required)
  • after_success or after_failure
  • after_script

.travis.yml allows space indentation only. You can use validator to check your file.

For more see official documentation.

Setting Up .travis.yml

Language

First of all, you need to tell Travis which language environment to select for your project. You can do so using language: php option. You can specify for which PHP versions will the tests be executed. Not introducing patch version tells Travis to use the latest available.

language: php

php:
  - 7.1
  - 7.2
  - 7.3
  - 7.4
  - nightly  # development version

Environment Variables

You can instruct Travis to do multiple runs with different sets of environment variables values. To do so, add env key. Each bullet is understood as a different environment and tests are run seperately. We are going to use TESTER_PHP_BIN later in the file to run tester with -p php-cgi option too.

env:
  - TESTER_PHP_BIN="php"
  - TESTER_PHP_BIN="php-cgi"

Combination of five php versions and two environment variations generates total of 10 runs.

Dependency Installation

This tutorial assumes that you use Composer for managing your project's dependencies and that you require nette/tester in dev section. For more information see tutorial on Composer.

In order to install your dependencies, use install section. Each bullet means single command. Composer installs your dev dependencies by default. You should use --no-interaction so composer doesn't ask questions Travis can't answer. You should also use --prefer-source. It prevents your tests from randomly failing if you run out of limited amount of GitHub API requests.

If you want to use the latest build, update Composer in before_install section.

before_install:
  - composer self-update

install:
  - composer install --no-interaction --prefer-source

Build Matrix

Depending on the configuration above, a build matrix is generated. The matrix contains all combinations of environment settings. A single combination is called a job and is run separately. You can modify the matrix in matrix section.

If you want to exclude a job, use exclude key. In our case, we don't want to use php-cgi binary for older PHP versions.

matrix:
  exclude:
    - php: 7.0
      env: TESTER_PHP_BIN="php-cgi"

    - php: 7.1
      env: TESTER_PHP_BIN="php-cgi"

Travis shows the build as passing only if every single job pass. However, you can define jobs that are allowed to fail without causing the whole build to shown as failed. To do so, declare allow_failures. For our sake, we allow nightly PHP to fail.

matrix:
  allow_failures:
    - php: nightly

Note that we have only listed the PHP version. Travis behaves in a way that does not include all combinations with the nightly version in the total result. Now if tests fail on nightly but pass in other environments, the whole build will be marked as passing as well.

Running the Tests

Tests are run in script section, where you only need to execute Tester. Let's assume your tests are in tests/ folder and you provide your own php.ini in the same folder. Additionaly, tell Tester to display information about skipped tests with -s option and to use value of earlier declared TESTER_PHP_BIN as PHP binary with -p option.

script:
  - ./vendor/bin/tester -p $TESTER_PHP_BIN -s -c ./tests/php.ini ./tests

If Test Fails

Section after_failure is executed if a test fails. Tester stores actual value of variables in case of assertion fail. We will use this section to print the actual values.

after_failure:
  # Prints *.actual files content
  - for i in $(find ./tests -name \*.actual); do echo "--- $i"; cat $i; echo; echo; done

Setting Up Other Services:

Travis comes with multiple popular services (e.g. MySQL) pre-installed. However, if you need to use for example Redis storage, you can tell Travis in services section.

services:
  - redis-server

Database Initialization:

MySQL runs on 127.0.0.1 and you can log in using travis or root as username. Password is not required. You can import your database in before_script section. Let's say your database set-up script is in tests/testbase.sql.

before_script:
  - mysql -u root -e 'CREATE DATABASE testbase;'
  - mysql -u root testbase < tests/testbase.sql

Result

The .travis.yml should look something like this by now:

language: php

php:
  - 7.1
  - 7.2
  - 7.3
  - 7.4
  - nightly

env:
  - TESTER_PHP_BIN="php"
  - TESTER_PHP_BIN="php-cgi"

matrix:
  allow_failures:
    - php: nightly

  exclude:
    - php: 7.0
      env: TESTER_PHP_BIN="php-cgi"

    - php: 7.1
      env: TESTER_PHP_BIN="php-cgi"

services:
  - redis-server

before_install:
  - composer self-update

install:
  - composer install --no-interaction --prefer-source

before_script:
  - mysql -u root -e 'CREATE DATABASE testbase;'
  - mysql -u root testbase < tests/testbase.sql

script:
  - ./vendor/bin/tester -p $TESTER_PHP_BIN -c ./tests/php.ini -s ./tests/

after_failure:
  # Prints *.actual files content
  - for i in $(find ./tests -name \*.actual); do echo "--- $i"; cat $i; echo; echo; done

GitHub Integration

As mentioned above, Travis is integrated with GitHub. However, you need to specify which repositories are to be tested. This is done using Webhook which notifies Travis about changes in your repository.

Activating Webhook

First, go to Travis CI and sign in with your GitHub account. After synchronizing your account, you`ll see all repositories you have access to. Flip switch to ON for all repositories you'd like to enable.

Travis will now add your repository to queue after every pushed commit or created pull-request. After a short while, your repository will be tested.

Status Image

Travis can generate a status image for you. You can embed this icon into your README.md file for example.

Skipping Commit

Some commits don't need testing. You can add [skip ci] somewhere in your commit message and Travis will ingore the commit.