Continuous Integration with Travis CI

You did it. You have covered your code with tests and now you can trust you code. Actually, you can't. One day, you will forget to run the tests, deploy to production and get an angry call from your boss later.

This needn't happen if you use contrinuous integration. In this tutorial you will learn how to setup Nette Tester with Travis CI.

  • configuring testing environment
  • installing dependencies using Composer
  • running tests using Nette Tester
  • setting up other services
  • TL;DR
  • GitHub integration

How Travis CI works

Travis CI (“Travis”) is a very popular service for hosted continuous integration. The service is free of charge and offers automated execution of your tests after pushing to your repository. It is also fully integrated with GitHub.

In order to configure Travis, you need to add a file named .travis.yml to the root of your repository. The file consists of sections described later in this tutorial. None of the sections is mandatory, however if we leave out the script section, Travis will attempt to run phpunit.

The sections are executed in following order:

  • before_install
  • install
  • before_script
  • script (required)
  • after_success or after_failure
  • after_script

.travis.yml allows space indentation only. You can use validator to check your file.

For more see official documentation.

Setting up .travis.yml

Language

First of all, you need to tell Travis which language environment to select for your project. You can do so using language: php option. You can specify for which PHP versions will the tests be executed. Not introducing patch version tells Travis to use the latest available.

language: php

php:
  - 5.3.3
  - 5.4
  - 5.5
  - 5.6
  - hhvm

Environment variables

You can instruct Travis to do multiple runs with different sets of environment variables values. To do so, add env key. Each bullet is understood as a different environment and tests are run seperately. We are going to use TESTER_PHP_BIN later in the file to run tester with -p hhvm option in HHVM environment.

env:
  - TESTER_PHP_BIN="php-cgi"
  - TESTER_PHP_BIN="hhvm"

Combination of five php versions and two environment variations generates total of 10 runs.

Dependency installation

This tutorial assumes that you use Composer for managing your project's dependencies and that you require nette/tester in dev section. For more information see tutorial on Composer.

In order to install your dependencies, use install section. Each bullet means single command. Composer installs your dev dependencies by default. You should use --no-interaction so composer doesn't ask questions Travis can't answer. You should also use --prefer-source. It prevents your tests from randomly failing if you run out of limited amount of GitHub API requests.

If you want to use the latest build, update Composer in before_install section.

before_install:
  - composer self-update

install:
  - composer install --no-interaction --prefer-source

Build matrix

Depending on the configuration above, a build matrix is generated. The matrix contains all combinations of environment settings. A single combination is called a job and is run separately. You can modify the matrix in matrix section.

If you want to exclude a job, use exclude key. In our case, we don't want to use -p hhvm parameter for standard PHP versions and -p php-cgi for HHVM.

matrix:
  exclude:
    - php: 5.3.3
      env: TESTER_PHP_BIN="hhvm"

    - php: 5.4
      env: TESTER_PHP_BIN="hhvm"

    - php: 5.5
      env: TESTER_PHP_BIN="hhvm"

    - php: 5.6
      env: TESTER_PHP_BIN="hhvm"

    - php: hhvm
      env: TESTER_PHP_BIN="php-cgi"

Travis shows the build as passing only if every single job pass. However, you can define jobs that are allowed to fail without causing the whole build to shown as failed. To do so, declare allow_failures. For our sake, we allow HHVM to fail.

matrix:
  allow_failures:
    - php: hhvm

Notice that when allowing failures, we only specified php version, not environment variables. This means all jobs with hhvm version will be allowed to fail, nevermind the environment variables values. It works with exclude aswell.

Running the tests

Tests are run in script section, where you only need to execute Tester. Let's assume your tests are in tests/ folder and you provide your own php.ini in the same folder. Additionaly, tell Tester to display information about skipped tests with -s option and to use value of earlier declared TESTER_PHP_BIN as PHP binary with -p option.

script:
  - ./vendor/bin/tester -p $TESTER_PHP_BIN -s -c ./tests/php.ini ./tests

If test fails

Section after_failure is executed if a test fails. Tester stores actual value of variables in case of assertion fail. We will use this section to print the actual values.

after_failure:
  # Prints *.actual files content
  - for i in $(find ./tests -name \*.actual); do echo "--- $i"; cat $i; echo; echo; done

Setting up other services:

Travis comes with multiple popular services (e.g. MySQL) pre-installed. However, if you need to use for example Redis storage, you can tell Travis in services section.

services:
  - redis-server

Database initialization:

MySQL runs on 127.0.0.1 and you can log in using travis or root as username. Password is not required. You can import your database in before_script section. Let's say your database set-up script is in tests/testbase.sql.

before_script:
  - mysql -u root -e 'CREATE DATABASE testbase;'
  - mysql -u root testbase < tests/testbase.sql

Result

The .travis.yml should look something like this by now:

language: php

php:
  - 5.3.3
  - 5.4
  - 5.5
  - 5.6
  - hhvm

env:
  - TESTER_PHP_BIN="php-cgi"
  - TESTER_PHP_BIN="hhvm"

matrix:
  allow_failures:
    - php: hhvm

  exclude:
    - php: 5.3.3
      env: TESTER_PHP_BIN="hhvm"

    - php: 5.4
      env: TESTER_PHP_BIN="hhvm"

    - php: 5.5
      env: TESTER_PHP_BIN="hhvm"

    - php: 5.6
      env: TESTER_PHP_BIN="hhvm"

    - php: hhvm
      env: TESTER_PHP_BIN="php-cgi"

services:
  - redis-server

before_install:
  - composer self-update

install:
  - composer install --no-interaction --prefer-source

before_script:
  - mysql -u root -e 'CREATE DATABASE testbase;'
  - mysql -u root testbase < tests/testbase.sql

script:
  - ./vendor/bin/tester -p $TESTER_PHP_BIN -c ./tests/php.ini -s ./tests/

after_failure:
  # Prints *.actual files content
  - for i in $(find ./tests -name \*.actual); do echo "--- $i"; cat $i; echo; echo; done

GitHub integration

As mentioned above, Travis is integrated with GitHub. However, you need to specify which repositories are to be tested. This is done using Webhook which notifies Travis about changes in your repository.

Activating Webhook

First, go to Travis CI and sign in with your GitHub account. After synchronizing your account, you`ll see all repositories you have access to. Flip switch to ON for all repositories you'd like to enable.

Travis will now add your repository to queue after every pushed commit or created pull-request. After a short while, your repository will be tested.

Status image

Travis can generate a status image for you. You can embed this icon into your README.md file for example.

Skipping commit

Some commits don't need testing. You can add [skip ci] somewhere in your commit message and Travis will ingore the commit.