Nette Tester – enjoyable unit testing

Even good programmers make mistakes. The difference between a good programmer and a bad one is that a good one detects it sooner by using automated tests.

“One who doesn't test is doomed to repeat his or her own mistakes.” (wise proverb)
“When we get rid of one error, another one appears.” (Murphy's Law)
“Do test, do test, do test” (Martin Iljič Fowler)

Have you ever written the following code in PHP?

$obj = new MyClass;
$result = $obj->process($input);

var_dump($result);

So, have you ever dumped a function call result just to check by eye that it returns what it should return? You surely do it many times per day. If everything works, do you delete this code and expect that the class will not be broken in the future? Murphy's Law guarantees the opposite :-)

In fact, we wrote the test. And if we didn't delete it we could run it any time in the future to verify that everything still works as it should. You may create a large amount of these tests over time, so it would be nice if we were able to run them automatically. It would be useful to slightly modify test not to require our inspection, simply to be able to check itself.

And Nette Tester helps exactly with that.

Installation and requirements

The Tester requires PHP version 5.3.0 or newer.

Preferred way of installation is by Composer and every following example assumes that. But the Tester can be used without it as will also be shown below.

Installation by Composer

Let's assume you have Composer up and running, and an application named demo with the following structure:

demo/
├── src/           # application code we want to test
├── tests/         # tests we are writing
├── vendor/
└── composer.json

Navigate in terminal to the application directory and add Tester as a dependency using Composer:

cd demo
php composer.phar require --dev nette/tester

Manual installation

Download Tester from GitHub and extract it or clone its repository by git clone https://github.com/nette/tester.git. Directory structure of our demo application is now as follows:

demo/
├── src/           # application code we want to test
├── tester/        # source of downloaded Tester
│   ├── src/
│   ├── tests/
│   ├── ...
│   └── readme.md
│
└── tests/         # tests we are writing

Running the Tester

Nette Tester is run from the command line. We can try it, without any arguments it will only show a help summary.

cd demo
php vendor/nette/tester/src/tester.php  # installation by Composer
php tester/src/tester.php               # manual installation

If Tester was installed by Composer we can also use the more convenient shortcut scripts:

cd demo
vendor/bin/tester      # UNIX
vendor\bin\tester.bat  # Windows

For simplification we only use the tester command further in this document. But one of the ways mentioned above with a full path is required to run it.

Running the tests

Our application has no tests yet. We create a simple class to be tested and we save it to file src/Greeting.php

<?php

class Greeting
{
    public function say($name)
    {
        if (!$name) {
            throw new InvalidArgumentException('Invalid name');
        }
        return "Hello $name";
    }
}

Let's write a test now. We save it to file tests/greeting.phpt. Don't be discouraged by its length, it is shown later how to simplify it.

<?php

use Tester\Assert;

# Load Tester library
require __DIR__ . '/../vendor/autoload.php';       # installation by Composer
require __DIR__ . '/../tester/src/bootstrap.php';  # manual installation

# Load the tested class. Composer or your autoloader surely takes
# care of that in practice.
require __DIR__ . '/../src/Greeting.php';


# Adjust PHP behaviour and enable some Tester features (described later)
Tester\Environment::setup();


$o = new Greeting;

Assert::same( 'Hello John', $o->say('John') );  # we expect the same

Assert::exception(function() use ($o) {         # we expect an exception
    $o->say('');
}, 'InvalidArgumentException', 'Invalid name');

The test is written we can run it from command line for the first time:

cd tests
php greeting.phpt

Yes, we have run the test as ordinary PHP script for the first time. Even it seems common, big potential is hidden in it. We can step through the test in IDE or load it via web browser.

The first running finds syntactic errors and if we didn't make a typo the test ends without error report. Let's change an assertion in the test to Assert::same( 'Hi John', $o->say('John') ); and let's watch what happens when run.

As the application grows number of tests grows with it. It would not be practical to run tests one by one. We use the Tester for running:

cd demo
tester tests/greeting.phpt  # we run a single test
tester tests                # we run all tests in directory

To see how more tests running looks like, let's try to run all tests which test the Tester itself. They are part of its installation:

tester vendor/nette/tester/tests  # installation by Composer
tester tester/tests               # manual installation

How the Tester runs

The Tester walks through passed directories at first, it creates a list of found tests and it runs them one by one. It runs every test as a new process, every test runs completely separated from others.

The Tester runs PHP processes with -n option, so without php.ini. More details in the Own php.ini chapter.

Test results evaluation

The Tester prints test results continuously during testing:

. (dot) – test passed
s – test has been skipped
F – test failed

Output may look like:

 _____ ___  ___ _____ ___  ___
|_   _/ __)( __/_   _/ __)| _ )
  |_| \___ /___) |_| \___ |_|_\  v1.7.0

PHP 5.4.4-14+deb7u7 | 'php-cgi' -n | 8 threads

........s................F.........

-- FAILED: tests/greeting.phpt
   Failed: 'Hello John' should be
       ... 'Hi John'

   in src/Framework/Assert.php(370)
   in src/Framework/Assert.php(52) Tester\Assert::fail()
   in tests/greeting.phpt(6) Tester\Assert::same()


FAILURES! (35 tests, 1 failures, 1 skipped, 1.7 seconds)

35 tests were run, one failed, one was skipped. If no one fails Tester's exit code is zero. Non-zero otherwise.

Environment::setup()

There has been mentioned a little bit cryptic Tester\Environment::setup() call in tests. What does it do?

improves error dump readability (coloring included), otherwise, default PHP dump is printed
enables check that assertions has been called in test, otherwise, tests without (e.g. forgotten) assertions pass too
automatically starts code coverage collector when --coverage is used (described later)

Using is optional but recommended.

Directories structure

It may seem early to talk about it but well-structured directory with tests saves work a lot. We separate tests into subdirectories by namespace of tested classes:

demo/
└── tests/
    ├── NamespaceOne/
    │   ├── MyClass.getUsers.phpt
    │   ├── MyClass.setUsers.phpt
    │   └── ...
    │
    ├── NamespaceTwo/
    │   ├── MyClass.creating.phpt
    │   ├── MyClass.dropping.phpt
    │   └── ...
    │
    ├── test.one.phpt
    ├── test.two.phpt
    ├── ...
    └── bootstrap.php

and we create a file bootstrap.php. It contains a common code for all tests. For example classes autoloading, environment configuration, temporary directory creation, helpers and similar. Every test loads the bootstrap and pays attention to testing only. The bootstrap can look like:

require __DIR__ . '/../vendor/autoload.php';

Tester\Environment::setup();
date_default_timezone_set('Europe/Prague');

define('TMP_DIR', '/tmp/demo-app-tests');

There is no difference to run the tests with changed directory structure. The Tester finds all *.phpt tests recursively and runs them:

cd demo
tester tests

But we can run tests for single namespace easily:

tester tests/NamespaceOne

Assertions

In the example test at the beginning we used two assertions: Assert::same() and Assert::exception(). Now we introduce all other types and we explain how to be used. They are methods of the Tester\Assert class but for simplification we use them further without namespace.

Assert::same($expected, $actual)

$expected must be the same as $actual. It is the same as PHP operator ===.

Assert::notSame($expected, $actual)

Opposite to Assert::same().

Assert::equal($expected, $actual)

$expected must be equal to $actual. Object identities and array keys order are ignored.

Assert::notEqual($expected, $actual)

Opposite to Assert::equal().

Assert::contains($needle, $actual)

$actual must contain $needle. If $actual is string, it must contain $needle substring. If it is an array, it must contain $needle item.

Assert::notContains($needle, $actual)

Opposite to Assert::contains().

Assert::true($value)

$value must be TRUE, so $value === TRUE.

Assert::truthy($value)

$value must be truthy, so $value == TRUE.

Assert::false($value)

$value must be FALSE, so $value === FALSE.

Assert::falsey($value)

$value must be falsey, so $value == FALSE.

Assert::count($count, $value)

$count must be number of elements in $value. Countable is an array or an object implementing Countable.

Assert::null($value)

$value must be NULL, so $value === NULL.

Assert::nan($value)

$value must be Not a Number.

Assert::type($type, $value)

$value must be of given type. As $type we can use string:

array
list – same as the array but keys must start by zero and must be incremented by one
bool
callable
float
int or integer
null
object
resource
scalar
string
class name or object directly then must pass $value instanceof $type

Assert::exception($callable, $class, $message = NULL)

On $callable invocation an exception of $class instance must be thrown. If we pass $message, message of the exception must match (see Assert::match()). And if we pass $code, code of the exception must be the same. For example, this test fails because message of the exception does not match:

Assert::exception(function() {
    throw new App\InvalidValueException('Zero value');
}, 'App\InvalidValueException', 'Value is to low');

The Assert::exception() is unique by returning the thrown exception. So we can test a previous exception:

$e = Assert::exception(function() {
    throw new MyException('Something is wrong', 0, new RuntimeException);
}, 'MyException', 'Something is wrong');

Assert::type('RuntimeException', $e->getPrevious());

Assert::error($callable, $type, $message = NULL)

If we pass a class name as $type, this assertion behaves as absolutely same as Assert::exception().

If $type is one of E_... constants, for example E_WARNING, the $callable must generate this error when invoked. And if we pass $message message of the error must match (see Assert::match()). For example:

Assert::error(function() {
    $i++;
}, E_NOTICE, 'Undefined variable: i');

And the last possibility, if $type is an array the $callable must generate all expected errors. An example shows it best:

Assert::error(function() {
    $a++;
    $b++;
}, [
    [E_NOTICE, 'Undefined variable: a'],
    [E_NOTICE, 'Undefined variable: b'],
]);

Assert::noError($callable)

Checks that the function $callable does not throw any PHP warning/notice/error or exception. It is useful for testing a piece of code where is no other assertion.

Assert::match($pattern, $actual)

$actual must match to $pattern. We can use two pattern types:

We pass regular expression as $pattern. To delimit it we must use ~ or #. Other delimiters are not supported. For example test where $var must contain only hexadecimal digits:

Assert::match( '#^[0-9a-f]$#i', $var );

The other variant is similar to string comparing but we can use some modifiers in $pattern:

%a% one or more of anything except the end of line characters
%a?% zero or more of anything except the end of line characters
%A% one or more of anything including the end of line characters
%A?% zero or more of anything including the end of line characters
%s% one or more white space characters except the end of line characters
%s?% zero or more white space characters except the end of line characters
%S% one or more of characters except the white space
%S?% zero or more of characters except the white space
%c% a single character of any sort (except the end of line)
%d% one or more digits
%d?% zero or more digits
%i% signed integer value
%f% floating point number
%h% one or more HEX digits

Examples:

# Again, hexadecimal number test
Assert::match( '%h%', $var );

# Generalized path to file and line number
Assert::match( 'Error in file %a% on line %i%', $errorMessage );

Assert::matchFile($file, $actual)

The assertion is identical to Assert::match() but the pattern is loaded from $file. It is useful for very long strings testing. Test file stands readable.

Assert::fail($message, $actual = NULL, $expected = NULL)

This assertion always fails. It is just handy. We can optionally pass expected and actual values.

Failed assertions investigation

The Tester shows where the error is when an assertion fails. When we compare complex structures, the Tester creates dumps of compared values and saves them into directory output. For example when imaginary test Arrays.recursive.phpt fails the dumps will be saved as follows:

demo/
└── tests/
    ├── output/
    │   ├── Arrays.recursive.actual    # actual value
    │   └── Arrays.recursive.expected  # expected value
    │
    └── Arrays.recursive.phpt          # failing test

We can change the name of the directory in Tester\Dumper::$dumpDir.

Skipping tests

Some tests can run only under certain circumstances. If these circumstances aren't met, we can skip the tests. For example, missing PHP extension:

if (!extension_loaded('DOM')) {
    Tester\Environment::skip('Test requires DOM extension to be loaded.');
}

Test will be stopped and marked as s – skipped. Later we find out how to skip the test using the @skip annotation.

TestCase

We have shown the simple test where assertions follow one by one at the beginning of this guide. Sometimes it is useful to enclose the assertions to test class and structure them in this way. The class must be descendant of Tester\TestCase and we talk about it simply as about testcase.

use Tester\Assert;

class GreetingTest extends Tester\TestCase
{
    public function testOne() {
        Assert::same(......);
    }

    public function testTwo() {
        Assert::match(......);
    }
}

# Testing methods run
$testCase = new GreetingTest;
$testCase->run();

We can enrich a testcase by setUp() and tearDown() methods. They are called before/after every testing method:

use Tester\Assert;

class NextTest extends Tester\TestCase
{
    public function setUp() {
        # Preparation
    }

    public function tearDown() {
        # Clean-up
    }

    public function testOne() {
        Assert::same(......);
    }

    public function testTwo() {
        Assert::match(......);
    }
}

# Testing methods run
$testCase = new NextTest;
$testCase->run();

/*
Method calls order
------------------
setUp()
testOne()
tearDown()

setUp()
testTwo()
tearDown()
*/

There are a few annotations available to help us with testing methods. We write them toward the testing method.

@throws

It is equal usage of Assert::exception() inside a testing method. But notation is more readable:

/**
 * @throws RuntimeException
 */
public function testOne()
{
    ...
}


/**
 * @throws LogicException  Wrong argument order
 */
public function testTwo()
{
    ...
}

@dataProvider

This annotation suits when we want to run the testing method multiple times but with different arguments. As argument we write method name which return parameters for testing method. The method must return an array or Traversable. Simple example:

public function getLoopArgs()
{
    return [
        [1, 2, 3],
        [4, 5, 6],
        [7, 8, 9],
    ];
}


/**
 * @dataProvider getLoopArgs
 */
public function testLoop($a, $b, $c)
{
    ...
}

The other annotation @dataProvider variation accepts a path to INI file (relatively to test file) as argument. The method is called so many times as the number of sections contained in INI file. File loop-args.ini:

[one]
a=1
b=2
c=3

[two]
a=4
b=5
c=6

[three]
a=7
b=8
c=9

and the method which uses the INI file:

/**
 * @dataProvider loop-args.ini
 */
public function testLoop($a, $b, $c)
{
    ...
}

Similarly, we can pass path to a PHP script instead of INI. It must return array or Traversable. File loop-args.php:

return [
   ['a' => 1, 'b' => 2, 'c' => 3],
   ['a' => 4, 'b' => 5, 'c' => 6],
   ['a' => 7, 'b' => 8, 'c' => 9],
];

Test file annotations

Following annotations are evaluated only if we run the test by the Tester, not manually as ordinary PHP script.

We write annotations at the beginning of the test file and the typing is case-insensitive. Imaginary example:

/**
 * TEST: Basic database query test.
 *
 * @dataProvider files/databases.ini
 * @exitCode 56
 * @phpVersion < 5.5
 */

require __DIR__ . '/../bootstrap.php';

TEST:

It is not an annotation actually. It only sets the test name which is printed on fail or into logs.

@skip

Test is skipped. It is handy for temporary tests deactivation.

@phpVersion

Test is skipped if is not run by corresponding PHP version. We write annotation as @phpVersion [operator] version. We can leave out the operator, default is >=. Examples:

/**
 * @phpVersion 5.3.3
 * @phpVersion < 5.5
 * @phpVersion != 5.4.5
 */

@dataProvider

We write annotation as @dataProvider file.ini. INI file path is relative to the test file. Test runs as many times as number of sections contained in the INI file. Let's assume the INI file databases.ini:

[mysql]
dsn = "mysql:host=127.0.0.1"
user = root
password = ******

[postgresql]
dsn = "pgsql:host=127.0.0.1;dbname=test"
user = postgres
password = ******

[sqlite]
dsn = "sqlite::memory:"

and the file database.phpt in the same directory:

/**
 * @dataProvider databases.ini
 */

$args = Tester\Environment::loadData();

The test runs three times and $args will contain values from sections mysql, postgresql or sqlite.

There is one more variation when we write annotations with a question mark as @dataProvider? file.ini. In this case test is skipped if the INI file doesn't exist.

Annotation possibilities have not been mentioned all yet. We can write conditions after the INI file. Test runs for given section only if all conditions match. Let's extend the INI file:

[mysql]
dsn = "mysql:host=127.0.0.1"
user = root
password = ******

[postgresql 8.4]
dsn = "pgsql:host=127.0.0.1;dbname=test"
user = postgres
password = ******

[postgresql 9.1]
dsn = "pgsql:host=127.0.0.1;dbname=test;port=5433"
user = postgres
password = ******

[sqlite]
dsn = "sqlite::memory:"

and we will use annotation with condition:

/**
 * @dataProvider  databases.ini  postgresql, >=9.0
 */

The test runs only once for section postgresql 9.1. Other sections don’t match the conditions.

Similarly, we can pass path to a PHP script instead of INI. It must return array or Traversable. File databases.php:

return [
    'postgresql 8.4' => [
        'dsn' => '...',
        'user' => '...',
    ],

    'postgresql 9.1' => [
        'dsn' => '...',
        'user' => '...',
    ],
];

@multiple

We write it as @multiple N where N is integer. Test runs exactly N-times.

@testCase

Annotation has no parameters. We use it when we write a test as Tester\TestCase classes. In this case the test runs as many times as the number of testing methods. Every testing method runs in separated process. It can dramatically speed up the whole testing procedure. We recommend usage of this annotation.

@exitCode

We write it as @exitCode N where N is the exit code of the test. For example if exit(10) is called in the test, we write annotation as @exitCode 10. It is considered to be fail if the test ends with a different code. Exit code 0 (zero) is verified if we leave out the annotation

@httpCode

Annotation is evaluated only if PHP binary is CGI. It is ignored otherwise. We write it as @httpCode NNN where NNN is expected HTTP code. HTTP code 200 is verified if we leave out the annotation. If we write NNN as a string evaluated as zero, for example any, HTTP code is not checked at all.

@outputMatch a @outputMatchFile

The behaviour of annotations is consistent with Assert::match() and Assert::matchFile() assertions. But pattern is found in test’s standard output. A suitable use case is when we assume the test to end by fatal error and we need to verify its output.

@phpIni

It sets INI configuration values for test. For example we write it as @phpIni precision=20 and it works in the same way as if we passed value from the command line by parameter -d precision=20.

Helpers

DomQuery

Tester\DomQuery class helps to test HTML or XML content. Description of all the methods is in API documentation. Here, we show basic usage:

# HTML which we want to test
$html = $template->render();

# We create DOM structure from HTML
$dom = Tester\DomQuery::fromHtml($html);

# We can test
Assert::true( $dom->has('form#registration') );
Assert::true( $dom->has('input[name="username"]') );
Assert::true( $dom->has('input[name="password"]') );
Assert::true( $dom->has('input[type="submit"]') );

We use CSS selector as DomQuery::has($selector) parameter.

FileMock

Tester\FileMock emulates files in memory and helps you to test a code which uses functions like fopen(), file_get_contents() or parse_ini_file(). For example:

# Tested class
class Logger
{
    private $logFile;

    public function __construct($logFile)
    {
        $this->logFile = $logFile;
    }

    public function log($message)
    {
        file_put_contents($this->logFile, $message . "\n", FILE_APPEND);
    }
}

# New empty file
$file = Tester\FileMock::create('');

$logger = new Logger($file);
$logger->log('Login');
$logger->log('Logout');

# Created content testing
Assert::same( "Login\nLogout\n", file_get_contents($file) );

purge()

Class Tester\Helpers offers method purge() which creates passed directory but only if doesn't exist yet. If so, it purges all its content. It is handy for temporary directory creation. For example in tests/bootstrap.php:

@mkdir(__DIR__ . '/tmp');  # @ - directory may already exist

define('TEMP_DIR', __DIR__ . '/tmp/' . getmypid());
Tester\Helpers::purge(TEMP_DIR);

lock()

Later we learn that tests run in parallel. Sometimes we need not to overlap the test running. Typically database tests need to prepare structure and data in database and they need nothing disturbs them during running time of the test. In these cases we use Tester\Environment::lock($name, $dir):

Tester\Environment::lock('database', __DIR__ . '/tmp');

The first argument is a lock name. The second one is a path to directory for saving the lock. The test which acquires the lock first runs. Other tests must wait till it is completed.

Command line options

We obtain command line options overview by running the Tester without parameters or with option -h.

-p <path>

The Tester uses this PHP binary for tests running:

tester -p /home/user/php-5.5.0/php-cgi tests

-c <path>

The option is the same as for PHP binary. Passed php.ini is used for tests. No php.ini is used in default. Example when we distribute php.ini along with tests:

tester -c tests/php.ini tests

-l | –log <path>

Progress of the test is written to file. All failed, skipped and also successful tests:

tester -log /var/log/tests.log tests

-d <key=value>

The option is the same as for PHP binary. It defines INI value for tests. The option can be used multiple times.

-s

Information about skipped tests will be shown.

--stop-on-fail

Tester stops testing upon the first failing test.

-j <num>

Tests run in <num> parallel threads. Default value is 8. If we wish to run tests in series, we use value 1.

-o <console|tap|junit|none>

Output format. Default is the console format.

console: the same as default, but logo is not printed in this case
tap: TAP format appropriate for machine processing (replaces the --tap option)
junit: JUnit XML format, appropriate for machine processing too
none: nothing is printed

-w | –watch <path>

The Tester doesn't end after tests are completed but it keeps running and watching given directory. It runs tests again whenever directory changes. Parameter can be used multiple times. It is handy during tests writing and debugging.

-i | –info

It shows information about a test running environment. For example:

tester -p /home/php/php-5.5.6 -c tests/php.ini --info


PHP binary:
/home/php/php-5.5.6

PHP version:
5.5.6 (cli)

Loaded php.ini files:
/var/www/dev/demo/tests/php.ini

Loaded extensions:
Core, ctype, date, dom, ereg, fileinfo, filter, hash, ...

--setup <path>

The Tester loads given PHP script on start. Variable Tester\Runner\Runner $runner is available in it. Let's assume file tests/runner-setup.php:

$runner->outputHandler[] = new MyOutputHandler;

and we run the Tester:

tester --setup tests/runner-setup.php tests

--colors 1|0

The Tester detects a colour-able terminal in default and colorizes its output. This option is over the autodetection. We can set colouring globally by a system environment variable NETTE_TESTER_COLORS.

--coverage <path>

Tester will generate a report with overview how much is the source code coveraged by tests. This option requires PHP extension Xdebug enabled, or PHP 7 with the PHPDBG SAPI, which is faster. The destination file extension determines the contents format. HTML or Clover XML.

tester tests --coverage coverage.html  # HTML report
tester tests --coverage coverage.xml   # Clover XML report

--coverage-src <path>

We use it with option --coverage simultaneously. The <path> is a path to source code for which we generate the report.

HHVM support

Tester supports HHVM 3.3.0 or newer. Usage is simple, just pass a HHVM binary path using the -p option.

tester -p hhvm

Tips

It is not good practice to use exit() and die() to end test with a fail message. For example exit('Error in connection') ends a test with exit code 0 (zero) and it means success. Always use echo 'Error in connection' followed by exit(1).
It is convenient to write accurate assertions Assert::same($a, $b), not Assert::true($a === $b). Only in the first case we get meaningful error message. In the other case we get FALSE should be TRUE only and it says nothing about $a and $b variables.
Use the Assert::nan() for NAN (Not a Value) testing. NAN value is very specific and assertions Assert::same() or Assert::equal() can work unpredictably.

Own php.ini

The Tester runs PHP processes with -n option, so no php.ini is loaded (not even that from /etc/php/conf.d/*.ini in UNIX). It ensures the most clean environment for the tests run, but it also deactivates all the external PHP extensions commonly loaded by system PHP.

If you need some extensions or some special INI settings, create own php.ini file and distribute it among the tests. Then we run Tester with -c option, e.g. tester -c tests/php.ini. The INI file may look like:

[PHP]

extension=php_pdo_mysql.dll
extension=php_pdo_pgsql.dll

memory_limit=512M

Running the Tester with a system php.ini in UNIX, e.g. tester -c /etc/php/cgi/php.ini, does not load other INI from /etc/php/conf.d/*.ini. That's the PHP behaviour.

Continuous integration

If you’re not familiar with continuous integration, this guide will show you what it is all about.