Writing Tests

Each test is as an ordinary PHP script. Even it seems common, big potential is hidden in it. We can step through the test in IDE or load it via web browser.

Assertions

In the example on guide page we used two assertions: Assert::same() and Assert::exception(). Now we introduce all other types and we explain how to be used. They are methods of the Tester\Assert class but for simplification we use them further without namespace.

Assert::same($expected, $actual)

$expected must be the same as $actual. It is the same as PHP operator ===.

Assert::notSame($expected, $actual)

Opposite to Assert::same().

Assert::equal($expected, $actual)

$expected must be equal to $actual. Object identities and array keys order are ignored.

Assert::notEqual($expected, $actual)

Opposite to Assert::equal().

Assert::contains($needle, $actual)

$actual must contain $needle. If $actual is string, it must contain $needle substring. If it is an array, it must contain $needle item.

Assert::notContains($needle, $actual)

Opposite to Assert::contains().

Assert::true($value)

$value must be true, so $value === true.

Assert::truthy($value)

$value must be truthy, so $value == true.

Assert::false($value)

$value must be false, so $value === false.

Assert::falsey($value)

$value must be falsey, so $value == false.

Assert::count($count, $value)

$count must be number of elements in $value. Countable is an array or an object implementing Countable.

Assert::null($value)

$value must be null, so $value === null.

Assert::nan($value)

$value must be Not a Number.

Assert::type($type, $value)

$value must be of given type. As $type we can use string:

  • array
  • list – same as the array but keys must start by zero and must be incremented by one
  • bool
  • callable
  • float
  • int or integer
  • null
  • object
  • resource
  • scalar
  • string
  • class name or object directly then must pass $value instanceof $type

Assert::exception($callable, $class, $message = null)

On $callable invocation an exception of $class instance must be thrown. If we pass $message, message of the exception must match (see Assert::match()). And if we pass $code, code of the exception must be the same. For example, this test fails because message of the exception does not match:

Assert::exception(function () {
    throw new App\InvalidValueException('Zero value');
}, App\InvalidValueException::class, 'Value is to low');

The Assert::exception() is unique by returning the thrown exception. So we can test a previous exception:

$e = Assert::exception(function () {
    throw new MyException('Something is wrong', 0, new RuntimeException);
}, MyException::class, 'Something is wrong');

Assert::type(RuntimeException::class, $e->getPrevious());

Assert::error($callable, $type, $message = null)

If we pass a class name as $type, this assertion behaves as absolutely same as Assert::exception().

If $type is one of E_... constants, for example E_WARNING, the $callable must generate this error when invoked. And if we pass $message message of the error must match (see Assert::match()). For example:

Assert::error(function () {
    $i++;
}, E_NOTICE, 'Undefined variable: i');

And the last possibility, if $type is an array the $callable must generate all expected errors. An example shows it best:

Assert::error(function () {
    $a++;
    $b++;
}, [
    [E_NOTICE, 'Undefined variable: a'],
    [E_NOTICE, 'Undefined variable: b'],
]);

Assert::noError($callable)

Checks that the function $callable does not throw any PHP warning/notice/error or exception. It is useful for testing a piece of code where is no other assertion.

Assert::match($pattern, $actual)

$actual must match to $pattern. We can use two pattern types:

We pass regular expression as $pattern. To delimit it we must use ~ or #. Other delimiters are not supported. For example test where $var must contain only hexadecimal digits:

Assert::match('#^[0-9a-f]$#i', $var);

The other variant is similar to string comparing but we can use some modifiers in $pattern:

  • %a% one or more of anything except the end of line characters
  • %a?% zero or more of anything except the end of line characters
  • %A% one or more of anything including the end of line characters
  • %A?% zero or more of anything including the end of line characters
  • %s% one or more white space characters except the end of line characters
  • %s?% zero or more white space characters except the end of line characters
  • %S% one or more of characters except the white space
  • %S?% zero or more of characters except the white space
  • %c% a single character of any sort (except the end of line)
  • %d% one or more digits
  • %d?% zero or more digits
  • %i% signed integer value
  • %f% floating point number
  • %h% one or more HEX digits
  • %w% one or more alphanumeric characters
  • %% one % character

Examples:

# Again, hexadecimal number test
Assert::match('%h%', $var);

# Generalized path to file and line number
Assert::match('Error in file %a% on line %i%', $errorMessage);

Assert::matchFile($file, $actual)

The assertion is identical to Assert::match() but the pattern is loaded from $file. It is useful for very long strings testing. Test file stands readable.

Assert::fail($message, $actual = null, $expected = null)

This assertion always fails. It is just handy. We can optionally pass expected and actual values.

Failed assertions investigation

The Tester shows where the error is when an assertion fails. When we compare complex structures, the Tester creates dumps of compared values and saves them into directory output. For example when imaginary test Arrays.recursive.phpt fails the dumps will be saved as follows:

demo/
└── tests/
    ├── output/
    │   ├── Arrays.recursive.actual    # actual value
    │   └── Arrays.recursive.expected  # expected value
    │
    └── Arrays.recursive.phpt          # failing test

We can change the name of the directory by Tester\Dumper::$dumpDir.

Skipping tests

Some tests can run only under certain circumstances. If these circumstances aren't met, we can skip the tests. For example, required OS type:

if (defined('PHP_WINDOWS_VERSION_BUILD')) {
    Tester\Environment::skip('Requires UNIX.');
}

Test will be stopped and marked as s – skipped. Later we find out how to skip the test using the @skip annotation.

TestCase

Assertions may follow one by one in simple tests. But sometimes it is useful to enclose the assertions to test class and structure them in this way. The class must be descendant of Tester\TestCase and we talk about it simply as about testcase.

use Tester\Assert;

class GreetingTest extends Tester\TestCase
{
    public function testOne()
    {
        Assert::same(......);
    }

    public function testTwo()
    {
        Assert::match(......);
    }
}

# Run testing methods
(new GreetingTest)->run();

We can enrich a testcase by setUp() and tearDown() methods. They are called before/after every testing method:

use Tester\Assert;

class NextTest extends Tester\TestCase
{
    public function setUp()
    {
        # Preparation
    }

    public function tearDown()
    {
        # Clean-up
    }

    public function testOne()
    {
        Assert::same(......);
    }

    public function testTwo()
    {
        Assert::match(......);
    }
}

# Run testing methods
(new NextTest)->run();

/*
Method calls order
------------------
setUp()
testOne()
tearDown()

setUp()
testTwo()
tearDown()
*/

If error occurs in a setUp() or tearDown() phase, test will fail. If error occurs in the testing method, the tearDown() method is called anyway, but with suppressed errors in it.

Annotations for TestCase

There are few annotations available to help us with testing methods. We write them toward the testing method.

@throws

It is equal usage of Assert::exception() inside a testing method. But notation is more readable:

/**
 * @throws RuntimeException
 */
public function testOne()
{
    ...
}


/**
 * @throws LogicException  Wrong argument order
 */
public function testTwo()
{
    ...
}

@dataProvider

This annotation suits when we want to run the testing method multiple times but with different arguments. As argument we write method name which return parameters for testing method. The method must return an array or Traversable. Simple example:

public function getLoopArgs()
{
    return [
        [1, 2, 3],
        [4, 5, 6],
        [7, 8, 9],
    ];
}


/**
 * @dataProvider getLoopArgs
 */
public function testLoop($a, $b, $c)
{
    ...
}

The other annotation @dataProvider variation accepts a path to INI file (relatively to test file) as argument. The method is called so many times as the number of sections contained in INI file. File loop-args.ini:

[one]
a=1
b=2
c=3

[two]
a=4
b=5
c=6

[three]
a=7
b=8
c=9

and the method which uses the INI file:

/**
 * @dataProvider loop-args.ini
 */
public function testLoop($a, $b, $c)
{
    ...
}

Similarly, we can pass path to a PHP script instead of INI. It must return array or Traversable. File loop-args.php:

return [
   ['a' => 1, 'b' => 2, 'c' => 3],
   ['a' => 4, 'b' => 5, 'c' => 6],
   ['a' => 7, 'b' => 8, 'c' => 9],
];

Test file annotations

Following annotations are evaluated only if we run the test by the Tester, not manually as ordinary PHP script.

We write annotations at the beginning of the test file and the typing is case-insensitive. Imaginary example:

/**
 * TEST: Basic database query test.
 *
 * @dataProvider files/databases.ini
 * @exitCode 56
 * @phpVersion < 5.5
 */

require __DIR__ . '/../bootstrap.php';

TEST:

It is not an annotation actually. It only sets the test title which is printed on fail or into logs.

@skip

Test is skipped. It is handy for temporary tests deactivation.

@phpVersion

Test is skipped if is not run by corresponding PHP version. We write annotation as @phpVersion [operator] version. We can leave out the operator, default is >=. Examples:

/**
 * @phpVersion 5.3.3
 * @phpVersion < 5.5
 * @phpVersion != 5.4.5
 */

@phpExtension

Test is skipped if all mentioned PHP extension are not loaded. Multiple extensions can be written in a single annotation, or we can use the annotation multiple times.

/**
 * @phpExtension pdo, pdo_pgsql, pdo_mysql
 * @phpExtension json
 */

@dataProvider

We write annotation as @dataProvider file.ini. INI file path is relative to the test file. Test runs as many times as number of sections contained in the INI file. Let's assume the INI file databases.ini:

[mysql]
dsn = "mysql:host=127.0.0.1"
user = root
password = ******

[postgresql]
dsn = "pgsql:host=127.0.0.1;dbname=test"
user = postgres
password = ******

[sqlite]
dsn = "sqlite::memory:"

and the file database.phpt in the same directory:

/**
 * @dataProvider databases.ini
 */

$args = Tester\Environment::loadData();

The test runs three times and $args will contain values from sections mysql, postgresql or sqlite.

There is one more variation when we write annotations with a question mark as @dataProvider? file.ini. In this case test is skipped if the INI file doesn't exist.

Annotation possibilities have not been mentioned all yet. We can write conditions after the INI file. Test runs for given section only if all conditions match. Let's extend the INI file:

[mysql]
dsn = "mysql:host=127.0.0.1"
user = root
password = ******

[postgresql 8.4]
dsn = "pgsql:host=127.0.0.1;dbname=test"
user = postgres
password = ******

[postgresql 9.1]
dsn = "pgsql:host=127.0.0.1;dbname=test;port=5433"
user = postgres
password = ******

[sqlite]
dsn = "sqlite::memory:"

and we will use annotation with condition:

/**
 * @dataProvider  databases.ini  postgresql, >=9.0
 */

The test runs only once for section postgresql 9.1. Other sections don’t match the conditions.

Similarly, we can pass path to a PHP script instead of INI. It must return array or Traversable. File databases.php:

return [
    'postgresql 8.4' => [
        'dsn' => '...',
        'user' => '...',
    ],

    'postgresql 9.1' => [
        'dsn' => '...',
        'user' => '...',
    ],
];

@multiple

We write it as @multiple N where N is integer. Test runs exactly N-times.

@testCase

Annotation has no parameters. We use it when we write a test as Tester\TestCase classes. In this case the test runs as many times as the number of testing methods. Every testing method runs in separated process. It can dramatically speed up the whole testing procedure. We recommend usage of this annotation.

@exitCode

We write it as @exitCode N where N is the exit code of the test. For example if exit(10) is called in the test, we write annotation as @exitCode 10. It is considered to be fail if the test ends with a different code. Exit code 0 (zero) is verified if we leave out the annotation

@httpCode

Annotation is evaluated only if PHP binary is CGI. It is ignored otherwise. We write it as @httpCode NNN where NNN is expected HTTP code. HTTP code 200 is verified if we leave out the annotation. If we write NNN as a string evaluated as zero, for example any, HTTP code is not checked at all.

@outputMatch a @outputMatchFile

The behaviour of annotations is consistent with Assert::match() and Assert::matchFile() assertions. But pattern is found in test’s standard output. A suitable use case is when we assume the test to end by fatal error and we need to verify its output.

@phpIni

It sets INI configuration values for test. For example we write it as @phpIni precision=20 and it works in the same way as if we passed value from the command line by parameter -d precision=20.

Helpers

DomQuery

Tester\DomQuery class helps to test HTML or XML content. Description of all the methods is in API documentation. Here, we show basic usage:

# HTML which we want to test
$html = $template->render();

# We create DOM structure from HTML
$dom = Tester\DomQuery::fromHtml($html);

# We can test
Assert::true($dom->has('form#registration'));
Assert::true($dom->has('input[name="username"]'));
Assert::true($dom->has('input[name="password"]'));
Assert::true($dom->has('input[type="submit"]'));

We use CSS selector as DomQuery::has($selector) parameter.

FileMock

Tester\FileMock emulates files in memory and helps you to test a code which uses functions like fopen(), file_get_contents() or parse_ini_file(). For example:

# Tested class
class Logger
{
    private $logFile;

    public function __construct($logFile)
    {
        $this->logFile = $logFile;
    }

    public function log($message)
    {
        file_put_contents($this->logFile, $message . "\n", FILE_APPEND);
    }
}

# New empty file
$file = Tester\FileMock::create('');

$logger = new Logger($file);
$logger->log('Login');
$logger->log('Logout');

# Created content testing
Assert::same("Login\nLogout\n", file_get_contents($file));

Environment::purge()

Class Tester\Helpers offers method purge() which creates passed directory but only if doesn't exist yet. If so, it purges all its content. It is handy for temporary directory creation. For example in tests/bootstrap.php:

@mkdir(__DIR__ . '/tmp');  # @ - directory may already exist

define('TEMP_DIR', __DIR__ . '/tmp/' . getmypid());
Tester\Helpers::purge(TEMP_DIR);

Environment::lock()

Later we learn that tests run in parallel. Sometimes we need not to overlap the test running. Typically database tests need to prepare database content and they need nothing disturbs them during running time of the test. In these cases we use Tester\Environment::lock($name, $dir):

Tester\Environment::lock('database', __DIR__ . '/tmp');

The first argument is a lock name. The second one is a path to directory for saving the lock. The test which acquires the lock first runs. Other tests must wait till it is completed.

Environment::bypassFinals()

Classes or methods marked as as final are hard to test. Calling the Tester\Environment::bypassFinals() in a test beginning causes that keywords final are removed during the code loading.

require __DIR__ . '/bootstrap.php';

Tester\Environment::bypassFinals();

class MyClass extends NormallyFinalClass  # <-- NormallyFinalClass is not final anymore
{
    ...
}

Environment::setup()

  • improves error dump readability (coloring included), otherwise, default PHP stack trace is printed
  • enables check that assertions has been called in test, otherwise, tests without (e.g. forgotten) assertions pass too
  • automatically starts code coverage collector when --coverage is used (described later)

Using is optional but recommended.

Environment::RUNNER

Could by useful to know if test is running by the Tester or manually as ordinary PHP script.

if (getenv(Tester\Environment::RUNNER)) {
    # run by Tester
} else {
    # other way

Environment::THREAD

Tester runs tests in parallel in a given number of threads. We will find a thread number in an environmental variable, when we are interested:

echo "I'm running in a thread number " . getenv(Tester\Environment::THREAD).

Tips

  • It is not good practice to use exit() and die() to end test with a fail message. For example exit('Error in connection') ends a test with exit code 0 (zero) and it means success. Always use echo 'Error in connection' followed by exit(1).
  • Choose the most accurate assertions. For example Assert::same($a, $b), not Assert::true($a === $b). Only in the first case we get meaningful error message. In the other case we get false should be true only and it says nothing about $a and $b variables contents.
  • Use the Assert::nan() for NAN (Not a Value) testing. NAN value is very specific and assertions Assert::same() or Assert::equal() can behave unpredictably.

API documentation

Generated API documentation is here.