Running Tests

The most visible part of Nette Tester is the command-line test runner. It is extremely fast and robust because it automatically runs all tests as separate processes in parallel using multiple threads. It can also run itself in watch mode.

The test runner is invoked from the command line. Pass the directory containing the tests as a parameter. For the current directory, simply enter a dot:

vendor/bin/tester .

The test runner scans the specified directory and all its subdirectories, looking for tests, which are files ending in *.phpt or *Test.php. It also reads and evaluates their annotations to determine which tests to run and how.

It then executes the tests. During execution, it prints characters to the terminal to indicate the progress:

  • . – test passed
  • s – test has been skipped
  • F – test failed

The output may look like this:

 _____ ___  ___ _____ ___  ___
|_   _/ __)( __/_   _/ __)| _ )
  |_| \___ /___) |_| \___ |_|_\  v2.5.2

Note: No php.ini is used.
PHP 8.3.2 (cli) | php -n | 8 threads

........s..........................

OK (35 tests, 1 skipped, 1.7 seconds)

When run again, it first executes the tests that failed in the previous run, so you immediately know if you have fixed the error.

The Tester's exit code is zero if no test fails. Otherwise, it is non-zero.

The Tester runs PHP processes without php.ini. More details in the Own php.ini section.

Command-Line Options

You can get an overview of command-line options by running Tester without parameters or with the -h option:

 _____ ___  ___ _____ ___  ___
|_   _/ __)( __/_   _/ __)| _ )
  |_| \___ /___) |_| \___ |_|_\  v2.5.2

Usage:
    tester [options] [<test file> | <directory>]...

Options:
    -p <path>                    Specify PHP interpreter to run (default: php).
    -c <path>                    Look for php.ini file (or look in directory) <path>.
    -C                           Use system-wide php.ini.
    -d <key=value>...            Define INI entry 'key' with value 'value'.
    -s                           Show information about skipped tests.
    --stop-on-fail               Stop execution upon the first failure.
    -j <num>                     Run <num> jobs in parallel (default: 8).
    -o <console|console-lines|tap|junit|log|none>  (e.g. -o junit:output.xml)
                                 Specify one or more output formats with optional file name.
    -w | --watch <path>          Watch directory.
    -i | --info                  Show tests environment info and exit.
    --setup <path>               Script for runner setup.
    --temp <path>                Path to temporary directory. Default by sys_get_temp_dir().
    --colors [1|0]               Enable or disable colors.
    --coverage <path>            Generate code coverage report to file.
    --coverage-src <path>        Path to source code.
    -h | --help                  This help.

-p <path>

Specifies the PHP binary that will be used to run tests. By default, it is php.

tester -p /home/user/php-7.2.0-beta/php-cgi tests

-c <path>

Specifies which php.ini will be used when running tests. By default, no php.ini is used. See Own php.ini for more information.

-C

A system-wide php.ini is used. So on UNIX platform, all the /etc/php/{sapi}/conf.d/*.ini files too. See Own php.ini section.

-d <key=value>

Sets the value of a PHP configuration directive for the tests. This parameter can be used multiple times.

tester -d max_execution_time=20

-s

Displays information about skipped tests.

--stop-on-fail

Tester stops testing upon the first failing test.

-j <num>

Specifies the number of parallel processes to run tests in. The default value is 8. To run tests sequentially, use the value 1.

-o <console|console-lines|tap|junit|log|none>

Sets the output format. The default is the console format. You can specify the name of the file into which the output will be written (e.g., -o junit:output.xml). The -o option can be repeated multiple times to generate multiple formats at once.

  • console: same as the default format, but the ASCII logo is not printed in this case
  • console-lines: similar to console, but the result of each test is listed on a separate line with additional information
  • tap: TAP format suitable for machine processing
  • junit: JUnit XML format, also suitable for machine processing
  • log: Outputs the testing progress. Includes all failed, skipped, and also successful tests
  • none: nothing is printed

-w | --watch <path>

After completing the tests, Tester does not exit but continues to run and watch PHP files in the specified directory. When a file changes, it runs the tests again. This parameter can be used multiple times if you want to watch multiple directories.

Useful when refactoring a library or debugging tests.

tester --watch src tests

-i | --info

Shows information about the runtime environment for tests. For example:

tester -p /usr/bin/php7.1 -c tests/php.ini --info

PHP binary:
/usr/bin/php7.1

PHP version:
7.1.7-1+0~20170711133844.5+jessie~1.gbp5284f4 (cli)

Code coverage engines:
(not available)

Loaded php.ini files:
/var/www/dev/demo/tests/php.ini

PHP temporary directory:
/tmp

Loaded extensions:
Core, ctype, date, dom, ereg, fileinfo, filter, hash, ...

--setup <path>

Tester loads the specified PHP script at startup. The variable Tester\Runner\Runner $runner is available within this script. Assume a file tests/runner-setup.php with the following content:

$runner->outputHandlers[] = new MyOutputHandler;

We run Tester with:

tester --setup tests/runner-setup.php tests

--temp <path>

Sets the path to the directory for Tester's temporary files. The default value is returned by sys_get_temp_dir(). You will be notified if the default value is not valid.

If you are unsure which directory is being used, run Tester with the --info parameter.

--colors 1|0

By default, Tester detects if the terminal supports colors and colorizes its output accordingly. This option overrides the auto-detection. You can set coloring globally using the NETTE_TESTER_COLORS environment variable.

--coverage <path>

Tester generates a report showing how much of the source code is covered by tests. This option requires the Xdebug or PCOV PHP extension to be installed, or PHP 7+ with the PHPDBG SAPI, which is faster. The extension of the target file determines its format: HTML or Clover XML.

tester tests --coverage coverage.html  # HTML report
tester tests --coverage coverage.xml   # Clover XML report

The priority for selecting the coverage engine is as follows:

  1. PCOV
  2. PHPDBG
  3. Xdebug

When using PHPDBG, extensive tests might fail due to memory exhaustion. Collecting code coverage information is memory-intensive. In this case, calling Tester\CodeCoverage\Collector::flush() within your test can help. It writes the collected data to disk and frees up memory. If data collection is not running or if Xdebug is used, the call has no effect.

See an `example HTML report with code coverage.

--coverage-src <path>

Used in conjunction with the --coverage option. <path> is the path to the source code for which the report is generated. Can be used multiple times.

Own php.ini

Tester runs PHP processes with the -n option, which means that no php.ini is loaded (not even those from /etc/php/conf.d/*.ini on UNIX systems). This ensures a consistent environment for running tests, but it also disables all external PHP extensions normally loaded by the system's PHP.

To preserve the loading of system php.ini files, use the -C parameter.

If you need specific extensions or special INI settings for your tests, we recommend creating your own php.ini file and distributing it with your tests. Then, run Tester with the -c option, for example, tester -c tests/php.ini tests. The INI file might look like this:

[PHP]

extension=php_pdo_mysql.dll
extension=php_pdo_pgsql.dll

memory_limit=512M

Running Tester on UNIX with a system php.ini, like tester -c /etc/php/cli/php.ini, does not load other INIs from /etc/php/conf.d/*.ini. This is a PHP behavior, not specific to Tester.