Running Tests

The most visible part of Nette Tester is the command-line test runner. It is extremely fast and robust because it starts automatically all tests as separate processes in parallel in multiple threads. It can also run itself in the so-called watch mode.

The Nette Tester test runner is invoked from the command-line. As a parameter, we will pass the test directory. For the current directory just enter a dot:

vendor/bin/tester .

When invoked, the test runner will scan the specified directory and all subdirectories and looks for tests, which are the *.phpt and *Test.php files. It also reads and evaluates their annotations to know which ones and how to run them.

It will then execute the tests. For each done test, the runner prints one character to indicate progress:

  • . – test passed
  • s – test has been skipped
  • F – test failed

The output may look like:

 _____ ___  ___ _____ ___  ___
|_   _/ __)( __/_   _/ __)| _ )
  |_| \___ /___) |_| \___ |_|_\  v2.5.2

Note: No php.ini is used.
PHP 8.3.2 (cli) | php -n | 8 threads

........s..........................

OK (35 tests, 1 skipped, 1.7 seconds)

When you run it again, it first runs tests that failed during the previous run, so you'll know right away if you fixed the error.

Tester's exit code is zero if no one fails. Non-zero otherwise.

The Tester runs PHP processes without php.ini. More details in the Own php.ini section.

Command-Line Options

We obtain command-line options overview by running the Tester without parameters or with option -h:

 _____ ___  ___ _____ ___  ___
|_   _/ __)( __/_   _/ __)| _ )
  |_| \___ /___) |_| \___ |_|_\  v2.5.2

Usage:
    tester [options] [<test file> | <directory>]...

Options:
    -p <path>                    Specify PHP interpreter to run (default: php).
    -c <path>                    Look for php.ini file (or look in directory) <path>.
    -C                           Use system-wide php.ini.
    -d <key=value>...            Define INI entry 'key' with value 'value'.
    -s                           Show information about skipped tests.
    --stop-on-fail               Stop execution upon the first failure.
    -j <num>                     Run <num> jobs in parallel (default: 8).
    -o <console|console-lines|tap|junit|log|none>  (e.g. -o junit:output.xml)
                                 Specify one or more output formats with optional file name.
    -w | --watch <path>          Watch directory.
    -i | --info                  Show tests environment info and exit.
    --setup <path>               Script for runner setup.
    --temp <path>                Path to temporary directory. Default by sys_get_temp_dir().
    --colors [1|0]               Enable or disable colors.
    --coverage <path>            Generate code coverage report to file.
    --coverage-src <path>        Path to source code.
    -h | --help                  This help.

-p <path>

Specifies the PHP binary that will be used to run tests. By default it is php.

tester -p /home/user/php-7.2.0-beta/php-cgi tests

-c <path>

Specifies which php.ini will be used when running tests. By default, no php.ini is used. See Own php.ini for more information.

-C

A system-wide php.ini is used. So on UNIX platform, all the /etc/php/{sapi}/conf.d/*.ini files too. See Own php.ini section.

-d <key=value>

Sets the value of the PHP configuration directive for tests. The parameter can be used multiple times.

tester -d max_execution_time=20

-s

Information about skipped tests will be shown.

--stop-on-fail

Tester stops testing upon the first failing test.

-j <num>

Tests run in a <num> parallel precesses. Default value is 8. If we wish to run tests in series, we use value 1.

-o <console|console-lines|tap|junit|log|none>

Output format. Default is the console format. You can specify the name of the file into which the output will be written (e.g., -o junit:output.xml). The -o option can be repeated multiple times to generate multiple formats at once.

  • console: the same as default, but the ASCII logo is not printed in this case
  • console-lines: similar to the console, but the result of each test is listed on a separate line with more information
  • tap: TAP format appropriate for machine processing
  • junit: JUnit XML format, appropriate for machine processing too
  • log: Outputs testing progress. All failed, skipped and also successful tests
  • none: nothing is printed

-w | --watch <path>

The Tester doesn't end after tests are completed but it keeps running and watching PHP files in given directory. When changed, it runs the tests again. Parameter can be used multiple times if we want to monitor multiple directories.

It is handy during refactoring a library or tests debugging.

tester --watch src tests

-i | --info

It shows information about a test running environment. For example:

tester -p /usr/bin/php7.1 -c tests/php.ini --info

PHP binary:
/usr/bin/php7.1

PHP version:
7.1.7-1+0~20170711133844.5+jessie~1.gbp5284f4 (cli)

Code coverage engines:
(not available)

Loaded php.ini files:
/var/www/dev/demo/tests/php.ini

PHP temporary directory:
/tmp

Loaded extensions:
Core, ctype, date, dom, ereg, fileinfo, filter, hash, ...

--setup <path>

The Tester loads given PHP script on start. Variable Tester\Runner\Runner $runner is available in it. Let's assume file tests/runner-setup.php:

$runner->outputHandlers[] = new MyOutputHandler;

and we run the Tester:

tester --setup tests/runner-setup.php tests

--temp <path>

Sets a path to directory for temporary files of Tester. Default value is returned by sys_get_temp_dir(). When default value is not valid, you will be noticed.

If we are not sure which directory is used, we can run Tester with the --info parameter.

--colors 1|0

The Tester detects a color-able terminal in default and colorizes its output. This option is over the autodetection. We can set coloring globally by a system environment variable NETTE_TESTER_COLORS.

--coverage <path>

Tester will generate a report with overview how much is the source code coveraged by tests. This option requires PHP extension Xdebug or PCOV enabled, or PHP 7 with the PHPDBG SAPI, which is faster. The destination file extension determines the format of the content. HTML or Clover XML.

tester tests --coverage coverage.html  # HTML report
tester tests --coverage coverage.xml   # Clover XML report

Priority to choose collecting mechanism is following:

  1. PCOV
  2. PHPDBG
  3. Xdebug

Extensive tests may fail during run by PHPDBG due to memory exhaustion. Coverage data collecting is memory consuming operation. In that case, calling the Tester\CodeCoverage\Collector::flush() inside a test may help. It will flush collected data into file and frees memory. When data collection is not in progress, or Xdebug is used, calling has no effect.

An example of HTML report with code coverage.

--coverage-src <path>

We use it with the option --coverage simultaneously. The <path> is a path to source code for which we generate the report. It can be used repeatedly.

Own php.ini

The Tester runs PHP processes with -n option, which means no php.ini is loaded (not even that from /etc/php/conf.d/*.ini in UNIX). It ensures the same environment for the tests run, but it also deactivates all the external PHP extensions commonly loaded by system PHP.

If you want to keep system configuration, use the -C parameter.

If you need some extensions or some special INI settings, we recommend to create own php.ini file and distribute it among the tests. Then we run Tester with -c option, e.g. tester -c tests/php.ini. The INI file may look like:

[PHP]

extension=php_pdo_mysql.dll
extension=php_pdo_pgsql.dll

memory_limit=512M

Running the Tester with a system php.ini in UNIX, e.g. tester -c /etc/php/cgi/php.ini, does not load other INI from /etc/php/conf.d/*.ini. That's the PHP behavior, not the Tester's.