Overview¶
Directory trees¶
The pytest-executable plugin deals with multiple directory trees:
the inputs
the outputs
the regression references
The inputs tree contains the files required to run an executable and to check its outcomes for different settings. It is composed of test cases as directories at the leaves of the tree. To create a test case, see Add a test case.
All the directory trees have the same hierarchy, this convention allows pytest-executable to work out what to test and what to check. The outputs tree is automatically created by pytest-executable, inside it, a test case directory typically contains:
symbolic links to the executable input files for the corresponding test case in the inputs tree
a runner shell script to execute executable
the files produced by the execution of executable
eventually, the files produced by the additional test modules
At the beginning, a regression reference tree is generally created from an existing outputs tree. In a regression references tree, a test case directory shall contain all the result files required for performing the comparisons for the regression testing. There can be more than one regression references trees for storing different sets of references, for instance for comparing the results against more than one version of executable.
Execution order¶
The pytest-executable plugin will reorder the execution such that the pytest tests are executed in the following order:
in a test case, the tests defined in the default test module (see
--exe-test-module
),any other tests defined in a test case directory, with pytest natural order,
any other tests defined in the parent directories of a test case.
The purposes of this order is to make sure that the runner shell script and the other default tests are executed first before the tests in other modules can be performed on the outcome of the executable. It also allows to create test modules in the parent directory of several test cases to gather their outcomes.
How to use¶
Run the executable only¶
pytest <path/to/tests/inputs> --exe-runner <path/to/runner> -k runner
This command will execute the executable for all the test cases that are found in
the input tree under path/to/tests/inputs
. A test case is identified by
a directory that contains a test-settings.yaml
file. For each of the test cases found,
pytest-executable will create an output directory with the same directory hierarchy and run
the cases in that output directory. By default, the root directory of the
output tree is tests-output
, this can be changed with the option
--exe-output-root
. Finally, the -k runner
option instructs
pytest to only execute the runner shell script and nothing more, see Standard pytest options for
more information on doing only some of the processing.
For instance, if the tests input tree contains:
path/to/tests/inputs
├── case-1
│ ├── input
│ └── test-settings.yaml
└── case-2
├── input
└── test-settings.yaml
Then the tests output tree is:
tests-output
├── case-1
│ ├── input -> path/to/tests/inputs/case-1/input
│ ├── output
│ ├── executable.stderr
│ ├── executable.stdout
│ ├── runner.sh
│ ├── runner.sh.stderr
│ └── runner.sh.stdout
└── case-2
├── input -> path/to/tests/inputs/case-2/input
├── output
├── executable.stderr
├── executable.stdout
├── runner.sh
├── runner.sh.stderr
└── runner.sh.stdout
For a given test case, for instance tests-output/case-1
,
the output directory contains:
- output
the output file produced by the execution of the executable, in practice there can be any number of output files and directories produced.
- input
a symbolic link to the file in the test input directory, in practice there can be any number of input files.
- executable.stderr
contains the error messages from the executable execution
- executable.stdout
contains the log messages from the executable execution
- runner.sh
a copy of the runner shell script defined with
--exe-runner
, eventually modified by pytest-executable for replacing the placeholders. Executing this script directly from a console shall produce the same results as when it is executed by pytest-executable. This script is intended to be as much as possible independent of the execution context such that it can be executed independently of pytest-executable in a reproducible way, i.e. it is self contained and does not depend on the shell context.- runner.sh.stderr
contains the error messages from the runner shell script execution
- runner.sh.stdout
contains the log messages from the runner shell script execution
If you need to manually run the executable for a test case, for debugging purposes for instance, just go to its output directory, for instance cd tests-output/case-1, and execute the runner shell script.
Check regressions without running the executable¶
pytest <path/to/tests/inputs> --exe-regression-root <path/to/tests/references> --exe-overwrite-output
We assume that the executable results have already been produced for the test cases
considered. This is not enough though because the output directory already
exists and pytest-executable will by default prevent the user from silently modifying any
existing test output directories. In that case, the option
--exe-overwrite-output
shall be used. The above command line will
compare the results in the default output tree with the references, if the
existing executable results are in a different directory then you need to add the
path to it with --exe-output-root
.
The option --exe-regression-root
points to the root directory with
the regression references tree . This tree shall have the same hierarchy as the
output tree but it only contains the results files that are used for doing the
regression checks.
Run the executable and do default regression checks¶
pytest <path/to/tests/inputs> --exe-runner <path/to/runner> --exe-regression-root <path/to/tests/references>
Note
Currently this can only be used when the executable execution is done on the same machine as the one that execute the regression checks, i.e. this will not work when the executable is executed on another machine.
Finally, checks are done on the executable log files to verify that the file
executable.stdout
exists and is not empty, and that the file
executable.stderr
exists and is empty.