Usage¶
The pytest-executable tool can be used in a wide variety of ways, the following sections explain how.
Run executable only¶
pytest --runner <path/to/runner> <path/to/tests/inputs> -k runner
This command will execute executable for all the test cases that are found in
the inputs tree under path/to/tests/inputs
. A test case is identified
by a directory that contains a test_case.yaml
file. For each of the test cases found,
pytest-executable will create an output directory with the same directory hierarchy and
run the cases in that output directory. By default, the root directory of the
output tree tests-output
, this can be changed with the option
--output-root
. Finally, the -k runner
option instructs
pytest to only execute the executable runner and nothing more, see
Standard pytest options for more informations on doing only some of the processing.
For instance, if the test inputs tree contains:
path/to/tests/inputs
├── dir-1
│ ├── case.input
│ └── test_case.yaml
└── dir-2
├── case.input
└── test_case.yaml
Then the output tree is:
tests-output
├── dir-1
│ ├── case.input -> path/to/tests/inputs/dir-1/case.input
│ ├── case.output
│ ├── executable.stderr
│ ├── executable.stdout
│ ├── run_executable.sh
│ ├── run_executable.stderr
│ └── run_executable.stdout
├── dir-2
├── case.input -> path/to/tests/inputs/dir-2/case.input
├── case.output
├── executable.stderr
├── executable.stdout
├── run_executable.sh
├── run_executable.stderr
└── run_executable.stdout
For a given test case, for instance tests-output/dir-1
,
the output directory contains:
- case.output
the output file produced by the execution of executable, in practice there can be any number of ouput files and directories produced.
- case.input
a symbolic link to the file in the test case input directory, in pratice there can be any number of input files.
- executable.stderr
contains the error messages from the executable execution
- executable.stdout
contains the log messages from the executable execution
- run_executable.sh
Executing this script directly from a console shall produce the same results as when it is executed by pytest-executable. This script is intended to be as much as possible independent of the execution context such that it can be executed independently of pytest-executable in a reproductible way, i.e. it is self contained and does not depend on the shell context.
run_executable.stderr
contains the error messages from therun_executable.sh
execution- run_executable.stdout
contains the log messages from the
run_executable.sh
execution
If you need to manually run executable for a test case, for debugging purposes
for instance, just go to its output directory, for instance cd
tests-output/dir-1, and execute run_executable.sh
.
Do default regression checking without running executable¶
pytest --regression-root <path/to/tests/references> <path/to/tests/inputs> --overwrite-output
We assume that executable results have already been produced for the test cases
considered. This is not enough though because the output directory already
exists and pytest-executable will by default prevent the user from silently modifying
any existing test output directories. In that case, the option
--overwrite-output
shall be used. The above command line will compare
the results in the default output tree with the references, if the existing
executable results are in a different directory then you need to add the path
to it with --output-root.
The option --regression-root
points to the root directory with the
regression references tree . This tree shall have the same hierarchy as the
output tree but it only contains the results files that are used for doing the
regression checks.
Run executable and do default regression checks¶
pytest --runner <path/to/runner> --regression-root <path/to/tests/references> <path/to/tests/inputs>
Note
Currently this can only be used when executable execution is done on the same machine as the one that execute the regression checks, i.e. this will not work when executable is submitted through a job scheduler.
Finally, checks are done on the executable log files to verify that the file
executable.stdout
exists and is not empty, and that the file
executable.stderr
exists and is empty.