Installation¶
Install using pip:
pip install pytest-executable
Install using conda:
conda install pytest-executable -c conda-forge
Command line interface¶
The pytest command line shall be executed from the directory that contains the inputs root directory.
Plugin options¶
- --exe-runner PATH¶
use the shell script at PATH to run the executable.
This shell script may contain placeholders, such as {{output_path}} or others defined in the Runner section of a
test-settings.yaml
. A final runner shell script, with replaced placeholders, is written in the output directory of a test case ({{output_path}} is set to this path). This final script is then executed before any other test functions of a test case. See Runner fixture for further information.If this option is not defined then the runner shell script will not be executed, but all the other test functions will.
A typical runner shell script for running the executable with MPI could be:
#! /usr/bin/env bash env=/path/to/env/settings exe=/path/to/executable source $env mpirun -np {{nproc}} \ $exe \ --options \ 1> executable.stdout \ 2> executable.stderr
- --exe-output-root PATH¶
use PATH as the root for the output directory tree, default: tests-output
- --exe-overwrite-output¶
overwrite existing files in the output directories
- --exe-clean-output¶
clean the output directories before executing the tests
- --exe-regression-root PATH¶
use PATH as the root directory with the references for the regression testing, if omitted then the tests using the regression_path fixture will be skipped
- --exe-default-settings PATH¶
use PATH as the yaml file with the default test settings instead of the built-in ones
- --exe-test-module PATH¶
use PATH as the default test module instead of the built-in one
- --exe-report-generator PATH¶
use PATH as the script to generate the test report
See
generate_report.py
in the report-conf directory for an example of such a script.Note
The report generator script may require to install additional dependencies, such as sphinx, which are not install by the pytest-executable plugin.
Standard pytest options¶
You can get all the standard command line options of pytest by executing pytest -h. In particular, to run only some of the test cases in the inputs tree, or to execute only some of the test functions, you may use one of the following ways:
Use multiple path patterns¶
Instead of providing the path to the root of the inputs tree, you may provide the path to one or more of its sub-directories, for instance:
pytest --exe-runner <path/to/runner> <path/to/tests/inputs/sub-directory1> <path/to/tests/inputs/sub/sub/sub-directory2>
You may also use shell patterns (with *
and ?
characters) in the paths
like:
pytest --exe-runner <path/to/runner> <path/to/tests/inputs/*/sub-directory?>
Use marks¶
A test case could be assigned one or more marks in the test-settings.yaml
file, see
Marks section. Use the -m
to execute only the test cases that
match a given mark expression. A mark expression is a logical expression that
combines marks and yields a truth value. For example, to run only the tests
that have the mark1 mark but not the mark2 mark, use -m "mark1 and not
mark2"
. The logical operator or
could be used as well.
Use sub-string expression¶
Like the marks, any part (sub-string) of the name of a test case or of a test
function can be used to filter what will be executed. For instance to only
execute the tests that have the string transition
anywhere in their name, use
-k "transition"
. Or, to execute only the functions that have runner
in their names, use -k "runner"
. Logical expressions could be used to
combine more sub-strings as well.
Process last failed tests only¶
To only execute the tests that previously failed, use --last-failed
.
Show the markers¶
Use --markers
to show the available markers without executing the
tests.
Show the tests to be executed¶
Use --collect-only
to show the test cases and the test events
(functions) selected without executing them. You may combine this option with
other options, like the one above to filter the test cases.