diff --git a/doc/conf.py b/doc/conf.py index a6f6a1af3b..c0ce3c0723 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -25,6 +25,9 @@ sys.path.insert(0, str(ZEPHYR_BASE / "doc" / "_scripts")) # for autodoc directives on runners.xyz. sys.path.insert(0, str(ZEPHYR_BASE / "scripts" / "west_commands")) +# Add the directory which contains the pytest-twister-pytest +sys.path.insert(0, str(ZEPHYR_BASE / "scripts" / "pylib" / "pytest-twister-harness" / "src")) + import redirects try: diff --git a/doc/develop/test/pytest.rst b/doc/develop/test/pytest.rst index f6ba54fa52..8edcc67f0a 100644 --- a/doc/develop/test/pytest.rst +++ b/doc/develop/test/pytest.rst @@ -1,4 +1,4 @@ -.. integration-with-pytest: +.. _integration_with_pytest: Integration with pytest test framework ###################################### @@ -46,43 +46,71 @@ sets the test result accordingly. How to create a pytest test *************************** -An example of a pytest test is given at :zephyr_file:`samples/subsys/testsuite/pytest/shell/pytest/test_shell.py`. -Twister calls pytest for each configuration from the .yaml file which uses ``harness: pytest``. -By default, it points to ``pytest`` directory, located next to a directory with binary sources. -A keyword ``pytest_root`` placed under ``harness_config`` section can be used to point to other -files, directories or subtests. +An example folder containing a pytest test, application source code and Twister configuration .yaml +file can look like the following: + +.. code-block:: none + + test_foo/ + ├─── pytest/ + │ └─── test_foo.py + ├─── src/ + │ └─── main.c + ├─── CMakeList.txt + ├─── prj.conf + └─── testcase.yaml + +An example of a pytest test is given at +:zephyr_file:`samples/subsys/testsuite/pytest/shell/pytest/test_shell.py`. Using the configuration +provided in the ``testcase.yaml`` file, Twister builds the application from ``src`` and then, if the +.yaml file contains a ``harness: pytest`` entry, it calls pytest in a separate subprocess. A sample +configuration file may look like this: + +.. code-block:: yaml + + tests: + some.foo.test: + harness: pytest + tags: foo + +By default, pytest tries to look for tests in a ``pytest`` directory located next to a directory +with binary sources. A keyword ``pytest_root`` placed under ``harness_config`` section in .yaml file +can be used to point to other files, directories or subtests (more info :ref:`here `). Pytest scans the given locations looking for tests, following its default -`discovery rules `_ -One can also pass some extra arguments to the pytest from yaml file using ``pytest_args`` keyword -under ``harness_config``, e.g.: ``pytest_args: [‘-k=test_method’, ‘--log-level=DEBUG’]``. -There is also an option to pass ``--pytest-args`` through Twister command line parameters. -This can be particularly useful when one wants to select a specific testcase from a test suite. -For instance, one can use a command: +`discovery rules `_. -.. code-block:: console +Passing extra arguments +======================= - $ ./scripts/twister --platform native_sim -T samples/subsys/testsuite/pytest/shell \ - -s samples/subsys/testsuite/pytest/shell/sample.pytest.shell \ - --pytest-args='-k test_shell_print_version' +There are two ways for passing extra arguments to the called pytest subprocess: + +#. From .yaml file, using ``pytest_args`` placed under ``harness_config`` section - more info + :ref:`here `. +#. Through Twister command line interface as ``--pytest-args`` argument. This can be particularly + useful when one wants to select a specific testcase from a test suite. For instance, one can use + a command: + + .. code-block:: console + + $ ./scripts/twister --platform native_sim -T samples/subsys/testsuite/pytest/shell \ + -s samples/subsys/testsuite/pytest/shell/sample.pytest.shell \ + --pytest-args='-k test_shell_print_version' -Note that ``--pytest-args`` can be passed multiple times to pass several arguments to the pytest. - -Helpers & fixtures -================== +Fixtures +******** dut ---- +=== -Give access to a DeviceAdapter type object, that represents Device Under Test. -This fixture is the core of pytest harness plugin. It is required to launch -DUT (initialize logging, flash device, connect serial etc). -This fixture yields a device prepared according to the requested type -(``native``, ``qemu``, ``hardware``, etc.). All types of devices share the same API. -This allows for writing tests which are device-type-agnostic. -Scope of this fixture is determined by the ``pytest_dut_scope`` -keyword placed under ``harness_config`` section. +Give access to a `DeviceAdapter`_ type object, that represents Device Under Test. This fixture is +the core of pytest harness plugin. It is required to launch DUT (initialize logging, flash device, +connect serial etc). This fixture yields a device prepared according to the requested type +(``native``, ``qemu``, ``hardware``, etc.). All types of devices share the same API. This allows for +writing tests which are device-type-agnostic. Scope of this fixture is determined by the +``pytest_dut_scope`` keyword placed under ``harness_config`` section (more info +:ref:`here `). .. code-block:: python @@ -93,13 +121,14 @@ keyword placed under ``harness_config`` section. dut.readlines_until('Hello world') shell ------ +===== -Provide an object with methods used to interact with shell application. -It calls ``wait_for_promt`` method, to not start scenario until DUT is ready. -Note that it uses ``dut`` fixture, so ``dut`` can be skipped when ``shell`` is used. -Scope of this fixture is determined by the ``pytest_dut_scope`` -keyword placed under ``harness_config`` section. +Provide a `Shell `_ class object with methods used to interact with shell application. +It calls ``wait_for_promt`` method, to not start scenario until DUT is ready. The shell fixture +calls ``dut`` fixture, hence has access to all its methods. The ``shell`` fixture adds methods +optimized for interactions with a shell. It can be used instead of ``dut`` for tests. Scope of this +fixture is determined by the ``pytest_dut_scope`` keyword placed under ``harness_config`` section +(more info :ref:`here `). .. code-block:: python @@ -109,16 +138,16 @@ keyword placed under ``harness_config`` section. shell.exec_command('help') mcumgr ------- +====== -Sample fixture to wrap ``mcumgr`` command-line tool used to manage remote devices. -More information about MCUmgr can be found here :ref:`mcu_mgr`. +Sample fixture to wrap ``mcumgr`` command-line tool used to manage remote devices. More information +about MCUmgr can be found here :ref:`mcu_mgr`. .. note:: This fixture requires the ``mcumgr`` available in the system PATH -Only selected functionality of MCUmgr is wrapped by this fixture. -For example, here is a test with a fixture ``mcumgr`` +Only selected functionality of MCUmgr is wrapped by this fixture. For example, here is a test with +a fixture ``mcumgr`` .. code-block:: python @@ -137,6 +166,132 @@ For example, here is a test with a fixture ``mcumgr`` mcumgr.reset_device() # continue test scenario, check version etc. +Classes +******* + +DeviceAdapter +============= + +.. autoclass:: twister_harness.DeviceAdapter + + .. automethod:: launch + + .. automethod:: connect + + .. automethod:: readline + + .. automethod:: readlines + + .. automethod:: readlines_until + + .. automethod:: write + + .. automethod:: disconnect + + .. automethod:: close + +.. _shell_class: + +Shell +===== + +.. autoclass:: twister_harness.Shell + + .. automethod:: exec_command + + .. automethod:: wait_for_prompt + + +Examples of pytest tests in the Zephyr project +********************************************** + +* Shell demo - :zephyr_file:`samples/subsys/testsuite/pytest/shell` +* MCUmgr tests - :zephyr_file:`tests/boot/with_mcumgr` +* LwM2M tests - :zephyr_file:`tests/net/lib/lwm2m/interop` +* GDB stub tests - :zephyr_file:`tests/subsys/debug/gdbstub` + + +FAQ +*** + +How to flash/run application only once per pytest session? +========================================================== + + ``dut`` is a fixture responsible for flashing/running application. By default, its scope is set + as ``function``. This can be changed by adding to .yaml file ``pytest_dut_scope`` keyword placed + under ``harness_config`` section: + + .. code-block:: yaml + + harness: pytest + harness_config: + pytest_dut_scope: session + + More info can be found :ref:`here `. + +How to run only one particular test from a python file? +======================================================= + + This can be achieved in several ways. In .yaml file it can be added using a ``pytest_root`` entry + placed under ``harness_config`` with list of tests which should be run: + + .. code-block:: yaml + + harness: pytest + harness_config: + pytest_root: + - "pytest/test_shell.py::test_shell_print_help" + + Particular tests can be also chosen by pytest ``-k`` option (more info about pytest keyword + filter can be found + `here `_ + ). It can be applied by adding ``-k`` filter in ``pytest_args`` in .yaml file: + + .. code-block:: yaml + + harness: pytest + harness_config: + pytest_args: + - "-k test_shell_print_help" + + or by adding it to Twister command overriding parameters from the .yaml file: + + .. code-block:: console + + $ ./scripts/twister ... --pytest-args='-k test_shell_print_help' + +How to get information about used device type in test? +====================================================== + + This can be taken from ``dut`` fixture (which represents `DeviceAdapter`_ object): + + .. code-block:: python + + device_type: str = dut.device_config.type + if device_type == 'hardware': + ... + elif device_type == 'native': + ... + +How to rerun locally pytest tests without rebuilding application by Twister? +============================================================================ + + This can be achieved by running Twister once again with ``--test-only`` argument added to Twister + command. Another way is running Twister with highest verbosity level (``-vv``) and then + copy-pasting from logs command dedicated for spawning pytest (log started by ``Running pytest + command: ...``). + +Is this possible to run pytest tests in parallel? +================================================= + + Basically ``pytest-harness-plugin`` wasn't written with intention of running pytest tests in + parallel. Especially those one dedicated for hardware. There was assumption that parallelization + of tests is made by Twister, and it is responsible for managing available sources (jobs and + hardwares). If anyone is interested in doing this for some reasons (for example via + `pytest-xdist plugin `_) they do so at their own + risk. + + Limitations *********** diff --git a/doc/develop/test/twister.rst b/doc/develop/test/twister.rst index 4677244725..0866653a63 100644 --- a/doc/develop/test/twister.rst +++ b/doc/develop/test/twister.rst @@ -514,14 +514,36 @@ harness_config: Only one fixture can be defined per testcase and the fixture name has to be unique across all tests in the test suite. +.. _pytest_root: + pytest_root: (default pytest) - Specify a list of pytest directories, files or subtests that need to be executed - when test case begin to running, default pytest directory is pytest. - After pytest finished, twister will check if this case pass or fail according - to the pytest report. + Specify a list of pytest directories, files or subtests that need to be + executed when a test case begins to run. The default pytest directory is + ``pytest``. After the pytest run is finished, Twister will check if + the test case passed or failed according to the pytest report. + As an example, a list of valid pytest roots is presented below: + + .. code-block:: yaml + + harness_config: + pytest_root: + - "pytest/test_shell_help.py" + - "../shell/pytest/test_shell.py" + - "/tmp/test_shell.py" + - "~/tmp/test_shell.py" + - "$ZEPHYR_BASE/samples/subsys/testsuite/pytest/shell/pytest/test_shell.py" + - "pytest/test_shell_help.py::test_shell2_sample" # select pytest subtest + - "pytest/test_shell_help.py::test_shell2_sample[param_a]" # select pytest parametrized subtest + +.. _pytest_args: pytest_args: (default empty) - Specify a list of additional arguments to pass to ``pytest``. + Specify a list of additional arguments to pass to ``pytest`` e.g.: + ``pytest_args: [‘-k=test_method’, ‘--log-level=DEBUG’]``. Note that + ``--pytest-args`` can be passed multiple times to pass several arguments + to the pytest. + +.. _pytest_dut_scope: pytest_dut_scope: (default function) The scope for which ``dut`` and ``shell`` pytest fixtures are shared. diff --git a/doc/requirements.txt b/doc/requirements.txt index 45bb461e51..e7747b5f13 100644 --- a/doc/requirements.txt +++ b/doc/requirements.txt @@ -13,3 +13,6 @@ sphinx-togglebutton # YAML validation. Used by zephyr_module. PyYAML>=5.1 pykwalify + +# Used by pytest-twister-harness plugin +pytest diff --git a/scripts/pylib/pytest-twister-harness/src/twister_harness/device/device_adapter.py b/scripts/pylib/pytest-twister-harness/src/twister_harness/device/device_adapter.py index 59be6c52af..ce8492c78b 100644 --- a/scripts/pylib/pytest-twister-harness/src/twister_harness/device/device_adapter.py +++ b/scripts/pylib/pytest-twister-harness/src/twister_harness/device/device_adapter.py @@ -25,7 +25,11 @@ logger = logging.getLogger(__name__) class DeviceAdapter(abc.ABC): - """Class defines an interface for all devices.""" + """ + This class defines a common interface for all device types (hardware, + simulator, QEMU) used in tests to gathering device output and send data to + it. + """ def __init__(self, device_config: DeviceConfig) -> None: """ @@ -54,8 +58,8 @@ class DeviceAdapter(abc.ABC): def launch(self) -> None: """ Start by closing previously running application (no effect if not - needed). Then, flash and run test application. Finally, start a reader - thread capturing an output from a device. + needed). Then, flash and run test application. Finally, start an + internal reader thread capturing an output from a device. """ self.close() self._clear_internal_resources() @@ -100,7 +104,7 @@ class DeviceAdapter(abc.ABC): def readline(self, timeout: float | None = None, print_output: bool = True) -> str: """ Read line from device output. If timeout is not provided, then use - base_timeout + base_timeout. """ timeout = timeout or self.base_timeout if self.is_device_connected() or not self._device_read_queue.empty(): @@ -125,13 +129,13 @@ class DeviceAdapter(abc.ABC): until following conditions: 1. If regex is provided - read until regex regex is found in read - line (or until timeout) + line (or until timeout). 2. If num_of_lines is provided - read until number of read lines is - equal to num_of_lines (or until timeout) + equal to num_of_lines (or until timeout). 3. If none of above is provided - return immediately lines collected so - far in internal queue + far in internal buffer. - If timeout is not provided, then use base_timeout + If timeout is not provided, then use base_timeout. """ timeout = timeout or self.base_timeout if regex: diff --git a/scripts/pylib/pytest-twister-harness/src/twister_harness/helpers/shell.py b/scripts/pylib/pytest-twister-harness/src/twister_harness/helpers/shell.py index f6eb807445..64f1cac63f 100644 --- a/scripts/pylib/pytest-twister-harness/src/twister_harness/helpers/shell.py +++ b/scripts/pylib/pytest-twister-harness/src/twister_harness/helpers/shell.py @@ -55,7 +55,7 @@ class Shell: Send shell command to a device and return response. Passed command is extended by double enter sings - first one to execute this command on a device, second one to receive next prompt what is a signal that - execution was finished. + execution was finished. Method returns printout of the executed command. """ timeout = timeout or self.base_timeout command_ext = f'{command}\n\n' diff --git a/tests/boot/with_mcumgr/README.rst b/tests/boot/with_mcumgr/README.rst index 5f0ef82d30..f3281f57c5 100644 --- a/tests/boot/with_mcumgr/README.rst +++ b/tests/boot/with_mcumgr/README.rst @@ -3,7 +3,7 @@ Upgrade testing with MCUmgr This application is based on :ref:`smp_svr_sample`. It is built using **sysbuild**. Tests are automated with pytest, a new harness of Twister -(more information can be found here :ref:`integration-with-pytest`) +(more information can be found :ref:`here `) .. note:: Pytest uses the MCUmgr fixture which requires the ``mcumgr`` available