doc: test: improve pytest documentation

Improve documentation about pytest integration with Twister. Add
examples of usage, improve description of available options and
introduce automatic doc generation of two plugin classes (DeviceAdapter
and Shell) basing on their docstrings from source code.

Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>

doc/conf.py

@@ -25,6 +25,9 @@ sys.path.insert(0, str(ZEPHYR_BASE / "doc" / "_scripts"))
 # for autodoc directives on runners.xyz.
 sys.path.insert(0, str(ZEPHYR_BASE / "scripts" / "west_commands"))
 
+# Add the directory which contains the pytest-twister-pytest
+sys.path.insert(0, str(ZEPHYR_BASE / "scripts" / "pylib" / "pytest-twister-harness" / "src"))
+
 import redirects
 
 try:

doc/develop/test/pytest.rst

@@ -1,4 +1,4 @@
-.. integration-with-pytest:
+.. _integration_with_pytest:
 
 Integration with pytest test framework
 ######################################
@@ -46,43 +46,71 @@ sets the test result accordingly.
 How to create a pytest test
 ***************************
 
-An example of a pytest test is given at :zephyr_file:`samples/subsys/testsuite/pytest/shell/pytest/test_shell.py`.
+An example folder containing a pytest test, application source code and Twister configuration .yaml
-Twister calls pytest for each configuration from the .yaml file which uses ``harness: pytest``.
+file can look like the following:
-By default, it points to ``pytest`` directory, located next to a directory with binary sources.
+
-A keyword ``pytest_root`` placed under ``harness_config`` section can be used to point to other
+.. code-block:: none
-files, directories or subtests.
+
+   test_foo/
+   ├─── pytest/
+   │    └─── test_foo.py
+   ├─── src/
+   │    └─── main.c
+   ├─── CMakeList.txt
+   ├─── prj.conf
+   └─── testcase.yaml
+
+An example of a pytest test is given at
+:zephyr_file:`samples/subsys/testsuite/pytest/shell/pytest/test_shell.py`. Using the configuration
+provided in the ``testcase.yaml`` file, Twister builds the application from ``src`` and then, if the
+.yaml file contains a ``harness: pytest`` entry, it calls pytest in a separate subprocess. A sample
+configuration file may look like this:
+
+.. code-block:: yaml
+
+   tests:
+      some.foo.test:
+         harness: pytest
+         tags: foo
+
+By default, pytest tries to look for tests in a ``pytest`` directory located next to a directory
+with binary sources. A keyword ``pytest_root`` placed under ``harness_config`` section in .yaml file
+can be used to point to other files, directories or subtests (more info :ref:`here <pytest_root>`).
 
 Pytest scans the given locations looking for tests, following its default
-`discovery rules <https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#conventions-for-python-test-discovery>`_
+`discovery rules <https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#conventions-for-python-test-discovery>`_.
-One can also pass some extra arguments to the pytest from yaml file using ``pytest_args`` keyword
-under ``harness_config``, e.g.: ``pytest_args: [‘-k=test_method’, ‘--log-level=DEBUG’]``.
-There is also an option to pass ``--pytest-args`` through Twister command line parameters.
-This can be particularly useful when one wants to select a specific testcase from a test suite.
-For instance, one can use a command:
 
-.. code-block:: console
+Passing extra arguments
+=======================
 
-   $ ./scripts/twister --platform native_sim -T samples/subsys/testsuite/pytest/shell \
+There are two ways for passing extra arguments to the called pytest subprocess:
-   -s samples/subsys/testsuite/pytest/shell/sample.pytest.shell \
+
-   --pytest-args='-k test_shell_print_version'
+#. From .yaml file, using ``pytest_args`` placed under ``harness_config`` section - more info
+   :ref:`here <pytest_args>`.
+#. Through Twister command line interface as ``--pytest-args`` argument. This can be particularly
+   useful when one wants to select a specific testcase from a test suite. For instance, one can use
+   a command:
+
+   .. code-block:: console
+
+      $ ./scripts/twister --platform native_sim -T samples/subsys/testsuite/pytest/shell \
+      -s samples/subsys/testsuite/pytest/shell/sample.pytest.shell \
+      --pytest-args='-k test_shell_print_version'
 
 
-Note that ``--pytest-args`` can be passed multiple times to pass several arguments to the pytest.
+Fixtures
-
+********
-Helpers & fixtures
-==================
 
 dut
----
+===
 
-Give access to a DeviceAdapter type object, that represents Device Under Test.
+Give access to a `DeviceAdapter`_ type object, that represents Device Under Test. This fixture is
-This fixture is the core of pytest harness plugin. It is required to launch
+the core of pytest harness plugin. It is required to launch DUT (initialize logging, flash device,
-DUT (initialize logging, flash device, connect serial etc).
+connect serial etc). This fixture yields a device prepared according to the requested type
-This fixture yields a device prepared according to the requested type
+(``native``, ``qemu``, ``hardware``, etc.). All types of devices share the same API. This allows for
-(``native``, ``qemu``, ``hardware``, etc.). All types of devices share the same API.
+writing tests which are device-type-agnostic. Scope of this fixture is determined by the
-This allows for writing tests which are device-type-agnostic.
+``pytest_dut_scope`` keyword placed under ``harness_config`` section (more info
-Scope of this fixture is determined by the ``pytest_dut_scope``
+:ref:`here <pytest_dut_scope>`).
-keyword placed under ``harness_config`` section.
 
 
 .. code-block:: python
@@ -93,13 +121,14 @@ keyword placed under ``harness_config`` section.
       dut.readlines_until('Hello world')
 
 shell
------
+=====
 
-Provide an object with methods used to interact with shell application.
+Provide a `Shell <shell_class_>`_ class object with methods used to interact with shell application.
-It calls ``wait_for_promt`` method, to not start scenario until DUT is ready.
+It calls ``wait_for_promt`` method, to not start scenario until DUT is ready. The shell fixture
-Note that it uses ``dut`` fixture, so ``dut`` can be skipped when ``shell`` is used.
+calls ``dut`` fixture, hence has access to all its methods. The ``shell`` fixture adds methods
-Scope of this fixture is determined by the ``pytest_dut_scope``
+optimized for interactions with a shell. It can be used instead of ``dut`` for tests. Scope of this
-keyword placed under ``harness_config`` section.
+fixture is determined by the ``pytest_dut_scope`` keyword placed under ``harness_config`` section
+(more info :ref:`here <pytest_dut_scope>`).
 
 .. code-block:: python
 
@@ -109,16 +138,16 @@ keyword placed under ``harness_config`` section.
       shell.exec_command('help')
 
 mcumgr
-------
+======
 
-Sample fixture to wrap ``mcumgr`` command-line tool used to manage remote devices.
+Sample fixture to wrap ``mcumgr`` command-line tool used to manage remote devices. More information
-More information about MCUmgr can be found here :ref:`mcu_mgr`.
+about MCUmgr can be found here :ref:`mcu_mgr`.
 
 .. note::
    This fixture requires the ``mcumgr`` available in the system PATH
 
-Only selected functionality of MCUmgr is wrapped by this fixture.
+Only selected functionality of MCUmgr is wrapped by this fixture. For example, here is a test with
-For example, here is a test with a fixture ``mcumgr``
+a fixture ``mcumgr``
 
 .. code-block:: python
 
@@ -137,6 +166,132 @@ For example, here is a test with a fixture ``mcumgr``
       mcumgr.reset_device()
       # continue test scenario, check version etc.
 
+Classes
+*******
+
+DeviceAdapter
+=============
+
+.. autoclass:: twister_harness.DeviceAdapter
+
+   .. automethod:: launch
+
+   .. automethod:: connect
+
+   .. automethod:: readline
+
+   .. automethod:: readlines
+
+   .. automethod:: readlines_until
+
+   .. automethod:: write
+
+   .. automethod:: disconnect
+
+   .. automethod:: close
+
+.. _shell_class:
+
+Shell
+=====
+
+.. autoclass:: twister_harness.Shell
+
+   .. automethod:: exec_command
+
+   .. automethod:: wait_for_prompt
+
+
+Examples of pytest tests in the Zephyr project
+**********************************************
+
+* Shell demo - :zephyr_file:`samples/subsys/testsuite/pytest/shell`
+* MCUmgr tests - :zephyr_file:`tests/boot/with_mcumgr`
+* LwM2M tests - :zephyr_file:`tests/net/lib/lwm2m/interop`
+* GDB stub tests - :zephyr_file:`tests/subsys/debug/gdbstub`
+
+
+FAQ
+***
+
+How to flash/run application only once per pytest session?
+==========================================================
+
+   ``dut`` is a fixture responsible for flashing/running application. By default, its scope is set
+   as ``function``. This can be changed by adding to .yaml file ``pytest_dut_scope`` keyword placed
+   under ``harness_config`` section:
+
+   .. code-block:: yaml
+
+      harness: pytest
+      harness_config:
+         pytest_dut_scope: session
+
+   More info can be found :ref:`here <pytest_dut_scope>`.
+
+How to run only one particular test from a python file?
+=======================================================
+
+   This can be achieved in several ways. In .yaml file it can be added using a ``pytest_root`` entry
+   placed under ``harness_config`` with list of tests which should be run:
+
+   .. code-block:: yaml
+
+      harness: pytest
+      harness_config:
+         pytest_root:
+            - "pytest/test_shell.py::test_shell_print_help"
+
+   Particular tests can be also chosen by pytest ``-k`` option (more info about pytest keyword
+   filter can be found
+   `here <https://docs.pytest.org/en/latest/example/markers.html#using-k-expr-to-select-tests-based-on-their-name>`_
+   ). It can be applied by adding ``-k`` filter in ``pytest_args`` in .yaml file:
+
+   .. code-block:: yaml
+
+      harness: pytest
+      harness_config:
+         pytest_args:
+            - "-k test_shell_print_help"
+
+   or by adding it to Twister command overriding parameters from the .yaml file:
+
+   .. code-block:: console
+
+      $ ./scripts/twister ... --pytest-args='-k test_shell_print_help'
+
+How to get information about used device type in test?
+======================================================
+
+   This can be taken from ``dut`` fixture (which represents `DeviceAdapter`_ object):
+
+   .. code-block:: python
+
+      device_type: str = dut.device_config.type
+      if device_type == 'hardware':
+         ...
+      elif device_type == 'native':
+         ...
+
+How to rerun locally pytest tests without rebuilding application by Twister?
+============================================================================
+
+   This can be achieved by running Twister once again with ``--test-only`` argument added to Twister
+   command. Another way is running Twister with highest verbosity level (``-vv``) and then
+   copy-pasting from logs command dedicated for spawning pytest (log started by ``Running pytest
+   command: ...``).
+
+Is this possible to run pytest tests in parallel?
+=================================================
+
+   Basically ``pytest-harness-plugin`` wasn't written with intention of running pytest tests in
+   parallel. Especially those one dedicated for hardware. There was assumption that parallelization
+   of tests is made by Twister, and it is responsible for managing available sources (jobs and
+   hardwares). If anyone is interested in doing this for some reasons (for example via
+   `pytest-xdist plugin <https://pytest-xdist.readthedocs.io/en/stable/>`_) they do so at their own
+   risk.
+
+
 Limitations
 ***********
 

doc/develop/test/twister.rst

@@ -514,14 +514,36 @@ harness_config: <harness configuration options>
         Only one fixture can be defined per testcase and the fixture name has to
         be unique across all tests in the test suite.
 
+.. _pytest_root:
+
     pytest_root: <list of pytest testpaths> (default pytest)
-        Specify a list of pytest directories, files or subtests that need to be executed
+        Specify a list of pytest directories, files or subtests that need to be
-        when test case begin to running, default pytest directory is pytest.
+        executed when a test case begins to run. The default pytest directory is
-        After pytest finished, twister will check if this case pass or fail according
+        ``pytest``. After the pytest run is finished, Twister will check if
-        to the pytest report.
+        the test case passed or failed according to the pytest report.
+        As an example, a list of valid pytest roots is presented below:
+
+        .. code-block:: yaml
+
+            harness_config:
+              pytest_root:
+                - "pytest/test_shell_help.py"
+                - "../shell/pytest/test_shell.py"
+                - "/tmp/test_shell.py"
+                - "~/tmp/test_shell.py"
+                - "$ZEPHYR_BASE/samples/subsys/testsuite/pytest/shell/pytest/test_shell.py"
+                - "pytest/test_shell_help.py::test_shell2_sample"  # select pytest subtest
+                - "pytest/test_shell_help.py::test_shell2_sample[param_a]"  # select pytest parametrized subtest
+
+.. _pytest_args:
 
     pytest_args: <list of arguments> (default empty)
-        Specify a list of additional arguments to pass to ``pytest``.
+        Specify a list of additional arguments to pass to ``pytest`` e.g.:
+        ``pytest_args: [‘-k=test_method’, ‘--log-level=DEBUG’]``. Note that
+        ``--pytest-args`` can be passed multiple times to pass several arguments
+        to the pytest.
+
+.. _pytest_dut_scope:
 
     pytest_dut_scope: <function|class|module|package|session> (default function)
         The scope for which ``dut`` and ``shell`` pytest fixtures are shared.

doc/requirements.txt

@@ -13,3 +13,6 @@ sphinx-togglebutton
 # YAML validation. Used by zephyr_module.
 PyYAML>=5.1
 pykwalify
+
+# Used by pytest-twister-harness plugin
+pytest

scripts/pylib/pytest-twister-harness/src/twister_harness/device/device_adapter.py

@@ -25,7 +25,11 @@ logger = logging.getLogger(__name__)
 
 
 class DeviceAdapter(abc.ABC):
-    """Class defines an interface for all devices."""
+    """
+    This class defines a common interface for all device types (hardware,
+    simulator, QEMU) used in tests to gathering device output and send data to
+    it.
+    """
 
     def __init__(self, device_config: DeviceConfig) -> None:
         """
@@ -54,8 +58,8 @@ class DeviceAdapter(abc.ABC):
     def launch(self) -> None:
         """
         Start by closing previously running application (no effect if not
-        needed). Then, flash and run test application. Finally, start a reader
+        needed). Then, flash and run test application. Finally, start an
-        thread capturing an output from a device.
+        internal reader thread capturing an output from a device.
         """
         self.close()
         self._clear_internal_resources()
@@ -100,7 +104,7 @@ class DeviceAdapter(abc.ABC):
     def readline(self, timeout: float | None = None, print_output: bool = True) -> str:
         """
         Read line from device output. If timeout is not provided, then use
-        base_timeout
+        base_timeout.
         """
         timeout = timeout or self.base_timeout
         if self.is_device_connected() or not self._device_read_queue.empty():
@@ -125,13 +129,13 @@ class DeviceAdapter(abc.ABC):
         until following conditions:
 
         1. If regex is provided - read until regex regex is found in read
-           line (or until timeout)
+           line (or until timeout).
         2. If num_of_lines is provided - read until number of read lines is
-           equal to num_of_lines (or until timeout)
+           equal to num_of_lines (or until timeout).
         3. If none of above is provided - return immediately lines collected so
-           far in internal queue
+           far in internal buffer.
 
-        If timeout is not provided, then use base_timeout
+        If timeout is not provided, then use base_timeout.
         """
         timeout = timeout or self.base_timeout
         if regex:

scripts/pylib/pytest-twister-harness/src/twister_harness/helpers/shell.py

@@ -55,7 +55,7 @@ class Shell:
         Send shell command to a device and return response. Passed command
         is extended by double enter sings - first one to execute this command
         on a device, second one to receive next prompt what is a signal that
-        execution was finished.
+        execution was finished. Method returns printout of the executed command.
         """
         timeout = timeout or self.base_timeout
         command_ext = f'{command}\n\n'

tests/boot/with_mcumgr/README.rst

@@ -3,7 +3,7 @@ Upgrade testing with MCUmgr
 
 This application is based on :ref:`smp_svr_sample`. It is built
 using **sysbuild**. Tests are automated with pytest, a new harness of Twister
-(more information can be found here :ref:`integration-with-pytest`)
+(more information can be found :ref:`here <integration_with_pytest>`)
 
 .. note::
    Pytest uses the MCUmgr fixture which requires the ``mcumgr`` available