Add a filter type to distinguish between testsuite filters that are
maintained in the testcase.yaml file and those provided on the command
line.
The issue was that when something is excluded deliberately on the command
line twister would report an error if that filter would filter out an
integration platform. We do not want that, because the filtering is
being done by the caller knowing that some tests would be not run, even
if those are integration platforms.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Removed tests for dropped functionality.
Renamed classes and files to reflect new structure.
As we refactor the reporting infrastructure, tests will follow once we
have everything in place.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
CSV file is expanded to store run_id and the corresponding test
requires a modification.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Test wirtten in ztest framework should print in output information about
test suite name defined in test's source code (as a first argument of
ztest_test_suite() function). This changes make it possible to find such
test names in c files. Next they are used during test execution to
verify if performed test was performed properly (verification can base
not only on existance of "PROJECT EXECUTION SUCCESSFUL" info).
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Changes will raise error and break program in following three situations:
1. Platform name pass in --platform option does not exist.
2. During using --all option, platform from platform_allow list does not
exist.
3. During using --integration option, platform from
integration_platforms list does not exist.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Introduce a weak implementation of test_main() which calls:
* ztest_run_registered_test_suites(NULL);
* ztest_verify_all_registered_test_suites_ran();
This will attempt to run all registered test suites and verify that
they each ran.
Signed-off-by: Yuval Peress <peress@chromium.org>
Add new functionality to ztest to improve test modularity. The two
primary new entry points are:
* ztest_register_test_suite
* ztest_run_registered_test_suites
When registering a new test suite, users provide the name as well as
an optional predicate used to filter the tests for each run. Using NULL
as the predicate ensures that the test is run exactly once (after which
it is automatically filtered from future runs).
Calls to ztest_run_registered_test_suites take a state pointer as an
argument. This allows the the pragma functions to decide whether the
test should be run.
The biggest benefit of this system (other than the ability to filter
tests and maintain a larger test state) is the ability to better
modularize the test source code. Instead of all the various tests
having to coordinate and the main function having to know which tests
to run, each source file manages registering its own test
suite and handling the conditions for running the suite.
Signed-off-by: Yuval Peress <peress@chromium.org>
We now get toolchain from cmake and can operate even if the toolchain
variant is not set in the envrionment, this test was verifying the old
behavior, so remove. There is not much twister related here to test and
the environment is not setup correctly to run all of this cmake code. We
are testing twister after all.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
use a loop instead of all() to make the test easier to debug.
remove the default platform test, as this is now being filtered
differently in the code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In the junit output the attribute containing the number of skipped tests
must be named "skipped" instead of "skip".
See e.g. https://github.com/junit-team/junit5/blob/main/platform-tests/
src/test/resources/jenkins-junit.xsd#L95
Signed-off-by: Christian Taedcke <christian.taedcke@lemonbeat.com>
Before sanitycehck was removing all `test_` chars in test cases'
names. This test has to be modify to work with the improved behavior
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Support running/building only specific tags and ignoring everything
else.
Adding this to a platform yaml file will enable tests with one of of
tags listed to be executed on the platform.
This is useful for special platform configurations used for testing
specific features for example.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
test_reporting_testsuite.py: Adding testcases for reporting
functions of Class testSuite.
test_testsuite_class.py: Testcase for add_instances function
conftest.py: Added fixture for instances_fixture
Signed-off-by: Aastha Grover <aastha.grover@intel.com>
test_testinstance.py: Add testcases to scan file and path for
sub testcases
test_data/testcases: Also added the ztest test files
test_testsuite_class.py: changed get_all_tests() to match
count of sub testcases in ztest files
Signed-off-by: Spoorthy Priya Yerabolu <spoorthy.priya.yerabolu@intel.com>
test_testsuite_class.py: Added testcases for load_From_file and
apply_filters functions of Class Testsuite.
conftest.py: Modified class_testsuite fixture to create the
outdir as temporary directory which gets deleted after execution
of testcases.
test_data/sanitycheck.csv: load_from_file function uses this
customized file to load the failed or last run testcases.
test_data/sanitycheck_keyerror.csv: file used by
test_load_from_file function to raise the appropriate error
if there is a keyerror.
scripts/requirements-build-test.txt: added mock & csv python libraries
Signed-off-by: Aastha Grover <aastha.grover@intel.com>
test_testsuite_class.py: Added testcases for add_configurations(),
get_all_testcases(), get_toolchain() and get_platforms() functions
test_data/board_config: board root directory for all board
configurations. This is also to add the platform configurations
in a list which is used for further testing.
conftest.py: Added fixtures for all_testcases_dict (returns a
dictionary of testcases) and platforms_list (returns a list of
all platforms using add_configurations function).
Signed-off-by: Aastha Grover <aastha.grover@intel.com>
test_testinstance.py: Add testcases for check_build_or_run()
create_overlay() and calculate_sizes()
conftest.py: Module for common pytest fixtures
test_data/board_config: Adding test data for board configurations
Signed-off-by: Spoorthy Priya Yerabolu <spoorthy.priya.yerabolu@intel.com>
test_testsuite_class.py: Add testcases for add_testcases function
of testsuite class in sanitycheck.
test_data/testcases/tests & /samples : Testcase root directory
to add all the testcases & to test add_testcases function.
conftest.py: Module for common pytest fixtures, also used for
passing data from one testcase to another.
Note: conftest.py has a class_testsuite fixture where board_root is
defined as the directory which will be added in a separate PR.
Signed-off-by: Aastha Grover <aastha.grover@intel.com>
This commit adds basic testcases for sanitycheck tool using pytest.
Coverage for the sanitycheck tool is obtained using coverage tool.
Instructions are included in the README.md in
scripts/tests/sanitycheck directory.
Signed-off-by: Aastha Grover <aastha.grover@intel.com>
