In verbose mode (-v) progress of running test cases number out of total
number is not printed. Thus, if there are many test cases, it is
impossible to follow the progress. This commit adds printing of current
test case and total numbers to the verbose output.
Signed-off-by: Ruslan Mstoi <ruslan.mstoi@intel.com>
Export tests to a file with Section, subsection and identifier as
columns making it easy to import testcases into test management system.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This section got added in commit 470349c25a ("Bluetooth: settings:
Add support for per-submodule handlers"), but sanitycheck didn't know
about it and was whining.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Ninja and GNU make don't play well with each other. Both try to start
enough processes to keep the system's CPUs busy, resulting in an
O(N^2) system load in the number of processors.
Long term, Ninja seems likely to support the GNU make jobserver
mechanism for sharing access to parallelism. But for now we can get
90% of the way there with a simple hack: just run ninja in serial mode
with -j1. Sanitycheck when run in non-trivial circumstances has
PLENTY of parallelism just from the number of test cases.
One interesting note is that even with -j1, system loads under ninja
are rather higher. That may be because of significant work done in
the (serial) makefiles that dilutes the parallelism of the eventual
build, or possibly because ninja itself is multithreaded in its setup
code. So I tweaked the number of jobs down to keep the load roughly
where it is with make.
With this change, I see no difference in behavior or system load, and a
~24% improvement in runtime.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
- Some tests start with test_, some do not, so make sure we parse both.
- Parse skipped tests
- Improve handling of test case identifier
- Handle Exceptions in device handler
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We are passing global arguments from one level to the next when those
variables are available globally. Reduce the arguments and remove unused
arguments as well.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
init class in one place, no need to duplicate all class members in every
subclass.
run_log is not needed in the handler class.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Parse the test results and create a test report with more granular
results that can be imported to into test management/reporting system.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This will allow us to run sanitycheck on real devices and get reporting
out of it the same way we do that with Qemu.
To use this, run sanitycheck with the following new options:
scripts/sanitycheck --device-testing --device-serial /dev/ttyACM0 -p
frdm_k64f -T tests/crypto/
--device-serial denotes the serial device the board is connected to.
This needs to be accessible by the user running sanitycheck. You can
run this on one board only at a time, the board is specified using the
--platform option.
This was tested with only a few boards, some board will not work
because how they reset the serial device during flashing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
After removing the -T linker warning for the POSIX arch
we can, and should, also treat its linker warnings as errors
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
This packs a few improvements:
- Add a smarter regex that will catch multiple combinations of
ztest_[user_]unit_test[_setup_teardown](NAME[, setup, teardown])
as well as single liners like:
ztest_test_suite(mutex_complex, ztest_user_unit_test(TESTNAME));
- Limit how much we look forward in suite_regex -- we don't have to
look past the first argument, otherwise we consume too much and the
loopup at suite_regex_match.start() will start too late.
- Remove include_regex, unused
- Fix the path where we warn about matches in achtung_regexes--it
needed a few decodes and to use error() vs the unexistant warning()
- Cleanup the path to produce the subcase names, doing the decode and
the purging of any test_ prefix in scan_path().
Signed-off-by: Inaky Perez-Gonzalez <inaky.perez-gonzalez@intel.com>
This addresses issue #7146.
The current helptext does not state that the directory
will be deleted.
Signed-off-by: Håkon Øye Amundsen <haakon.amundsen@nordicsemi.no>
Parse all yaml file and create a list of declared testcases. This does
list the individual tests inside test projects, not only the projects
containing the tests, for example:
$ sanitycheck --list-tests -T tests/net/socket/
- net.socket.udp.send_recv_2_sock
- net.socket.udp.v4_sendto_recvfrom
- net.socket.udp.v6_sendto_recvfrom
- net.socket.udp.v4_bind_sendto
- net.socket.udp.v6_bind_sendto
- net.socket.getaddrinfo_ok
- net.socket.getaddrinfo_no_host
- net.socket.tcp.v4_send_recv
- net.socket.tcp.v6_send_recv
- net.socket.tcp.v4_sendto_recvfrom
- net.socket.tcp.v6_sendto_recvfrom
- net.socket.tcp.v4_sendto_recvfrom_null_dest
- net.socket.tcp.v6_sendto_recvfrom_null_dest
13 total.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This parses the tests that run within a test project/application from
the source code and gives us a view of what was run, skipped and what
was blocked due to early termination of the test.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If we have some error parsing a testcase or other files we treat these
as errors and will exit before continuing on building other tests.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
In many cases we are calling cmake twice with the same options, first to
build, and then we do the same thing when we want to call 'make run'.
Just call cmake once.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When running large set of tests we always get a huge list of footprint
changes that mask the test results making them impossible to see on the
screen. Show the footprint results only on-demand and not on every
build.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
With this commit it is possible to add priority to sent or received
network packets. So user is able to send or receive higher priority
packets faster than lower level packets.
The traffic class support is activated by CONFIG_NET_TC_COUNT option.
The TC support uses work queues to separate the traffic. The
priority of the work queue thread specifies the ordering of the
network traffic. Each work queue thread handles traffic to one specific
work queue. Note that you should not enable traffic classes unless
you really need them by your application. Each TC thread needs
stack so this feature requires more memory.
It is possible to disable transmit traffic class support and keep the
receive traffic class support, or vice versa. If both RX and TX traffic
classes are enabled, then both will use the same number of queues
defined by CONFIG_NET_TC_COUNT option.
Fixes#6588
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
Move IP address settings from net_if to separate structs.
This is needed for VLAN support.
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
For those still using old variable ZEPHYR_GCC_VARIANT.
raise an error if the variable is not defined.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add the STM32 ccm_bss, ccm_noinit, abd ccm_data sections
to the list of allowed sections so the sanity check passes.
Signed-off-by: Erwin Rol <erwin@erwinrol.com>
We want to support other toolchain not based on GCC, so the variable is
confusing, use ZEPHYR_TOOLCHAIN_VARIANT instead.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
It is important that it is easy to reproduce CI issues locally. Using
the same sanitycheck options locally and in CI helps in this
regard. Specifically, Ninja and Make can produce different results and
therefore the default generator should be the same for sanitycheck and
shippable.
This patch makes four changes:
The sanitycheck option '--make' is introduced to allow specifying Make
as a generator.
CI no longer passes the option '--ninja' to sanitycheck.
Sanitycheck defaults to using Ninja.
Sanitycheck documents the --ninja option as deprecated.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
When generating coverage report:
* Also generate branch coverage
* If the report is properly generated tell user where to find it
* Handle case in which no ztests are run (it would crash before)
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
Aligned sanitycheck command line options in doc
with the actual ones.
+
Minor fix in --coverage description.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
sanitycheck: Compile unit tests with coverage enabled always
+ run also first unit tests together with native_posix
shippable: also include unit_testing coverage into report to
codecov
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
This patch adds app_pad and priv_stacks to the rw section list so that
tests which validate the sections found in the binary pass when run on
systems which contain MPUs.
Signed-off-by: Andy Gross <andy.gross@linaro.org>
Rather than having handler generate differently named log files,
generate the same for all handlers, so qemu, native and unit will now
generate handler.log.
This will simplify things and reporting will be consistent. Also, this
fixes a bug where we only included qemu.log in generated reports.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When building the testcase path, replace symlinks in path
with the realpath both in the testcase root and in ZEPHYR_BASE
For more info see issue #5772Fixes: #5772
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
Changed the default behavior for when we run sanitycheck to match the -R
flag that turns on assertions. Introduced a --disable-asserts option if
we desire to explicitly turn of asserts.
This matches behavior that our CI builds have been doing and addresses
part of #5726.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Add 2 classes, one to handle the current TestCase scenario, and one more
for handling generic Console with regex matching.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some boards are supported natively by qemu. This option will allow us to
run tests using those platforms directly without having to go via a
dedicated qemu board definition.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
if we are using command line platform filter, no need to list every
other platform as excluded, we know that already. Show only the
discards that apply to the selected platforms on the command line
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When we load tests from a file, we do not have the discarded list, this
would have been done already when saving the test file.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When checking if a testcase passed or failed, allow there
to be prefixes or postfixes in the line, around
PROJECT EXECUTION SUCCESSFUL/FAILED
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
CONFIG_ASSERT is being set by cmake, so it is not possible to filter
using the generated config, add this as a standalone filter.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Generating coverage data is split over two CI jobs which means the
service will need to merge results and reports wrong coverage data when
only 1 job is finished. This puts the native_posix board first making
sure we run on the first job and generate data in one place.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Extend exception handling to cover not just YAML loading, but any
error while accessing parsed data too. That may catch e.g. schema
mismatch errors (for folks who don't have pykwalify installed, which
is optional). So, now error will be logged, but processing of other
tests will continue.
For example, I had a local, uncommitted test which wasn't converted
per 23f81eeb42 and caused:
Traceback (most recent call last):
File "./scripts/sanitycheck", line 2456, in <module>
main()
File "./scripts/sanitycheck", line 2324, in main
options.outdir, options.coverage)
File "./scripts/sanitycheck", line 1445, in __init__
for name in parsed_data.tests.keys():
AttributeError: 'list' object has no attribute 'keys'
Signed-off-by: Paul Sokolovsky <paul.sokolovsky@linaro.org>
