All-in-one Linux Testing Framework
Project description
What is Kirk?
Kirk application is a fork of runltp-ng and it aims to merge multiple Linux testing frameworks in one tool, providing support for remote testing via Qemu, SSH, LTX, parallel execution and much more.
Host information
System: Linux
Node: susy
Kernel Release: 5.14.21-150400.24.60-default
Kernel Version: #1 SMP PREEMPT_DYNAMIC Wed Apr 12 12:13:32 UTC 2023 (93dbe2e)
Machine Architecture: x86_64
Processor: x86_64
Temporary directory: /tmp/kirk.acer/tmpz49sip95
Connecting to SUT: host
Starting suite: math
abs01: pass | tainted (0.001s)
atof01: pass | tainted (0.002s)
float_bessel: pass | tainted (0.673s)
float_exp_log: pass | tainted (0.667s)
float_iperb: pass | tainted (0.252s)
float_power: pass | tainted (0.562s)
float_trigo: pass | tainted (0.646s)
fptest01: pass | tainted (0.002s)
fptest02: pass | tainted (0.003s)
nextafter01: pass | tainted (0.002s)
Suite Name: math
Total Run: 10
Elapsed Time: 3.039s
Passed Tests: 22
Failed Tests: 0
Skipped Tests: 0
Broken Tests: 0
Warnings: 0
Kernel Version: Linux 5.14.21-150400.24.60-default #1 SMP PREEMPT_DYNAMIC Wed Apr 12 12:13:32 UTC 2023 (93dbe2e)
CPU: x86_64
Machine Architecture: x86_64
RAM: 15569424 kB
Swap memory: 2095424 kB
Distro: opensuse-leap
Distro Version: 15.4
Disconnecting from SUT: host
Quickstart
The tool works out of the box by running kirk
script.
Minimum python requirement is 3.6+ and optional dependences are the following:
kirk
will detect if dependences are installed and activate the corresponding
support. If no dependences are provided by the OS's package manager,
virtualenv
can be used to install them:
# download source code
git clone git@github.com:acerv/kirk.git
cd kirk
# create your virtual environment (python-3.6+)
virtualenv .venv
# activate virtualenv
source .venv/bin/activate
# SSH support
pip install asyncssh
# LTX support
pip install msgpack
# run kirk
./kirk --help
Some basic commands are the following:
# run LTP syscalls testing suite on host
./kirk --framework ltp --run-suite syscalls
# run LTP syscalls testing suite on qemu VM
./kirk --framework ltp \
--sut qemu:image=folder/image.qcow2:user=root:password=root \
--run-suite syscalls
# run LTP syscalls testing suite via SSH
./kirk --framework ltp \
--sut ssh:host=myhost.com:user=root:key_file=myhost_id_rsa \
--run-suite syscalls
# run LTP syscalls testing suite in parallel on host using 16 workers
./kirk --framework ltp --run-suite syscalls --workers 16
# run LTP syscalls testing suite in parallel via SSH using 16 workers
./kirk --framework ltp \
--sut ssh:host=myhost.com:user=root:key_file=myhost_id_rsa \
--run-suite syscalls --workers 16
# pass environment variables (list of key=value separated by ':')
./kirk --framework ltp --run-suite net.features \
--env 'VIRT_PERF_THRESHOLD=180:LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1'
It's possible to run a single command before running testing suites using
--run-command
option as following:
./kirk --framework ltp \
--run-command /mnt/setup.sh \
--sut qemu:image=folder/image.qcow2:virtfs=/home/user/tests:user=root:password=root \
--run-suite syscalls
Every session has a temporary directory that can be found in
/<TMPDIR>/kirk.<username>
. Inside this folder there's a symlink
called latest
, pointing to the latest session's temporary directory.
In certain cases, kirk
sessions can be restored. This can be really helpful
when we need to restore the last session after a system crash:
# restore the latest session
./kirk --framework ltp \
--restore /tmp/kirk.<username>/latest \
--run-suite syscalls
Setting up console for Qemu
To enable console on a tty device for a VM do:
- open
/etc/default/grub
- add
console=$tty_name, console=tty0
toGRUB_CMDLINE_LINUX
- run
grub-mkconfig -o /boot/grub/grub.cfg
Where $tty_name
should be ttyS0
, unless virtio serial type is used (i.e.
if you set the serial=virtio
backend option, then use hvc0
)
Implementing SUT
Sometimes we need to cover complex testing scenarios, where the SUT uses particular protocols and infrastructures, in order to communicate with our host machine and to execute tests binaries.
For this reason, kirk
provides a plugin system to recognize custom SUT
class implementations inside the libkirk
folder. Please check host.py
or ssh.py
implementations for more details.
Once a new SUT class is implemented and placed inside the libkirk
folder,
kirk -s help
command can be used to see if application correctly
recognise it.
Implementing Framework
Every testing framework has it's own setup, defining tests folders, data and
variables. For this reason, Framework
class provides a generic API that, once
implemented, permits to define a specific testing framework. The class
implementation must be included inside the libkirk
folder and it will be
used as an abstraction layer between kirk
scheduler and the specific testing
framework.
Development
The application is validated using pytest
and pylint
.
To run unittests:
pytest
To run linting checks:
pylint --rcfile=pylint.ini ./libkirk
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.