testharness 1.0.3

This is intended to be used as a generic functional test framework.

===================
Test Harness Readme
===================

:author: Oisin Mulvihill

.. contents::


Introduction
============

This is intended to be used as a generic functional test framework.

This uses a simple script system, based on the python console, to load and run commands from test
modules. The test modules are python classes that implement the details of how to test the project or
program. The module provides a simple set of functions that can be called from a script. The functions
can for example, take care of the complexity of starting some service or actually setting/getting some
value in a database. In the script each function call can then be asserted to test what's returned. This
the is used to pass or fail a test.

I've used this project to test interactions between network services, database driven applications, GUI
testing via pyGTK or in many other situations. I use it mainly to implement user acceptance stories to
verify the behavior of a some user functionality. I don't use it to test web pages as there are better
things out there for this (I use selenium remote control).

I originally developed this project back in 2001 and I've used it to test a few commercial products, that
my company and others have developed. I've got several versions of this project in various places. I've
its time to bring them all together into one release. This is also an exercise for to learn how to write
packages using setuptools.


Running the test harness
========================

Currently the test harness is run out of the pytester directory. The following
instructions apply only to this situation.

* Copy the file ./lib/testharness/config/tester.ini.in to ./tester.ini

* Copy the file ./lib/testharness/testharness_log.ini.in to ./testharness_log.ini

* Edit the tester.ini if needed.

* Edit the log.ini if needed.

* From the command line issues the following to run the test harness. All the tests mentioned in the tester.ini will now be run one after another.::

PYTHONPATH=lib python ./lib/testharness/scripts/runacceptance

Or

goharness



Command line arguments
======================

The test harness takes a number of command line arguments.

-c, --config <config_file>
--------------------------

This specifies the configuration file the test harness is going to use. This
file is required and if it is not specified, then the file "tester.ini" is
looked for in the current path.


-l, --logconfig <log_config>
----------------------------

This specifies the logging configuration use. This file set up where log
messages go, etc. By default the test harness looks for a file called
"log.ini" in the current path.


-i, --interactive
-----------------

This tells the test harness not to start running tests mentioned in the
tester ini file, but instead to go into interaction mode. This allows the
user to type the commands in manually.

--testpath <path_to_tests>
--------------------------

This tells the test harness where to find all the tests and test modules it
is going to use. If this is not specified the the testharness will look for
the "acceptancetests" directory in the current path.


tester.ini configuration
-------------------------

This file is used to configure the test harness and any modules that are
loaded. The TestHarness section contains, as you might expect, settings
used by the test harness. The following is a typical tester.ini::

[TestHarness]
assert_equals_failure_causes_exit = yes
tests = my_default_tests_to_run.script


TestHarness section
-------------------

assert_equals_failure_causes_exit
+++++++++++++++++++++++++++++++++

This tells the test harness to stop processing a test script if an assertEquals
fails in a test. If this was "no" the test harness would try to continue
processing the script. A "yes" value here is the default, "no" is only used in
debugging and is not recomended in normal situations.

tests
+++++

This is a list of scripts the test harness will run when it starts. These are
file names seperated by spaces and all on the same line. Currently you cannot
span multiple lines. Note also that the test harness expects to find these
files in the --testpath <path_to_tests> directory.



Test scripts
============

All tests usually start with \*.script. Thats just a convention I started using.
The files can be called whatever you want.


Test script and test harness console commands
---------------------------------------------

The test harness has a number of built in functions that can be called. These
functions can be called from a script file or from the console. There is no real
distinction.


load <module name>
------------------

This loads a test module into the test harness. Test harness modules are python
script files that contain classes derived from TestBase. When you specify the
module you don't specify '.py' extension.

*Note*

Only class derived from TestBase will be loaded everything else is ignored.


help [<function name>]
----------------------

This function gives help on a command name or lists all functions in the
TestHarness.


exit
----

This tells the test harness to exit after call shutdown.


rem
----

This is only really useful inside a script file. Any line starting with this is
ignored.


print <string to be printed>
----------------------------

This is only really useful inside a script file. This function prints a line of
text to stdout.


shutdown
--------

This function is usually called just before the test harness exits. This
function goes through each loaded module and calls its start() method.


start <module name>
-------------------

This function call a loaded modules start() method.


stop <module name>
-------------------

This function call a loaded modules start() method.

list
----

This function lists all loaded modules in the test harness.


inspect <module name>
---------------------

This function calls a modules inspect() method to print out a list of available
function calls.

Note: It is up to module writer to actually write the information that is
returned for printing.


evaluate <module name> <method call> <arg 1>,<arg 2>,<arg 3>,...<arg X>
-----------------------------------------------------------------------

This function calls a method in a module. Normally this function isn't called by
the user directly. Instead anything which isn't a built in function is assumed
to be a module method call.

For example:

A module called 'mytest' has one class called 'MainTest'. The 'MainTest'
function has a method called 'doMyTest1' and it takes two arguments, a number
and a string.

This could be invoked using the evaluate function as follows::

evaluate mytest.MainTest doMyTest1 123,this is a string

However the following also works::

mytest.MainTest doMyTest1 123,this is a string

*Notes*

Arguments are seperated by commas and these cannot be used by the user in
the script.

Arguemts should have no white space before or after them, unless you want
this as it will be passed to the function.

For example::

doMyTest1 123,this is a string,4321
doMyTest2 123, this is a string ,4321 ,

doMyTest1 will passed the args "123" "this is a string" and "4321"
doMyTest1 will passed the args "123" " this is a string " and "4321 "


assertEquals <error message if fails>,<compare value>
------------------------------------------------------

This function compares the provided <compare value> string, with that returned
by the last user module script call.

For example::

mytest.MainTest doMyTest1 123,this is a string

This call returns a number. This function is converted to a string and stored by
the test harness. Now the assertEquals can be used to test the return as follows::

assertEquals doMyTest1 didn't work,12

If the returned value from doMyTest1 was 12 then assertEquals will pass. If
doMyTest1 didn't return 12 then the message 'doMyTest1 didn't work' will be
printed.

*Note*

If the test harness assert config option is 'yes' the a failing
assertEquals call will cause the test harness to stop running the script and
return with a failure code.