tools-make/TestFramework

TestFramework
=============

This testsuite is a general framework for testing the core GNUstep
libraries. Since, in part, we are testing the very basic level of
an Objective-C runtime, we support testing plain C or C++ as well
as Objective-C, and aim for better flexibility, ease-of-use, and
and portability than older frameworks such as as OCUnit.

The aim of this framework is to provide a very simple, yet reasonably
comprehensive regression test mechanism, primarily for Objective-C
development.

Please run the GNUstep testsuite (using this framework) often, when
adding new features, fixing bugs and running on new platforms.

Where working on features common to both Apple's Cocoa/iOS APIs and
to GNUstep, please try creating and running test cases in the Apple
environment before implementing/changing GNUstep code, so that you
are sure the behavior is the same in both cases.


License 
-------

The testing framework and many of the test cases in the testsuite are
copyright by the FSF and distributed under the GPL.  However, some tests
may not be copyright by the FSF, but retain the copyright of the original
owner (e.g tests submitted as bug reports). You should feel free to add tests
that are not copyright by the FSF. The copyright of these tests should
be clearly stated, however, and they should still be distributed under the
GPL version 3 or later.


Running Tests
-------------

To run a testsuite, use the gnustep-tests script along with the name of
the project testsuite (directory) you wish to test:

gnustep-tests base

or where a group of tests within a project is to be run:

gnustep-tests base/NSArray

You may run an individual test file by using the gnustep-tests script
with the name of the Objective-C test source file:

gnustep-tests mytest.m
gnustep-tests base/NSDate/general.m

Alternatively, you may run tests from within a project/directory. eg.

cd base
gnustep-tests .

If you supply no arguments, the gnustep-test script will examine all the
subdirectories of the current directory, and attempt to run tests in each.

During testing, temporary files are created and removed for each test,
but any temporary files from the most recent test in each directory
are left (to help you debugging), as are the log files.
The summary log of a test run is left in the tests.sum file.
The detailed log of a test run is left in the tests.log file.
The log from any previous run is left in oldtests.sum (and oldtests.log).
You can run 'gnustep-tests --clean' to remove left-over temporary files
and all log files.


Interpreting the output
-----------------------

The summary output lists all test failures ... there should not be any.
If a test fails then either there is a problem in the software being
tested, or a problem in the test itself. Either way, you should try to
fix the problem and provide a patch, or at least report it at:
https://savannah.gnu.org/bugs/?group=gnustep"

After the listing of any failures is a summary of counts of events as follows.

Passed tests:
  The number of individual tests which passed.

Failed tests:
  The number of individual tests failed ... this should really not appear.

Failed sets:
  The number of sets of tests which have been abandoned part way through
  because of some individual test failure or an exception in support code
  between tests.

Failed builds:
  The number of separate test files which did not even build/compile.

Failed files:
  The number of separate test files which failed while running.

Dashed hopes:
  The number of hopeful tests which did not pass, but which were not
  expected to pass (new code being worked on etc).

Skipped sets:
  The number of sets of tests which were skipped entirely ...
  eg. those for features which work on some platforms, but not on yours.

The binary executable of the most recently executed test file in each test
directory is left in the obj subdirectory.  So you can easily debug a failed
test by:
1. running gnustep-tests with the single test file as its argument.
2. running gdb using obj/filename as an argument.
3. setting a breakpoint at the exact test which failed, running to there,
4. and then stepping through slowly to see exactly what is going wrong.

You can use the --failfast option with gnustep-tests to tell it to abandon
testing after the first failure ... in which case you know that the
executable of the failed test will be available (unless the test file
failed to even compile of course).  In this case, any core dump file will
also be left available.

As a convenience for debugging, the tests use a testStart() function in which
they set the line number of the test, so you can stop in that function and
upon stepping out of it the debugger will be examining the specified test.
eg.
(gdb) break testStart if line == 42
(gdb) run

If you use the --debug command line option, and any tests fail, then the
debugger (gdb) will automatically be started for you with breakpoints set
to the testStart() function of the failed tests.

You can use the --debug command line option in conjunction with the --failfast
option to have testing stopped at the first failure and the gdb debugger
automatically launched to debug the failed testcase with a breakpoint set
in the testStart() function for that testcase.

You can use the --notimestamps command line option to turn off timestamps of
individual testcases if you want a less verbose output (though test code may
override this default by setting the testTimestamps variable itself).

You can also use the --developer command line option to define the TESTDEV
pre-processor variable (to turn on developer only test cases, and to have
all 'hopes' treated as actual 'tests' with pass/fail results).

Writing Tests
-------------

The test framework may be used for testing Objective-C, ObjectiveC++, C,
and C++ code.  The test source files must have a .m (for Objective-C and C)
or a .mm (for Objective-C++ and C++) file extension in order to be recognised.

A minimal test should be a file importing the header "Testing.h"
(which defines global variables, functions, and standard test macros)
and containing a main() function implementation which executes the
actual test code.
Groups of tests should be placed between calls to the START_SET() and
END_SET() macros.

You should look at the example test files in the
$GNUSTEP_MAKEFILES/TestFramework directory
for how to write test cases, and you should examine Testing.h
in the same directory to see full documentation of the range
of macros provided.

The main workhorse of the test framework is the pass() function, which has
a variable number of arguments ... first an integer expression, and second
a printf style format string describing what is being tested.
If you are calling the function directly you should use "%s,%d" at the
start of the format string and pass __FILE__ and __LINE__ as the next two
parameters (for consistency with the test macros).
The function uses the global variable 'testHopeful' to decide whether a
test which did not pass is a 'FAIL' (when testHopeful==NO) or a 'DASHED'
hope (when testHopeful==YES).
The function sets the global variable 'testPassed' to a BOOL reflecting
the result of the test (YES if the test passed, NO otherwise).
The only other functions are for occasional use to report sections of
the testsuite as not having run for some reason.

There are just four basic test macros.
All have uppercase names beginning with 'PASS'.
All wrap test code and a call to the pass() function in exception handlers.
All provide file name and line number information in the description string.
All are code blocks and do not need a semicolon terminator.
Code fragments must be enclosed in round brackets if they contain commas.

PASS		passes if an expression resulting in an integer value is
		non-zero
PASS_EQUAL	passes if an expression resulting in an object is identical
		to or -isEqual: to another object (if the expected object
                implements the -isEqualForTestcase: method, that is used
                instead of -isEqual:)
PASS_EXCEPTION	passes if a code fragment raises an exception
PASS_RUNS	passes if a code fragment runs without raising an exception

There is a boolean variable called 'testHopeful' which, if set to YES,
means that tests which do not pass are considered to be 'Dashed hopes'
rather than failed tests.  You should set this for tests of code which
is under development (or which is testing a feature which may be
unsupported in the package under test) ... to indicate that the test is
not to be considered a failure if it doesn't pass.

Tests are grouped together, along with any associated non-test code, between
paired calls to the START_SET and END_SET macros. Any setting of testHopeful
within a set is automatically restored at the end of a set, so it makes sense
to group hopes together in a set.

You can skip an entire set by calling the SKIP() macro just after the start,
in which case the entire set will be reported as being Skipped.
It is appropriate to skip sets of tests if you have checked and found that
some feature you are testing is not available in the version of the package
under test.

Any uncaught exception (ie one which occurs outside a one of the four test
macros and is not caught by an exception handler you write yourself) will
cause the remaining tests in a set to be omitted.  In this case the set
will be reported as Failed.

You may also arrange to jump to the end of the set if a test fails by wrapping
the test in a NEED macro.  Doing this also causes the set to be reported as
Failed if the needed test does not pass.

It's likely that you are writing new tests for a library or framework ...
and those tests will need to link with that framework.  You should add the
instructions for that to a GNUmakefile.preamble if the directory containing
your tests, or a GNUmakefile.super in the directory above in the case where
you have multiple test directories.
eg.
ADDITIONAL_OBJC_LIBS=-lmyLibrary

When contributing to a test suite, please bracket your new test code using
#if defined(TESTDEV)
...
#endif /* TESTDEV */
so that it is only built when gnustep-tests is invoked with the --developer
command line argument.
This ensures that the new code won't break any existing test code when
people are simply running the testsuite, and once you are sure that the
new testcases are correct (and portable to all operating systems), the
check for TESTDEV can be removed.

Ignoring failed test files
--------------------------

When a test file crashes during running, or terminates with some sort of
failure status (eg the main() function returns a non-zero value) the framework
treats the test file as having failed ... it assumes that the program
crashed during the tests and the tests did not complete.

On rare occasions you might actually want a test program to abort this way
and have it treated as normal completion.  In order to do this you simply
create an additional file with the same name as the test program and a
file extension of '.abort'.
eg. If myTest.m is expected to crash, you would create myTest.abort to have
that crash treated as a normal test completion.


Advanced building
-----------------

In most cases, all you need to do is write an objective-c file as described
above, and the test framework will build it and run it for you automatically,
but occasionally you may need to use your own build process.

Where tests must make use of external resources or ensure that other tests
have already been run before they are run, you can make use of the gnustep
make package facilities to control dependencies etc.

Normally the tests in a directory are built run using a makefile generated in
the directory.  This makefile uses the standard conventions of including
GNUmakefile.preamble before test-tool.make and including GNUmakefile.postamble
after test-tool.make, which gives you a high degree of control over how the
tests in the directory are built.

In addition to the preamble/postamble mechanism, the file ../GNUmakefile.super
is included at the start of the generated makefile (if it exists).  This allows
all the test directories in a suite to use a common makefile fragment to provide
information for the whole testsuite.
You can also use the GSTESTROOT environment variable to locate resources common
to the whole testsuite ... it is set automatically by gnustep-test to be the
absolute path to the topmost directory in the testsuite.

Your system should not make any assumption about the order in which test
files are built ... the test framework may build many test files in parallel
in order to make effective use of multiple processors.  In fact the make
program will normally build up to four tests at a time, but you can change
that by setting the MAKEFLAGS environment variable to '-j N' where N is the
number of simultaneous builds you want to be permitted (or you can simply
use 'gnustep-tests --sequential' to force building of one test at a time).

Running of the tests is, by default, done sequentially in alphabetical order,
but this may be overridden to change the order of sequential tests and to run
tests concurrently.  The mechanism for this is to set values in the TestInfo
file.

For total control, the framework checks to see if a 'GNUmakefile.tests' file
exists in the directory, and if it does it uses that file as a template to
create the GNUmakefile rather than using its own make file.
This template makefile may use @TESTNAMES@ where it wants a list of the
tests to be run, and @TESTRULES@ where it wants the rules to build the
tests to be included.
It should also use @TESTOPTS@ near the start of the file to permit necessary
makefile control to say where the executables should be stored.
The GNUmakefile.tests script should build each individual test when it is
invoked with that test name as a target, and it should also build all tests
if it is invoked without a target, and have a 'clean' target to clean up
before and after all tests.


Directory layout
----------------

A test suite is considered to be a collection of individual test files in
a single directory or a collection of directories in a hierarchy.
All directories which contain test files must also contain a TestInfo file
to mark them as containing files used by the framework, and the root of
the test suite is considered to be the topmost directory in the hierarchy
which contains a TestInfo file.  The test framework sets the GSTESTROOT
environment variable to the absolute path of the root of the test suite
being executed, so scripts and makefiles can use this to locate resources.

The test framework ignores any directory which does not contain a TestInfo
file.  This feature prevents accidental attempts to treat a project source
code directory as a testsuite.  This is also useful in conjunction with the
various makefile options listed above ... the makefiles may be used to
build resources for tests in subdirectories which are ignored by the
test framework itself.

In addition to being a marker, the TestInfo file is a shell script which 
is sourced before execution of each test program in its directory, typically
it is used to set up environment variables (eg. LD_LIBRARY_PATH to tell the
program where to find dynamic libraries the tests use).


Providing extra control and information
---------------------------------------

If a Start.sh script is present in a test directory, it will be run
immediately before tests are performed in that directory.  It is able
to append information to the log of the test run using the GSTESTLOG
variable.

If an End.sh file is present in a test directory, it will be run immediately
after the tests in that directory are performed.  It is able to append
information to the log of the test run using the GSTESTLOG variable.

In both cases, you must make sure that the file does not do anything
which would confuse the test framework at the point when it analyses the
log ... so you need to avoid starting a line in the log with any of the
special phrases generated to mark a passed test or a particular type of
failure.

If a Summary.sh file is present in a test directory and gnustep-tests is
used to run just those tests in that directory, the shell script will be
executed in order to provide the summary of the test results.  In all other
cases the summary is done by the Summary.sh script provided in the test
framework.