tools-make/TestFramework

TestFramework
=============

This testsuite is a general framework for testing the core GNUstep
libraries. Since, in part, we are testing the very basic level of
an Objective-C runtime, we can't use more complete unit tests, such
as OCUnit <http://www.sente.ch/software/ocunit/>.
The aim of this framework is to provide a simple, yet reasonably
comprehensive regression test mechanism for Objective-C development.

Please run the GNUstep testsuite (using this framework) often, when
adding new features, fixing bugs and running on new platforms.

Where working on features common to both Apple's Cocoa/iOS APIs and
to GNUstep, please try creating and running test cases in the Apple
environment before implementing/changing GNUstep code, so that you
are sure the behavior is the same in both cases.


License 
-------

The testing framework and many of the test cases in the testsuite are
copyright by the FSF and distributed under the GPL.  However, some tests
may not be copyright by the FSF, but retain the copyright of the original
owner (e.g tests submitted as bug reports). You should feel free to add tests
that are not copyright by the FSF. The copyright of these tests should
be clearly stated, however, and they should still be distributed under the
GPL version 3 or later.


Running Tests
-------------

To run a testsuite, use the gnustep-tests script along with the name of
the project testsuite (directory) you with to test:

gnustep-tests base

or where a group of tests within a project is to be run:

gnustep-tests base/NSArray

You may run individual test files by using the gnustep-tests script with the
name(s) of the Objective-C test source file(s):

gnustep-tests base/NSDate/general.m
gnustep-tests ./mytest.m base/NSDate/general.m

Alternatively, you may run tests from within a project/directory. eg.

cd base
gnustep-tests 


Writing Tests
-------------

A minimal test should be a file importing the header "Testing.h"
(which defines global variables, functions, and standard test macros)
and containing a main() function implementation which executes the
actual test code.
Groups of tests should be placed between calls to the START_SET() and
END_SET() macros.

You should look at the example test files in the same directory as
this README for how to write test cases, and you should examine Testing.h
to see full documentation of the range of macros provided.

The main workhorse of the test framework is the pass() function, which has
two arguments ... first an integer expression, and second a string
describing what is being tested.
The function uses the global variable 'testHopeful' to decide whether a
test which did not pass is a 'FAIL' (when testHopeful==NO) or a 'DASHED'
hope (when testHopeful==YES).
The function sets the global variable 'testPassed' to a BOOL reflecting
the result of the test (YES if the test passed, NO otherwise).
The only other functions are for occasional use to report sections of
the testsuite as not having run for some reason.

There are just four test macros.
All have uppercase names beginning with 'PASS'.
All wrap test code and a call to the pass() function in exception handlers.
All are code blocks and do not need a semicolon terminator.

PASS		passes if an expression resulting in an integer value is
		non-zero
PASS_EQUAL	passes if an expression resulting in an object is identical
		to or -isEqual: to another object.
PASS_EXCEPTION	passes if a code fragment raises an exception
PASS_RUNS	passes if a code fragment runs without raising an exception

Tests are grouped together, along with any associated non-test code, between
paired calls to the START_SET and END_SET macros.

You can skip an entire set by supplying NO as the argument to the START_SET
macro, in which case the entire set will be reported as UNSUPPORTED.

Any uncaught exception (ie one which occurs outside a one of the four test
macros and is not caught by an exception handler you write yourself) will
cause the remaining tests in a set to be skipped.  In this case the set
will be reported as UNRESOLVED.

You may also arrange to jump to the end of the set if a test fails by wrapping
the test in a NEED macro.  Doing this also causes the set to be reported as
UNRESOLVED.


Advanced building
-----------------

In most cases, all you need to do is write an objective-c file as described
above, and the test framework will build it and run it for you automatically,
but occasionally you may need to use your own build process.

Where tests must make use of external resources or ensure that other tests
have already been run before they are run, you can make use of the gnustep
make package facilities to control dependencies etc.

Normally each test is built and run by generating a makefile in the directory
containing the test.  This makefile uses the standard conventions of including
GNUmakefile.preamble before test-tool.make and including GNUmakefile.postamble
after test-tool.make, which gives you a high degree of control over how the
tests in the directory are built.

In addition to the preamble/postamble mechanism, the file ../GNUmakefile.super
is included at the start of the generated makefile (if it exists).  This allows
all the tests in a suite to use a common makefile fragment which can (for
instance) build common resources before any tests are run.

For total control, the runtest.sh script checks to see if a 'Custom.mk' file
exists in the directory, and if it does it uses that file to build the tests
rather than generating its own make file.

You may also specify a GNUmakefile.tests in a project level directory (ie one
containing subdirectories of tests), and that makefile will be executed when
you run tests on the project.

Ignoring directories
--------------------

If, when given the name of a test to be run, the runtest.sh script finds a
file named 'IGNORE' in the same directory as the named file, it skips
running of the test.  The effect of this is that the presence of an IGNORE
file causes a directory to be ignored.  This is useful in conjunction
with ../GNUmakefile.super so that projects to build resources for other tests
can be ignored by the scripts running the tests, and just built as required
by ../GNUmakefile.super