|
MyUtf : My Little Unit Test Framework - Felix John COLIBRI. |
- abstract : this simplified unit test framework has a very simple code (two lists), while still keeping all the traditional unit test functionality
- key words : Unit Test - Regression Test - eXtreme Programming - method
call by name
- software used : Windows XP Home, Delphi 6
- hardware used : Pentium 2.800Mhz, 512 M memory, 140 G hard disc
- scope : Delphi 1 to 2006, Turbo Delphi for Windows, Rad Studio 2007
- level : Delphi developer
- plan :
1 - Understanding Unit Test Every project uses some form of test before being used. The level of testing
depends on many factors like - usage (single usage utilities for the developer, standard project, critical software)
- budget and time
If we decide to produce a reasonable bug free project, Unit Test is one of the
best techniques. Basically it allows to write as many test cases as we want, involving as many objects as we want (or even no objects at all), and run these test as many time as we want.
We test individual features, and each time we modify the software, we can run all the tests again with a simple click of the mouse (regression test: check that the modification do not break previous code)
This article is not a tutorial about how to use Unit Test or some of its implementation frameworks (dUnit, xUnit4Delphi, nUnit etc). Our purpose is to present a simple project implementing unit testing, and explaining how to use
it. Meanwhile, for those who never looked at Unit Testing, this article might be an introduction to the topic, using a "how it works" approach.
2 - My Little Unit Test Framework
2.1 - How Unit Testing Works Let's take a simple invoicing system. It uses an invoice list, and each invoice references a customer. Shooting from the hip, the simplified Uml Class Diagram could like this:
To test a project, we want to test - individual functionalities (is the current data <= Now, is the address Zip
code valid etc)
- single Classes (for the c_invoice_list Class, add_invoice must result in an increased f_invoice_count result, after clear_invoices the list must really be empty)
- several Classes working together (when we add an invoice, the customer must already exist)
When testing a Class or a group of Classes, we usually have to perform
several tests. So we group those tests on some entity in a test case, and we can encapsulate each test case in a test case Class. Here is an example of such a test case Class:
On the terminology side: - since the actual test are written in methods, we will name them method
tests. A c_test_case contains several test methods.
- therefore to test our entity (individual test, Class, Class group), we iterate over its test methods.
All the test cases are gathered in a test case list (a test suite), and to perform our test, we simply iterate over this list to run each c_test_case.
Basically our test structure looks like this
with a very simple testing algorithm:
build test_case_list from individual test cases
run_test_case_list
FOR all test cases
run_test_case
FOR all test methods
run_test_method |
When we write our project - we start a separate test project which will check the project in a non
invasive way (you do not have to modify the project's code in any way. You just test some part of it)
- this is done by writing one, usually several, test cases
- each test case is dedicated to test some part of the system, and contains
several test methods which check the different functionalities of this part
- when we add or modify the code
- to test the addition or modification
- we add some test methods to existing test cases
- or write some new test case(s)
- we run the test list (test suite) to check
- that the previous functionalities were not broken by the changes
- that the new code works as expected
2.2 - Unit Test Framework 2.2.1 - Manually running the method tests With the previous Class organization - the actual c_test_case Classes must descend from a common c_test_case
Class, in order to be added to the m_c_test_case_list container
- the ancestor c_test_case contains a Virtual; Abstract; run_test_case
- each actual c_test_case descendent overrides this run_test_case which calls all its test methods
This certainly works, and would be the easiest to build and understand test framework.
2.2.2 - Automatically running the test methods Most unit test frameworks however automate the iteration over the test methods.
This is achieved in the following manner - we add a new c_test_method_ref Class which contains a method pointer which will be initialized with an individual test method address
- the c_test_case
- contains an list of such c_test_method_refs
- the run_test_case procedure simply iterates over this method reference list and call the actual test methods
The new Uml Class Diagram, with two actual test cases the becomes: and when we build the test case list, each actual test method will spawn a
separate c_test_method_ref instance which points to the actual test method code (the red arrows):
Taking our invoicing system, this would look like this: and:
- the test framework (dUnit, xUnit4Delphi, MyUtf) has been written once (the green part)
- for our project
- we write the actual code (the blue part)
- if we desire (and we should)
- we write as many c_test_case descendent as possible, which may instanciate (but not modify) our actual code (the yellow part)
- we build the c_test_case_list and run the tests
Note that
- this architecture nicely allows us to isolate the general framework in a Unit and let the developer write his test case in project specific Units.
- test cases are not tied to a specific Class or group of Classes:
- there can be test cases to check global variables or testing global Procedures or Funtions
- several test cases can operate on the same Class
- several test cases can test the same combination of Classes
- we would group the test functionally, for instance placing all the tests of
some Class (or the same group of Classes) in the same test case, but there is no obligation to do so.
2.3 - check_xxx 2.3.1 - Checking a test condition
In each test method, we can use any error reporting technique we want. Traditionally, the Unit Test Frameworks report the error to the main unit test program (a console application, a windows application with some tMemo for
reporting, a special Unit Test tForm with a tTreeView). To notify the main program of any failure (or exception), the best technique of course is to use exceptions:
- in the ancestor c_test_case, we enclose the call to the descendent test method in a Try Except
- in our specific test method, when an error is detected (the actual outcome
does not match the expected outcome), we raise an Exception, and this is caught in the ancestor c_test_case Try Except. From the Except End block, the exception can be notified to the main program
2.3.2 - Separate failure and Exception Any exception would work at this level (Abort, Raise Exception.Create(''),
Assert etc). However, using a general Exception would notify the main program in the same way - whether there was an exception in the code under test (a bug in our code under test)
- or when we in our test method we Raise an exception because some test condition was not met
Therefore, in order to be able to distinguish both exception kinds, we simply
create an Exception descendent, and the type of exception can be tested using the On Exception construct:
// -- in the framework
type c_failure= Class(Exception);
procedure c_test_case.run_test_case;
begin
For l_c_test_method_ref In test_method_ref_list
Try
// -- call the test method
l_c_test_method_ref.m_po_test_method ();
Except
On e : c_failure Do
report_a_failure
On e : Exception Do
report_an_exception
End; // try except
End; // run_test_case
// -- our actual test case code
Procedure c_invoices_test.test_invoices_clear;
Var l_c_invoices: c_invoices;
Begin
l_c_invoices:= c_invoices.Create;
l_c_invoices.add_invoice( ... );
l_c_invoices.clear_invoices;
// -- here a test failure if clear_invoices is wrong
If l_c_invoices.f_invoice_count<> 0
Then Raise c_failure.Create('count failure')
l_c_invoices.Free;
End; // test_invoices_clear
Procedure c_invoices_test.test_invoices_add;
Var l_c_invoices: c_invoices;
Begin
// -- here an exception becaus we forgot to call Create
l_c_invoices.add_invoice( ... );
If l_c_invoices.f_invoice_count<> 1
Then Raise Exception.Create
l_c_invoices.Free;
End; // test_invoices_add
|
2.3.3 - Custom check_xxx In the standard Unit Test frameworks, the raising of the exception is triggerd using some kind of Check() or customized Assert Procedures. And there are
usually also gadzillions of heavily overloaded Procedures (test that 2 Integers, Double, String and what not are equal, add a test message or not, etc).
In reality, it all comes down to triggering a c_test_exception with two parameters: - a boolean, which compares the actual outcome to its expected value
- a text which can be displayed in the main program
So we sticked with a simple check procedure:
procedure check(p_condition: Boolean; p_text: String);
|
Note that - our check procedure is similar to Delphi's Assert method
- using Assert (or some code encapsulating it to be able to distinguish the
failures from the exceptions) has additional benefits, since Assert also reports information about the Unit and the code position
- in a single test method, we can place as many checks as we want. We could for instance
- create the invoice list, and check that the Delphi container has been created. And f_invoice_count should return 0
- add an invoice and check the count
- add a couple of invoices, still check the count
- delete an invoice, check that the count decreased, and some find_by_invoice_number returns Nil
- when we have several checks, in our code the next checks OF THIS test
method are skipped. Continue testing could raise false failures caused by the first exception side effects
- the check method can be encapsulated in the c_test_case Class, but could
also be a global Procedure (or a c_test_case Class Procedure)
2.4 - Setup and Teardown To run each test method, we have to
- prepare the data (initialize some globals, create Class instances, call some preprocessing (like adding some object before testing delete)
- run the test (call the procedures of the item under test)
- check that the result is correct
In many cases, all or part of the preparation of the data (and its corresponding cleanup) uses the same code for each test method. We naturally
factor those common parts in two methods, traditionally named Setup and Teardown. Those methods could be called at the start and the end of each of our c_test_case test methods:
Type c_invoices_test=
Class(c_test_case)
m_c_invoices: c_invoices;
Procedure Setup;
Procedure invoices_test_clear;
Procedure TearDown;
end; // c_invoices_test
Procedure c_invoices_test.Setup;
begin
m_c_invoices:= c_invoices.Create;
end; // Setup
Procedure c_invoices_test.invoices_test_clear;
Begin
Setup;
m_c_invoices.add_invoice( ... );
m_c_invoices.clear_invoices;
If m_c_invoices.f_invoice_count<> 0
Then Raise c_failure.Create('count failure')
TearDown;
End; // invoices_test_clear
Procedure c_invoices_test.Teardown;
begin
m_c_invoices:= c_invoices.Create;
end; // Teardown
|
But this can obviously be automated by - in the framework
- placing Virtual Setup and Teardown definitions in the ancestor c_test_case
- in the c_test_case.run_test_case, calling those method before and after each call of the individual test methods
- in the testing code
- in each c_test_case descendent, writing an Override version of Setup and Teardown
So the framework code would look like this:
type c_test_case=
Class
Procedure Setup; Virtual; Abstract;
Procedure runt_test_case;
Procedure TearDown; Virtual; Abstract;
end;
procedure c_test_case.run_test_case;
begin
For l_c_test_method_ref In test_method_ref_list
Try
Setup;
l_c_test_method_ref.m_po_test_method ();
Teardown:
Except
report_errors;
End; // try except
End; // run_test_case
|
and our application specific test case could be:
Type c_invoices_test=
Class(c_test_case)
m_c_invoices: c_invoices;
Procedure Setup; Override;
Procedure test_invoices_clear;
Procedure TearDown; Override;
end; // c_invoices_test
Procedure c_invoices_test.Setup;
begin
m_c_invoices:= c_invoices.Create;
end; // Setup
Procedure c_invoices_test.test_invoices_clear;
Begin
l_c_invoices.add_invoice( ... );
l_c_invoices.clear_invoices;
If l_c_invoices.f_invoice_count<> 0
Then Raise c_failure.Create('count failure')
End; // test_invoices_clear
Procedure c_invoices_test.Teardown;
begin
m_c_invoices:= c_invoices.Create;
end; // Teardown
|
The addition of Setup and Teardown also forces us to rewrite the exception
code. We obviously want our framework to report any exception occuring in SetUp and Teardown, so this code now looks like:
procedure c_test_case.run_test_case;
begin
For l_c_test_method_ref In test_method_ref_list
Try
Setup;
l_c_test_method_ref.m_po_test_method ();
Teardown;
Except
report_failure_or_exception;
End; // try except
End; // run_test_case
|
In summary - our test program will systematically loop over all test cases
- the test case will loop over each test method
- each test method calls
- Setup (can be empty)
- the test method
- Teardown
This in fact guarantees that each test method gets a clean situation, avoiding any leftover side effects from some previous test method
Also note that
- writing Setup and Teardown code is optional
- therefore in our framework, Setup and TearDown are not Abstract but have an empty implementation
- if we want Teardown to be called even if the test method failed, we should use
procedure c_test_case.run_test_case;
begin
For l_c_test_method_ref In test_method_ref_list
Try
Setup;
Try
l_c_test_method_ref.m_po_test_method ();
Finally
Teardown;
End;
Except
report_failure_or_exception;
End; // try except
End; // run_test_case
| - obviously if some test methods (but not all) require additional
initializations, they have to be placed in those methods. Matching cleanup should then be performed before leaving the test case, usually in a Try Finally block
- the factoring of Setup and Teardown might be used as a criterion for creating new test cases: group all test methods requiring the same setup in a test case
- for the record, when a test case contains non-empty Setup and Teardown, it is no longer called a "test case", but is promoted to a "fixture". It's still looks like a test case to me, but, well, that's the way it is named in
the litterature.
2.5 - Creating and Running the test case list 2.5.1 - Adding an actual test case to c_test_case_list The simplest way is to create the test case and add it to the test case list
container in the main form. For instance:
Var my_c_test_case_list= c_test_case_list;
my_customer_test: c_customer_test;
my_invoice_test: c_invoice_test;
my_c_test_case_list:= c_test_case_list.Ceate;
my_customer_test:= c_customer_test.Create;
my_c_test_case_list.add_test_case(my_customer_test);
my_invoice_test:= c_invoice_test.Create;
my_c_test_case_list.add_test_case(my_invoice_test);
|
Since add_test_case uses a c_test_case parameter, and since c_my_test.Create returns such an instance, we could directly call the Constructor as the add_test_case parameter:
Var my_c_test_case_list= c_test_case_list;
my_c_test_case_list:= c_test_case_list.Ceate;
my_c_test_case_list.add_test_case(c_customer_test.Create);
my_c_test_case_list.add_test_case(c_invoice_test.Create);
|
We can even go a step further, by passing a c_test_case Class reference, and let add_test_case create the test case object. The framework would be:
type c_test_case=
Class(c_basic_object)
End; // c_test_case
c_test_case_ref= Class Of c_test_case; // class reference
c_test_case_list=
Class
Procedure add_test_case_class(p_c_test_case_ref: c_test_case_ref);
end; // c_test_case_list
Procedure c_test_case_list.add_test_case_class(
p_c_test_case_ref: c_test_case_ref);
begin
my_c_test_case_list.Add(p_c_test_case_ref.Create);
end; // add_test_case_class
| and in the main program, the part adding the actual test case:
Var my_c_test_case_list= c_test_case_list;
my_c_test_case_list:= c_test_case_list.Ceate;
my_c_test_case_list.add_test_case(c_customer_test);
my_c_test_case_list.add_test_case(c_invoice_test);
| This is the code we are actually using
However, we could still automate the creation better: - in the Initialization part of the framework Unit, we automatically create the g_c_test_case_list
- in each Unit containing some actual test case Class, in their Initialization part we call the addition of this test case to the list
The framework Unit would be:
Unit u_c_framework;
Interface
type c_test_case_list=
Class
Procedure add_test_case_class(
p_c_test_case_ref: c_test_case_ref);
End; // c_test_case_list
procedure add_the_test_case_class(p_c_test_case_ref: c_test_case_ref);
Implementation
...
var g_c_test_case_list: c_test_case_list= Nil;
procedure add_the_test_case_class(p_c_test_case_ref: c_test_case_ref);
begin
g_c_test_case_list.add_test_case_class(p_c_test_case_ref);
end; // add_the_test_case_class
Initialization
g_c_test_case_list:= c_test_case_list.Create;
end.
|
and a sample Unit containing an actual test case:
Unit u_c_invoice_test;
Interface
uses u_c_framework;
Type c_invoice_test=
Class(c_test_case)
end; // c_invoice_test
Implementation
Initialization
add_the_test_case_class(c_invoice_test);
End.
|
2.5.2 - Adding the test methods manually After including a new actual test case, we must generate the c_test_method_ref
for all the test methods which should be called. Manually, this can easily be done: - in the framework during the creation of the c_test_case ancestor we call a Virtual add_test_method method:
- in each test case, we override this method and add the test methods to the list
So the framework code is
type t_po_test_method= Procedure of Object;
c_test_method_ref=
Class
m_po_test_method: t_po_test_method;
Constructor Create(p_po_test_method: t_po_test_method);
end; // c_test_method_ref
c_test_case=
Class(c_basic_object)
m_c_method_ref_list: tList;
Constructor Create;
Procedure add_test_method(p_po_test_method: t_po_test_method);
Procedure add_test_methods; Virtual; Abstract;
End; // c_test_case
Constructor c_test_case.Create;
begin
...
add_test_methods;
end; // Create
Procedure c_test_case.add_test_method(p_po_test_method: t_po_test_method);
begin
m_c_method_list.Add(c_test_methdod_ref.Create(p_po_test_method));
end; // add_test_method
| and the actual test case code:
Type c_invoices_test=
Class(c_test_case)
procedure add_test_methods; Override;
Procedure test_invoices_add;
Procedure test_invoices_clear;
end; // c_invoices_test
Procedure c_invoices_test.add_test_methods;
begin
add_test_method(test_invoices_add);
add_test_method(test_invoices_clear);
end; // add_test_methods
Procedure c_invoices_test.test_invoices_add;
...
|
2.5.3 - Adding test methods automatically Instead of asking the developer to manually include all his test methods in a test case, we could automatically build this list by analyzing the test case.
This uses a rather low-level hack which explores the VMT of the c_test_case Class descendent: - The VMT contains the Published method names, and we can explore this list to get all the method names
- we get the method address using the MethodAddress(method_name) Function
- this is only the code address. To have a method pointer, we have to add the
data part. In this case, the Data is the c_test_case.Self (more accurately, the Self of the actual c_test_case descendent)
Here is this masterpiece of readabitlity and portability:
procedure c_test_case.add_published_methods(p_c_class_ref: tClass);
procedure _add_test_method(p_method_name: ShortString);
var l_pt_method_address: Pointer;
l_method_code_and_data: TMethod;
l_c_test_method_ref: c_test_method_ref;
l_po_test_method: t_po_test_method;
begin
display(p_method_name);
// -- only test under some condition
// -- in our case, check that
// -- - the Published method was not yet manually added
// -- - the method name contains "test" somewhere
if (m_c_test_method_ref_list.IndexOf(p_method_name)>= 0)
or (Pos('test', LowerCase(p_method_name))<= 0)
then Exit;
// -- get the method address from its name
l_pt_method_address:= MethodAddress(p_method_name);
if not (l_pt_method_address= nil)
then begin
// -- create the c_method_test, with a NIL method pointer for now
l_c_test_method_ref:= c_test_method_ref.create_test_method_ref(p_method_name, Nil);
// -- create a (code+data) method record.
// -- the code is this method
l_method_code_and_data.Code:= l_pt_method_address;
// -- the data is the c_test_case (NOT the c_test_method_ref)
l_method_code_and_data.Data:= Self;
// -- initialize the method pointer
l_c_test_method_ref.m_po_test_method:= t_po_test_method(l_method_code_and_data);
// -- add this test method to the list
if f_c_find_test_method_ref(l_c_test_method_ref)= Nil
then m_c_test_method_ref_list.AddObject(p_method_name, l_c_test_method_ref)
else l_c_test_method_ref.Free;
end
else display_bug_stop('no_methodaddress_for '+ p_method_name);
end; // _add_test_method
type t_method_table= packed record
m_vmt_method_count: SmallInt;
// -- the methods name array
end;
var l_pt_method_table: ^t_method_table;
l_pt_name: ^ShortString;
l_method_name_index, l_method_name_in_table_index: Integer;
begin // add_published_methods
// -- uses information aboutn System.MethodName
asm
mov EAX, [p_c_class_ref]
// -- get the pointer to the method table
mov EAX, [EAX].vmtMethodTable
mov[l_pt_method_table], EAX
end;
if l_pt_method_table<> nil
then begin
l_pt_name:= Pointer(PChar(l_pt_method_table)+ 8);
// -- analyze the Vmt
for l_method_name_index:= 1 to l_pt_method_table.m_vmt_method_count do
begin
_add_test_method(l_pt_name^);
// -- next name
l_pt_name:= Pointer(PChar(l_pt_name)+ length(l_pt_name^)+ 7)
end; // for l_method_name_index
end; // if l_pt_method_table<> nil
end; // add_published_methods
|
Please note that
- if we want to still be able to add to our actual test case Published methods which should not be run as part of the testing, we can adopt some naming convention. Asking the test method to contain "test" in its name it
the most frequent choice
- we fail to see why one would have a Published method in a test case, unless we would like to place some test cases on the Palette, to be able to
build the test framework from Palette test cases, which is usually not the case
- since methods could be used either by manually calling add_test_method or by letting the framework explore the Vmt, we must make sure that there is no
duplication.
- checking name duplication is easy, since our m_c_test_method_ref_list is a tStringList, and tStringList.IndexOf can be used.
- if the named given to add_test_method is not the real test method name (the developer used another name), we have to compare the method addresses.
Here is our duplication checking code:
function c_test_case.f_c_find_test_method_ref(
p_c_test_method_ref: c_test_method_ref): c_test_method_ref;
var l_test_method_ref_index: Integer;
begin
// -- first search by name
Result:= f_c_find_test_method_ref_by_name(p_c_test_method_ref.m_name);
if Result= Nil
then
for l_test_method_ref_index:= 0 to f_test_method_ref_count- 1 do
begin
if tMethod(f_c_test_method(l_test_method_ref_index).m_po_test_method).Code
= tMethod(p_c_test_method_ref.m_po_test_method).Code
then begin
Result:= f_c_test_method(l_test_method_ref_index);
Break;
end;
end;
end; // f_c_find_test_method_ref
| - to be published, a method is
- either a method of a $M+ Class, which is the case for tForms, for instance (default Published)
- or declared in an explicit Published section
For presentation articles, we usually do not use security levels (we do in production code though). So we first tried to surrounded the c_test_case Class with $m6 and $M-, and all the c_test_case's descendent methods will
be Published. This did work until we added a non-object member after the CLASS identifier (the boolean Enabled that will be presented below). So this $M+ rule only works if there are no non-object members. We could place
the non-object members in a later Private section, and use a Property to make them Public. But this is worse than not using security levels)
- so we chose the explicit Published route.
Once we start using security levels, we must go all the way and: - in the c_test_case ancestor, place
- the Constructor in a Public section
- the Virtual Setup and Teardown in Protected
- in the actual test case descendent, place
- the members which are initialized by Setup in Private
- the Override Setup and Teardown in Protected
- the test methods in Published
2.6 - The Main Program To run the test case list, we must
- build the test case list
- run it. Usually once is enough, but if we introduced side effects and dependencies among tests, we could have to run the test list (or some test cases) several time
After each test method run, the main program should be notified of the test result. This can be easily achieved using call-backs: in the framework:
c_test_case=
Class(c_basic_object)
m_c_parent_test_case_list: c_test_case_list;
Constructor create_test_case(p_name: String;
p_c_parent_test_case_list: c_test_case_list);
procedure run_test_case;
end; // c_test_case
t_po_after_test_method_event=
Procedure(p_c_test_method: c_test_method_ref;
p_success: Boolean; p_error_message: String) of Object;
c_test_case_list=
Class(c_basic_object)
m_after_test_method: t_po_after_test_method_event;
end; // c_test_case_list
procedure c_test_case.run_test_case;
begin
For l_c_test_method_ref In test_method_ref_list
Try
l_c_test_method_ref.m_po_test_method ();
Except
with m_c_parent_test_case_list do
If Assigned(m_after_test_method)
then m_after_test_method( ... );
End; // try except
End; // run_test_case
|
and in the main test form: type TForm1 =
class(TForm)
procedure build_test_case_list_Click(Sender: TObject);
private
procedure handle_after_test_method(
p_c_test_method: c_test_method_ref;
p_success: Boolean; p_error_message: String);
end; // tForm1
procedure TForm1.build_test_case_list_Click(Sender: TObject);
begin
g_c_test_case_list:= c_test_case_list.create_test_case_list('');
with g_c_test_case_list do
begin
add_test_case_class(c_stringlist_test);
...
m_after_test_method:= handle_after_test_method;
end; // with g_c_test_case_list
end; // build_test_case_list_Click
procedure TForm1.handle_after_test_case(p_c_test_case: c_test_case;
p_run_count, p_error_count: Integer);
begin
// display, log or whatever
end; // handle_after_test_case
|
Note that - since the main program only knows about the c_test_case_list (not the actual test cases). So the call-back events should be placed in the
c_test_case_list (and not in the individual c_test_cases). And to be able to call those, each c_test_case uses a m_c_test_case_list_parent link to its parent
- in the main program, could use any kind of display, log, journal or whatever we fell adequate to tell the user whether some of the tests failed.
2.6.1 - Enabling / Disabling test methods and test cases
When the developer is working in a narrow area of the project, it could be interesting to only run the tests concerned with this area. Of course this is contrary to the regression test concept, but could be a temporary time
optimization which should be removed after the temporary project fix. The selection of test methods or test cases can be implemented by a m_enabled Boolean which is tested before we run a test method or a c_test_case.
Since the main program only knows about the c_test_case_list, we would have to use "find_by_name" methods to toggle the c_test_method_ref.m_enabled or c_test_case.m_enabled members.
We already have the standard find_by_name_xxx function. However we have not implemented the toggling of m_enabled in main tForm example, since we are
going to use a tTreeView and will use this tTreeView to toggle the enabling.
2.6.2 - Test case list tTreeView Nearly all unit test framework, instead of using a textual feedback from the
tests, offer a graphical interface, with a tTreeView where - the main nodes represent the test cases
- the sub-nodes represent the test methods
In addition to the tTreeView, there usually some kind of textual output which presents the detail of the test exceptions and failures.
In our implementation
- the tTreeView is constructed from the c_test_case_list. This is the classical list-to-tTreeview code. And, as usual, each tTreeNode contains in
its Data member a reference to the c_test_case or c_test_method_ref:
procedure TForm1.build_test_case_list_Click(Sender: TObject);
procedure build_treeview;
var l_test_case_index: Integer;
l_c_test_case_treenode: tTreeNode;
l_test_method_index: Integer;
begin
with g_c_test_case_list, TreeView1.Items do
begin
BeginUpdate;
Clear;
for l_test_case_index:= 0 to f_test_case_count- 1 do
with f_c_test_case(l_test_case_index) do
begin
// -- add this at the level 0
l_c_test_case_treenode:= AddObject(Nil, m_name, f_c_self);
for l_test_method_index:=0 to f_test_method_ref_count- 1 do
with f_c_test_method(l_test_method_index) do
AddChildObject(l_c_test_case_treenode, m_name, f_c_self);
end;
TreeView1.FullExpand;
EndUpdate;
end; // with g_c_test_case_list, TreeView1.Items
end; // build_treeview
begin // build_test_case_list_Click
g_c_test_case_list:= c_test_case_list.create_test_case_list('');
with g_c_test_case_list do
begin
add_test_case_class(c_stringlist_test);
...
build_treeview;
end; // with g_c_test_case_list
end; // build_test_case_list_Click
| - clicking on a tTreeNode toggles the m_enabled Boolean. This is simply
performed by using the tTreeNode.Data reference of the c_test_xxx object
- the visual feedback uses
- for enabling / disabling a test case or a test method, 2 images (a checked square and an empty square)
- placed in a state_imagelist component
- referenced by tTreeView.StateImages
- the click on a tTreeNode assigns the correct image using the tTreeNode.StateIndex property
Also disabling a test_case disables all test_method sub_nodes (not necessary for the code, but better for the visual feedback)
- for success or failure, 3 image (before start, grey, success green and failure or exception red)
- those bitmaps are placed in a run_imagelist component
- this image list is referenced by tTreeView.Images
- the BeforeRun and AfterRun handlers set the color using the tTreeNode.ImgIndex property
And an error in one of the test methods also flags the parent test case as unsuccessful
2.7 - The Unit Test Framwork Classes Here are our Class definitions:
type c_failure= Class(Exception);
t_po_test_method= Procedure of Object;
c_test_method_ref= // one "test_method"
Class(c_basic_object)
// -- m_name: an optional method name
m_po_test_method: t_po_test_method;
m_enabled: Boolean;
Constructor create_test_method_ref(p_name: String;
p_po_test_method: t_po_test_method);
function f_c_self: c_test_method_ref;
function f_display_test_method_ref: String;
end; // c_test_method_ref
c_test_case_list= Class; // forward
c_test_case= // one "test_case"
Class(c_basic_object)
Public
// -- m_name: the test case name
m_c_test_method_ref_list: tStringList;
// -- to be able to fire the form events
m_c_parent_test_case_list: c_test_case_list;
m_enabled: Boolean;
Constructor create_test_case(p_name: String;
p_c_parent_test_case_list: c_test_case_list);
function f_c_self: c_test_case;
function f_display_test_case: String;
procedure display_test_method_list;
function f_test_method_ref_count: Integer;
function f_c_test_method(p_test_method_index: Integer):
c_test_method_ref;
function f_index_of(p_test_method_name: String): Integer;
function f_c_find_test_method_ref_by_name(
p_test_method_name: String): c_test_method_ref;
function f_c_find_test_method_ref(
p_c_test_method_ref: c_test_method_ref): c_test_method_ref;
procedure add_test_method(p_method_name: String;
p_c_test_method: t_po_test_method);
procedure add_test_methods; Virtual;
procedure add_published_methods(p_c_class_ref: tClass);
procedure check(p_condition: Boolean; p_text: String);
procedure run_test_case;
Destructor Destroy; Override;
Protected
procedure Setup; Virtual;
procedure Teardown; Virtual;
end; // c_test_case
c_test_case_class_ref= Class of c_test_case;
t_po_after_test_method_event=
Procedure(p_c_test_method: c_test_method_ref; p_success: Boolean;
p_error_message: String) of Object;
t_po_before_test_case_event=
Procedure(p_c_test_case: c_test_case) Of Object;
t_po_after_test_case_event= Procedure(p_c_test_case: c_test_case;
p_run_count, p_error_count: Integer) of Object;
c_test_case_list= // "test_case" list
Class(c_basic_object)
m_c_test_case_list: tStringList;
// -- mainly for gui display
m_after_test_method: t_po_after_test_method_event;
m_before_test_case: t_po_before_test_case_event;
m_after_test_case: t_po_after_test_case_event;
Constructor create_test_case_list(p_name: String);
function f_c_self: c_test_case_list;
function f_test_case_count: Integer;
function f_c_test_case(p_test_case_index: Integer): c_test_case;
function f_index_of(p_test_case_name: String): Integer;
function f_c_find_by_test_case(p_test_case_name: String): c_test_case;
procedure add_test_case(p_test_case_name: String; p_c_test_case: c_test_case);
procedure display_test_case_list;
procedure add_test_case_class(p_c_test_case_ref: c_test_case_class_ref);
procedure run_test_case_list;
Destructor Destroy; Override;
end; // c_test_case_list
|
The implementation has been presented in the previous sections, and you will find the full source code in the
downloadable .ZIP file
3 - Mini Manual
3.1 - The standard approach To use My Unit Test Framework  | write the code you want to test | |
| to test this code
| | create a TEST\ sub folder
| |
| place the U_C_TEST_CASE_LIST.PAS code in some known folder (can be in TEST\ of course)
| | |
write one or several test cases. This is the test case we used to explore and test the framework (not to be considered in any way as an example for testing lists or container, but syntactically correct):
unit u_c_tstringlist_test;
interface
uses Classes
, u_c_test_case_list_3
;
type c_stringlist_test=
class(c_test_case)
private
m_c_stringlist: TStringList;
procedure add_test_methods; Override;
// -- this test method has to be
// -- included from add_test_methods
procedure test_non_published_method_ok;
protected
procedure Setup; override;
procedure Teardown; override;
published
// -- all those test methods will be
// -- automatically included
procedure test_add_to_stringlist_with_failure;
procedure test_add_to_stringlist_ok;
procedure test_exception;
// -- a published method which is not a test
procedure published_method_no_in_tst;
end; // c_stringlist_test
implementation
USES SysUtils, u_display;
// -- c_stringlist_test
procedure c_stringlist_test.add_test_methods;
begin
add_test_method('add_fail_fff', test_add_to_stringlist_with_failure);
add_test_method('add_ok___ooo', test_add_to_stringlist_ok);
add_test_method('non_published_ok', test_non_published_method_ok);
end; // add_test_methods
procedure c_stringlist_test.Setup;
// -- create a stringlist with 3 strings
begin
m_c_stringlist:= TStringList.Create;
m_c_stringlist.Add('one');
m_c_stringlist.Add('two');
m_c_stringlist.Add('three');
end; // Setup
// -- the actual test methods
procedure c_stringlist_test.test_add_to_stringlist_with_failure;
begin
check(m_c_stringlist[1]= 'one', 'String[1] = ''one''');
end; // test_add_to_stringlist_with_failure
procedure c_stringlist_test.test_add_to_stringlist_ok;
begin
check(m_c_stringlist[0]= 'one', 'String[0] = ''one''');
end; // test_add_to_stringlist_ok
procedure c_stringlist_test.test_non_published_method_ok;
begin
check(m_c_stringlist[0]= 'one', 'String[0] = ''one''');
end; // test_non_published_method_ok
procedure c_stringlist_test.test_exception;
var l_x: Double;
begin
l_x:= 0;
l_x:= 1/ l_x;
end; // test_exception
procedure c_stringlist_test.Teardown;
begin
m_c_stringlist.Free;
end; // Teardown
procedure c_stringlist_test.published_method_no_in_tst;
begin
//
end; // published_method_no_in_tst
end | An empty skeletton would be:
unit u_c_my_xxx_test;
interface
uses Classes
, u_c_test_case_list_3
;
type c_my_xxx_test=
class(c_test_case)
private
// TO ADD 1 optional members
// initializes in Setup, cleaned up in TearDown
procedure add_test_methods; Override;
protected
procedure Setup; override;
procedure Teardown; override;
published
// TO ADD 2 the test methods
procedure test_xxx;
end; // c_my_xxx_test
implementation
USES SysUtils, u_display;
// -- c_my_xxx_test
procedure c_my_xxx_test.add_test_methods;
begin
// TO ADD 3 (optional) calls to
// add_test_method(name, method_identifier)
end; // add_test_methods
procedure c_my_xxx_test.Setup;
// -- create a stringlist with 3 strings
begin
// TO ADD 4 (optional)
// code to initialize some data or classes
end; // Setup
// -- the actual test methods
// TO ADD 5 the test method implementation
procedure c_my_xxx_test.test_xxx;
begin
// TO ADD 6
// some computation
// and some calls to check to see if actual=expected
end; // test_add_to_stringlist_with_failure
procedure c_my_xxx_test.Teardown;
begin
// TO ADD 7 (optional)
// code to cleanup the Setup Initialisations
end; // Teardown
end.
| | |
| place the P_TEST_xxx.DPR, P_TEST_xxx.PAS and U_TEST_xxx.DFM
| | Rename them
|
| | P_TESTt_xxx.PAS locate build_test_case_list_Click and write the lines which add your test case(s) to the test_case_list:
|
procedure TForm1.build_test_case_list_Click(Sender: TObject);
begin
g_c_test_case_list:= c_test_case_list.create_test_case_list('');
with g_c_test_case_list do
begin
// TO ADD 8 add the test case to the list
add_test_case_class(c_my_xxx_test);
...
end; // with g_c_test_case_list
end; // build_test_case_list_Click
| | | run
| | | click "build_test_case_list_" to create the c_test_case_list Here is a snapshot with our example (the tStringList test case, plus 2
other test case used to verify the failure and exception)  | |
| click "run_test_case_list_" to run the tests Here is a snapshot after the test run: 
| | |
Please note: - 8 steps looks quite a lot, but many of those are optional
- in addition, to ease the process, by using automatic actual test case generation and / or automating the creation of the test case list
3.2 - Automating the test case list construction 3.2.1 - Automatic creation implementation We already described how it is possible to automatically initialize the c_test_case_list:
- the g_c_test_case_list must be create before anything (or better created as soon as referenced using a singleton pattern)
unit u_c_test_case_list_3;
interface
...
function f_c_test_case_list: c_test_case_list;
implementation
...
// -- singleton
var g_c_test_case_list: c_test_case_list= Nil;
function f_c_test_case_list: c_test_case_list;
begin
if g_c_test_case_list= Nil
then g_c_test_case_list:=
c_test_case_list.create_test_case_list('');
Result:= g_c_test_case_list;
end; // f_c_test_case_list
begin // u_c_test_case_list
end. // u_c_test_case_list
| - each actual test case unit calls the method adding its Class to the test
case list:
unit u_c_automatic_stringlist_test;
interface
uses Classes
, u_c_test_case_list_3
;
type c_automatic_stringlist_test=
class(c_test_case)
...
end; // c_automatic_stringlist_test
implementation
...
Initialization
f_c_test_case_list.add_test_case_class(
c_automatic_stringlist_test);
end.
|
- we rewrite the main project .DPR to force the loading of all the actual test cases BEFORE the creation and initialization of the gui form creation:
program p_automatic_stringlist_test;
uses
// TO ADD all the units with actual test cases
u_c_automatic_stringlist_test
, Forms
// -- the main gui
, u_test_gui
;
begin
// -- here the g_c_test_case_list has been created
// -- and all the test_cases in the units u_c_xxx have
// -- been added to this list
Application.Initialize;
// -- the tTreeview creation will be performed in Form1.OnCreate
Application.CreateForm(TForm1, Form1);
Application.Run;
end.
|
All the tTreeView initialization is moved in a initialize_treeview method, and this method must be called AFTER the c_test_case_list has been created and initialized. The two options are
- either place all actual test case units at the start of the Uses clause, and include a call of initialize_treeview in tForm1.OnCreate (the solution above)
- or just add the actual test case units somewhere in the Uses, but manually call Form1.initialize_treeview manually after Application.CreateForm(TForm1, Form1)
Note that
- the beauty of this approach is to keep the main Gui form totally independent of any actual test case. So all the tester has to do is
- write his actual test case Units
- add the small .DPR
- the 3 lines in the .DPR body even do not depend on any actual test case. So we can
- place them in a procedure, say initialize_the_test, nested in the main form Unit
- call THIS method from the main body:
program p_automatic_test_2;
uses u_test_gui
// TO ADD all the units with actual test cases
, u_c_automatic_stringlist_test
;
// -- avoid a "console" icon
{$R *.RES}
begin
initialize_test_gui;
end
|
- on the downside, this automatic technique forces the tester to write a .DPR (that's not very difficult), and when he runs this simple .DPR, he will see
the test form popping up from nowhere. He can certainly guess what's going on, but this is still unusual from a standard Delphi programming point of view.
3.2.2 - Automatic creation Manual The overall organization is:
And the steps are the following - prepare for testing
|
| place the U_C_TEST_CASE_LIST_3.PAS, U_TEST_GUI.PAS and U_TEST_GUI.DFM files in some known, application independent, folder (like HELPERS\UNIT_TEST\ in our case
| - code and test
|
| write the project (or the components or whatever) to be tested in some folder (say PROGRAM\)
| | |
to test this code
- create a subfolder (usually a subfolder named TEST, like PROGRAM\TEST\)
- write your actual test case units
- add your c_test_case Class descendents
- AND in the Initialization part of the Unit, call add_test_case(...) for each of these Classes
- write a simple .DPR text, with
- include U_C_TEST_CASE_LIST_3 and U_TEST_GUI in this project
(either in the same directory, or use "Project | Add", or initialize "Project | Options | Directories | Search pathes)
- in the USES clause
- add U_TEST_GUI
- add all your actual test units names somewhere in the Uses
clause
- in the body, add the call of initialize_test_gui
|
3.3 - Actual test case generation The structure of the code of each actual test case Unit is quite standard:
- a class inheriting from c_test_case
- optionally definitions and bodies for Setup and Teardown
- at least one test method definition and its body
The skeletton for such Units can be derived from the the Class under test (changing c_invoice in PROGRAM\U_C_INVOICE.PAS into c_invoice_test in a new PROGRAM\TEST\U_C_INVOICE_TEST.PAS).
We've already wrote and presented here many articles containing Delphi code for scanners and lexical analyzers (finding all xxx=CLASS) as well as unit generators (writing Unit u_c_test_xxx; Interface etc in a .PAS disc file).
But this does not solve the problem, mainly because there is no correspondance between a Class to test and its test case - some Classes are trivial and do not require any test
- some Classes may need several test cases (to benefit from different Setup and Teardown), and, as the project grows, many Classes are included in some test cases (to test their interaction)
- some test case are not tied to any Class at all (testing the date etc)
So we prefer to use a small set of predefined skeletton Units and use find and replace in the copies of those Units.
Here is an example of the "test case with Setup and Teardown" skeletton:
unit u_c_my_xxx_test;
interface
uses Classes
, u_c_test_case_list_3
;
type c_my_xxx_test=
class(c_test_case)
private
protected
procedure Setup; override;
procedure Teardown; override;
published
procedure test_yyy;
end; // c_my_xxx_test
implementation
USES SysUtils, u_display;
// -- c_my_xxx_test
procedure c_my_xxx_test.Setup;
begin
end; // Setup
procedure c_my_xxx_test.test_yyy;
begin
end; // test_yyy
procedure c_my_xxx_test.Teardown;
begin
end; // Teardown
end.
|
4 - Comments and Improvements 4.1 - Improvements As usual, there are many things that could be improved. To name a few, in the framework unit: - use different callbacks (or some parameter) for setup exception, test
exception, teardown exception.
- add stack trace on error
- create all the check_boolean, check_string, check_equals etc
- create a common c_enabled_test with only a single m_enabled member, and
derive c_test_method and c_test_case from this Class. This would remove the IS test in the Gui
- add a Decorator Class to add loops and repetitions in the test methods, or
allowing specific code before and after each test
On the Gui side - use different colors to display the different types of errors (failure, and the 3 kind of exceptions)
- save and load the the enabled/disabled test cases / test methods from one run to the next one
Last, but not least, build a test suite for the unit test framework ! Granted
our example c_stringlist_test together with c_setup_exception_test and c_teardown_exception_test Classes are some kind of visual test. But not Unit
Test (calls to check to flag that actual<> expected), and by not reasonably covering all situations by a long shot.
4.2 - couple of things we learned Here is the tObject and related Uml Class Diagram:
The vmtMethodTable is defined in SYSTEM.PAS, and many examples of Mov xxx, [Eax].vmtMethodTable
can be found in this Unit
And for the record, it took us - a couple of days (long ago) to analyze the dUnit code, one day for xUnit4Delphi
- around 2 days to code the MyUtf framework
- around 2 days to write the article
4.3 - Why not use dUnit or xUnit4Delphi ? Many might scoff at our coding attempt as being a consequence of a king-sized
"not invented here" syndrom, or a compulsive drive to understand "how it works". In addition to those deeply rooted motivations, we were unhappy with writing this .DPR which would trigger this unknown GUI form. It turns out that this is
seems to be the best alternative, but having written MyUtf we understand the cause and the motivation. While trying to understand the working of dUnit, we spent a couple of days to figure out the architecture of the framework unit. We later looked at
xUnit4Delphi which is half as complex. Of course, there is no obligation to understand the details of a framework in order to use it. We use all day long the Delphi framework without having the possibility to look at the actual IDE
implementation, and have no anxiety to work with it. Just to give you an idea of the complexity size of the three frameworks: - dUnit
- framework unit :
- 84 K (7 Interfaces and 7 Classes, 151 global functions)
- Interfaces like iTestListener, Classes like tStatusToResultAdapter
- gui form : 40 K
- xUnit4Delphi
- framework
- 46 K (4 Interfaces and 8 Classes, 8 global functions)
- 9 K assert Unit (AssertEquals Assert_xxx)
- code with
- 3 InterlockedDecrement or InterlockedIncrement
- iTestListener, tListIterator, tTestIterator, iTestFailureIterator
- gui form : 16 K
- MyUtf
- framework
- 22 K
- 3 classes (well, 2 classes and a c_test_method_ref which could be a Record) and a global function
- gui 16 K
For dUnit, the main feature seems to be the ability to include as many design
patterns as possible. This is no surprise, since the original was written by some of the founding fathers of the design pattern movement. xUnit4Delphi is mainly characterized by the "design to the interface" concept.
This explains this flood of iterators. Granted they allow to implement the two testing loops (over the test cases and over the test methods) in any way we want (Arrays, Lists etc). And the Listener allows to use different kinds of
main program (console, Gui). There are also general development concepts like The Dependency Inversion Principle Our little piece of code can be considered as following Niklaus WIRTH's
favorite quote from Albert EINSTEIN: "make things as simple as possible, but not simpler". So instead of iterators we used two For loops, and instead of Listeners, 3 event callbacks. Still, overall, to the best of our knowledge,
the functionality is the same (if you consider this to be wrong, you are welcome to add a comment below).
So our code can be looked down as overly simplistic. It is like driving a Ford
Model T, compared to flying the Space Shuttle or piloting a 747. But we also wrote this simple Unit Test Framework in order to able to easily modify or upgrade it. For instance, we could add:
- a stack trace (which are the calls which let to the exception / failure)
- a memory leak check
- a profiling of the code performance
So the same motivation that brought Unit Test (a framework for non intrusive
testing many aspects of our code, and regression testing) are the same for many areas of software development (non intrusive and regressive loop over some code interaction pieces). You sure are clever enough to put your foot in those
Interfaces, Listeners, Adapters etc, in order to add other functionalities to a Unit Test Framework. We feel more happy to work on simpler code.
4.4 - What's next
The really important thing is to really use the Unit Test during coding. Become "test infected" as Kent BECK explained. And using "test code test code test code", instead of "code code code (maybe) test" also has many important side effects:
- force us to specify what a Class should do, before starting to code the Class
- "code to the interface", which, in a nutshell, means look at the functionality, not the implementation
- apply the "Dependency Inversion Principle"
In addition, Unit Test is also an essential part of eXtreme Programming, which stresses that it is impossible to forecast everything at project start,
and that an incremental development will uncover unforseable problems. Unit Test is then use to make sure that new added increment do not break previous steps.
However presenting those topics is beyond the scope of this article.
5 - Download the Sources Here are the source code files: The .ZIP file(s) contain: - the main program (.DPR, .DOF, .RES), the main form (.PAS, .DFM), and any
other auxiliary form
- any .TXT for parameters, samples, test data
- all units (.PAS) for units
Those .ZIP - are self-contained: you will not need any other product (unless expressly mentioned).
- for Delphi 6 projects, can be used from any folder (the pathes are RELATIVE)
- will not modify your PC in any way beyond the path where you placed the .ZIP (no registry changes, no path creation etc).
To use the .ZIP:
- create or select any folder of your choice
- unzip the downloaded file
- using Delphi, compile and execute
To remove the .ZIP simply delete the folder.
The Pascal code uses the Alsacian notation, which prefixes identifier by program area: K_onstant, T_ype, G_lobal, L_ocal, P_arametre, F_unction, C_lass etc. This notation is presented in the Alsacian Notation paper.
The .ZIP file(s) contain:
- the main program (.DPROJ, .DPR, .RES), the main form (.PAS, .ASPX), and any other auxiliary form or files
- any .TXT for parameters, samples, test data
- all units (.PAS .ASPX and other) for units
Those .ZIP
- are self-contained: you will not need any other product (unless expressly mentioned).
- will not modify your PC in any way beyond the path where you placed the .ZIP
(no registry changes, no path outside from the container path creation etc).
To use the .ZIP: - create or select any folder of your choice.
- unzip the downloaded file
- using Delphi, compile and execute
To remove the .ZIP simply delete the folder. The Pascal code uses the Alsacian notation, which prefixes identifier by program area: K_onstant, T_ype, G_lobal, L_ocal, P_arametre,
F_unction, C_lass etc. This notation is presented in the Alsacian Notation paper.
As usual:
- please tell us at fcolibri@felix-colibri.com if you found some errors, mistakes, bugs, broken links or had some problem downloading the file. Resulting corrections will
be helpful for other readers
- we welcome any comment, criticism, enhancement, other sources or reference suggestion. Just send an e-mail to fcolibri@felix-colibri.com.
- or more simply, enter your (anonymous or with your e-mail if you want an answer) comments below and clic the "send" button
- and if you liked this article, talk about this site to your fellow developpers, add a link to your links page ou mention our articles in your blog or newsgroup posts when relevant. That's the way we operate:
the more traffic and Google references we get, the more articles we will write.
6 - References Just a couple of links
7 - The author
Felix John COLIBRI works at the Pascal Institute. Starting with Pascal in 1979, he then became involved with Object Oriented Programming, Delphi, Sql, Tcp/Ip, Html, UML. Currently, he is mainly
active in the area of custom software development (new projects, maintenance, audits, BDE migration, Delphi
Xe_n migrations, refactoring), Delphi Consulting and Delph
training. His web site features tutorials, technical papers about programming with full downloadable source code, and the description and calendar of forthcoming Delphi, FireBird, Tcp/IP, Web Services, OOP / UML, Design Patterns, Unit Testing training sessions.
|
