Course Content‎ > ‎

Section 11: Testing

Software testing is a process used to verify that software:
  • works as expected
  • fulfils the specified design requirements
  • satisfies the end-product customers
According to the ANSI/IEEE 1059 Standard, Testing can be defined as:
"A process of analysing a software item to detect the differences between existing and required conditions (that is defects/errors/bugs) and to evaluate the features of the software item."

In an organisation, a number of different individuals can be responsible for testing.  Larger organisations typically have a team with responsibility for testing, but a number of different types of users could be involved:
  • Software Testers  - specifically hired for this role
  • Software Developers - generally through unit testing
  • Project Managers/Leads
  • End Users     - for usability testing and commonly some compatibility testing
There are a range of different types of testing including:
- Unit Testing - testing of small components (units) of code within developed software
- Functional Testing - testing of software functions as a "black box" by feeding them inputs and examining outputs
- Integration Testing - combining of modules (previously unit tested) and testing the modules as a combined group to examine expected outputs
- Usability Testing - testing how usable are developed systems and customer experiences with these systems.  Do customers like the system?
- Compatibility Testing - testing how the software behaves under different operating systems, browser types and devices.
- many more...

While we could build an entire course/module on testing, we will focus on just one of these for this module -> Unit Testing

What is Unit Testing?

Unit tests allow us to verify that small sections of software code are working correctly.  "Units" are intended to be the smallest testable part of an application and generally would be code in a method or function. Unit tests are generally written and executed by developers to ensure that code meets design specifications and behaves as intended.  There are a number of approaches that can be taken with Unit testing, but in essence it means that developers would code tests into every small unit of code they write.

Why perform Unit Testing?

From the description above, an obvious disadvantage is highlighted; developers are now spending more time writing tests in addition to the core code they are already expected to write.  There is certainly a significant overhead with this on initial writing of code.  So let us consider just a few of the advantages:
  1. Problem Debugging: When unit tests are written, an effective agreement is set up to indicate what the code must satisfy.  This means that when we run our tests at a later date, every single unit is tested simultaneously.  Frequent execution of unit tests, particularly in a multi-user programming environment will not only help to debug problems, but will often prevent them from occurring in the first instance.
  2. Change Management: Consider a high level API method/function written, which is in turn used by dozens of other classes, potentially written by other developers.  Without testing, a single developer might make code changes in a method which does not affect their own code negatively.  They commit the changes and there are subsequent problems discovered in classes which had not been changed.
    Debugging Changes
     Considering each of the colours above as representing a different developer - BLUE makes a change to the parent API and manually tests his own code.  BLUE may not be aware that the code changes to the API class function may negatively affect code elsewhere in the company application.  ORANGE's code starts failing through no fault of their own and considerable time might be spent in tracking down sudden failures in their code.  How does unit testing help here?  With unit testing, all four programmers can view the effective "contract" that the API class must adhere to.  They write their own code accordingly in line with the terms of that contract.  When BLUE makes changes now that break this contract, this will be immediately picked up by unit testing.  The problem must then be acknowledged and fixed immediately in the API class, rather than potentially passing problems to other developers.
  3. Documentation: By using unit testing, developers essentially create a dynamic, every-changing set of documentation for the code of a system.  All developers can subsequently look at the unit tests in order to gain an understanding of the unit's programming interface.  In doing this, they can discover where and when it is appropriate to use a particular unit of code (and likewise when it should not be used).  In many ways, this can be an improvement on other forms of documentation, particularly where the documentation has not been kept up to date.  

What is JUnit?

JUnit is a unit testing framework for the Java programming language.  It is open source and allows us to write and run repeatable tests and to incrementally build test suites to detect unintended side effects.   Tests can be run continuously, with results provided immediately, meaning that problems can be detected and remedied quickly.  The JUnit framework can be easily integrated with Eclipse, IntelliJ, Ant, Maven etc. and has it's own graphical user interface (GUI).


Installing JUnit

Installing JUnit is a relatively straightforward set of steps, considering that we have already set up environment variables previously for this module.  
  1. Visit junit.org 
  2. Visit the download area
  3. Download the latest version of the JUnit JAR file and the latest version of the Hamcrest JAR file (Hamcrest is a framework that assists in writing software tests and matchers). At the time of writing these notes, the versions are: junit-4.11.jar and hamcrest-core-1.3.jar
  4. Configure our path to include these jar files during compilation/execution.  This is similar to the steps described in our Downloads section relating to the installation of Java.
    On Windows 7 open the search bar and type 'environment' and choose 'Edit the System Environment Variables' -> Environment Variables 
  • Under 'System Variables' select 'New' -> For the name add 'JUNIT_HOME' and add the location where you have stored the junit-4.11.jar file.  In my case this is C:\Courses\EE417\jars.  
  • If the 'CLASSPATH' environment variable does not already exist, similarly add the content below.  This sets the CLASSPATH to include the JUnit and Hamcrest jar files and the current working directory (represented by the dot).  If you have an existing CLASSPATH variable, append this appropriately.             
CLASSPATH
.;%JUNIT_HOME%\junit-4.11.jar;%JUNIT_HOME%\hamcrest-core-1.3.jar;  
Assuming our following examples work correctly - JUnit has now been set up correctly.

Running Some Tests

So let's write our first test class.  This class has one test for now, which does a simple string compare assertion (more on this later!).

import org.junit.Test; 
import static org.junit.Assert.assertEquals; 

 public class MyTest { 
   @Test 
   public void firstTest() {
System.out.println("MyTest: Inside firstTest()"); 
  String str= "First Test Passed"; 
  assertEquals("First Test Passed",str); 
   } 
} 
Source file: MyTest.java     

To compile this code, navigate to the folder where you have saved the file and use:

javac MyTest.java

If this completes successfully, you will not see any output.  A new file will have been created in the same folder 'MyTest.class'
MyTest compilation output
                                Figure: Output from the compilation of MyTest.java

You may have noticed that this Java class does not have a main method (nor is it an applet/servlet/JSP etc.) so it can not be run directly using 'java MyTest'.  If you attempt to do this, you will receive a 'Main method not found' error message.
So let us create a class which will run our test(s).  

import org.junit.runner.*; 
import org.junit.runner.notification.Failure; 

public class MyTestRunner { 
 public static void main(String[] args) { 
  Result result = JUnitCore.runClasses(MyTest.class); 
  for (Failure failure : result.getFailures()) { 
  System.out.println(failure.toString()); 
  } 
  System.out.println("Tests Successful = " + result.wasSuccessful()); 
 } 
} 
Source file: MyTestRunner.java     (TODO)

After compiling with javac, we can now perform the following:

java MyTestRunner

to receive the following output:
Tests Successful = true

Congratulations - you have just written your first test!


Example #1

Consider the following code.  The purpose of this code is to take a single argument and test whether this argument is a prime number.  The definition of a prime number is : 
"An integer greater than one is called a prime number if its only positive divisors (factors) are one and itself. "

import java.math.BigInteger;
class CheckPrime {
    public static void main(String[] args) {
        System.out.println("Welcome to TestPrime"); 
        if (args.length!=1) {
              System.out.println("Please provide a number, in the format: java CheckPrime 15");
              System.exit(0);
        }
        else {
                     System.out.println("Your inputted number is " + args[0]);
              System.out.println("Prime: " + isPrimeMethod(args[0]));
        }
    }
    
    public static boolean isPrimeMethod(String s) {
        BigInteger i = new BigInteger(s);
        return i.isProbablePrime(1);   
    }
   
}

Question:

i) Describe a scenario where the method "isPrimeMethod(String s)" will fail.  
ii) Create a unit test class 'PrimeTest.java' which will test this scenario and report a failure.  
iii) Fix the code in CheckPrime.java to satisfy the test from part ii)

Notes: The following PrimeTestRunner.java file has been provided.
You can assume that the certainty value of "isProbablePrime(certainty)" method will be accurate.  In reality this is quite a complex method, which allows a trade-off of speed vs accuracy using various mathematical algorithms.

import org.junit.runner.*; 
import org.junit.runner.notification.Failure; 

public class PrimeTestRunner { 
 public static void main(String[] args) { 
  Result result = JUnitCore.runClasses(PrimeTest.class); 
  for (Failure failure : result.getFailures()) { 
  System.out.println("Test Failure: " + failure.toString()); 
  } 
  System.out.println("Tests Successful = " + result.wasSuccessful()); 
 } 
} 

Solution:

i) The isPrimeMethod(String s) method will fail in the case where 's' cannot be converted to a BigInteger.  For example, we will not be able to run the application with 'java CheckPrime abc'.  This will cause a NumberFormatException at run time.

ii) The following file contains two tests, one which checks normal numeric operation and one which tests non-numeric operation. If a non-numeric parameter is passed in, we should expect isPrimeMethod() to return false.  This test will report a failure in this case though due to a NumberFormatException.
 
import org.junit.Test; 
import static org.junit.Assert.assertEquals; 

public class PrimeTest { 
 @Test 
 public void testPrimeNumeric() {
System.out.println("PrimeTest: testPrimeNumeric()");  
  assertEquals(true,CheckPrime.isPrimeMethod("5")); 
 } 

 @Test 
 public void testPrimeNonNumeric() {
System.out.println("PrimeTest: testPrimeNonNumeric()");                          
    assertEquals(false,CheckPrime.isPrimeMethod("a")); 
 } 

} 

iii)
To fix our code, we should handle the NumberFormatException that would be thrown in this case.  We can do this easily by catching the exception and returning false.  

import java.math.BigInteger;

class CheckPrime {
    public static void main(String[] args) {
        System.out.println("Welcome to TestPrime"); 
        if (args.length!=1) {
              System.out.println("Please provide a number, in the format: java CheckPrime 15");
              System.exit(0);
        }
        else {
System.out.println("Your inputted number is " + args[0]);
              System.out.println("Prime: " + isPrimeMethod(args[0]));
        }
    }
    
   public static boolean isPrimeMethod(String s) {
try {
              BigInteger i = new BigInteger(s);
              return i.isProbablePrime(1);   
} catch (NumberFormatException e) { return false; }
   }
   
}

If we try to pass a non-numeric to isPrimeMethod, an exception will occur when we try to call the constructor for BigInteger.  We can catch this Exception and return false whenever the constructor fails.  This meets the requirements for what we are trying to demonstrate and if we were to run our test again, we will see that the tests all pass successfully.

Solution Output


When to test?  (How small are units?)

When we write unit tests, we are testing our code and not the compiler.  A good example of this relates to asking 'Should we test get() and set() methods in our JavaBeans?'.  Let us consider a simple Java bean similar to those encountered earlier in the course: 

public class User {
private String firstname;
private String surname;

public User(String firstname, String surname) {
this.firstname = firstname;
this.surname = surname;
}
 
  public String getFirstname() {
return firstname;
}
public void setFirstname(String firstname) {
this.firstname = firstname;
}
public String getSurname() {
return surname;
}
public void setSurname(String surname) {
this.surname = surname;
}
}
Source: User.java

Now let us create a test like the following:
import org.junit.Test; 
import static org.junit.Assert.assertEquals; 

public class UserTest { 
 @Test
    public void testGetSetFirstname() {
 System.out.println("UserTest: Inside testGetSetFirstname()"); 
         User user = new User();
 user.setFirstname("David");
         assertEquals("David", user.getFirstname());
 }
} 

The previous 'MyTestRunner' can be modified slightly as below, executed using 'java MyTestRunner' and all of our tests should be performed and passed.  

Result result = JUnitCore.runClasses(MyTest.class,UserTest.class);
 

But what have we actually tested?
Let us consider our method in a shorter form:
public void testGetSetFirstname() {
    firstname = "David";
    assertEquals("David", firstname);
}

and let us go a step further:
public void testGetSetFirstname() {
    assertEquals("David", "David");
}

By the time we have written our test in this minimised format, we will have realised that what we are testing is the Java compiler and not actually any particular functionality we have written ourselves.  As a developer, it is not your responsibility to test Java (let Oracle do this!).  In fact, your starting point as a Java programmer needs to be to assume that the programming language will perform as expected.

It is for this reason that get() and set() methods would not generally be tested.  However, it might be appropriate where the developer has written a more complex get() or set() method which has some form of implemented business logic such as parsing, sub-stringing, null-checking or appending.  Unit tests are designed to identify scenarios where something might break.  If you have written code that might break, then it is appropriate to add in a unit test.  

Equally, you might want to test whether an object has already been set appropriately, which can be done by adding a test method to test the constructor.  For example:

@Test
public void testCreateUser() {
     assertEquals("David", new User("David", "Molloy").getFirstname());
}
This test makes sure that we have initialised our variables correctly in the constructor and can be useful where we might have different constructors initialising things differently.  


Taking things a step further

Taking our User class (we'll now call it Client to avoid confusion) and the constructor idea as a starting point, let us create a new scenario.  We want to add in a 'getInitials()' method which will rely on some business logic to have been previously performed in the constructor.  Let's look at a full class here that we wish to test:

public class Client {
     private String firstname;
     private String surname;
  private String initials;

  public Client() {}

     public Client(String firstname, String surname) {
         this.firstname = firstname;
         this.surname = surname;
     this.initials = firstname.substring(0,1) + surname.substring(0,1);
     }
 
     public String getFirstname() {
         return firstname;
     }
     public void setFirstname(String firstname) {
         this.firstname = firstname;
     }
     public String getSurname() {
         return surname;
     }
     public void setSurname(String surname) {
         this.surname = surname;
     }
  public void setInitials(String initials) {
     this.initials = initials;
  }      
     public String getInitials() {
   return initials;
}
}
Source: Client.java  

The part that we are really interested in testing is whether the code marked in red will function correctly for all test cases.  A more experienced programmer is going to immediately recognise certain scenarios where this code would either success or fail.  Examples:
  • Firstname: "David", Lastname: "Molloy"  -> getInitials should work correctly returning "DM"       Desired Output: 'DM'       Actual output: 'DM'
  • Firstname: "", Lastname: "Molloy" -> getInitials will fail because we cannot get substring(0,1) on a string with 0 characters.   Desired Output: 'M'       Actual output: Runtime Error
  • Firstname: "David", Lastname: null   -> getInitials will fail with a NullPointerException as you cannot get substring(0,1) on a null variable      Desired Output: 'D'       Actual output: Runtime Error
These are known as runtime exceptions, meaning that the problems only occur at run time and not at compilation time (we had no problem using 'javac').  The problems will occur in some scenarios and not in others - hence the usefulness for testing.

We have three main options here:
1) Write the production code, do some local ad-hoc testing and consider the code working          (Not a formal testing approach)
2) Write the production code, then write some formal unit tests and ensure they pass                 (As we did in Example 1)
3) "Test Driven Development"


Test-Driven Development

Upon identifying the business logic to be implemented, the majority of software developers tend to jump straight into writing code.  The code is written, loosely tested for functionality and then deployed.  

Test-driven development is a fundamentally different approach focused around test-first development where the software developer writes tests before writing any production code.  This encourages the developer to think through the design and requirements of the software system before writing functional code.  

1) The developer firstly writes an automated test case which defines the new functionality or business logic to be implemented.  
2) This should initially fail as the production code should not yet exist.  If the test does not fail, it is either dysfunctional or the functionality already exists in the production code.  
3) After the test has been written, the developer writes the minimal amount of production code required to pass the test.  At this stage, the code is not expected to be elegant or final in any way, but should be enough to pass the test.
4) All of the tests are now run again - If this test succeeds, then the code meets all of the tested requirements.  
5) Code can now be cleaned up, duplication removed, comments added.  Following this stage, we should have final, clean production code which covers this new functionality and any functionality previously implemented under our test-driven process.  

Test-driven development Cycle
Test-driven Development Cycle  - Source: Wikipedia : (http://en.wikipedia.org/wiki/File:Test-driven_development.PNG)

Let's start again, using test-driven development.  Our starting code will look like this:  

public class Client{
     private String firstname;
     private String surname;

     public Client(String firstname, String surname) {
         this.firstname = firstname;
         this.surname = surname;
     } 
     public String getFirstname() {   return firstname;   }
     public void setFirstname(String firstname) {   this.firstname = firstname;     }
     public String getSurname() {   return surname;     }
     public void setSurname(String surname) {      this.surname = surname;     }
     public String getInitials() { return null; }// empty for now
     public void setInitials(String initials) {} // empty for now
}
Source: Client.java      

Let us now consider that we have been given some form of design brief:
1) We need to provide a method whereby we can get the initials of a client (first letter of firstname followed by first letter of surname). E.g. "David Molloy -> "DM"
2) If either surname or firstname are null they should be replaced by a blank character for the purposes of initials
3) If either surname or firstname are blank they should be replaced by a blank character for the purpose of initials

Because we are using TDD, we are not going to write production code yet.  Instead, we will write a test first!  In fact, we are going to write three separate tests to cater for the three points of functionality in the design brief.  The first of these is to test the standard scenario (non-null and non-blank)

import org.junit.Test; 
import static org.junit.Assert.assertEquals; 

public class ClientTest { 
 @Test 
 public void testInitialsStandard() {
System.out.println("ClientTest: Inside testIntialsStandard()"); 
  Client client = new Client("David", "Molloy");
    assertEquals("DM",client.getInitials()); 
 } 
} 
Source: ClientTest.java

We will need to write a simple runner class in order to run this test (and subsequent tests).  

import org.junit.runner.*; 
import org.junit.runner.notification.Failure; 

public class ClientTestRunner { 
 public static void main(String[] args) { 
  Result result = JUnitCore.runClasses(ClientTest.class); 
  for (Failure failure : result.getFailures()) { 
  System.out.println("Test Failure: " + failure.toString()); 
  } 
  System.out.println("Tests Successful: " + result.wasSuccessful()); 
 } 
} 

After compiling, and running our ClientTestRunner we FAIL the test as expected (and hoped):
Client Test Runner Output

As this test, has now failed, the next step is to write some production code to make it pass.  We can do this easily and have in fact done this before.
We simply edit Client.java to add in this new functionality.

public class Client{
     private String firstname;
     private String surname;
     private String initials;

     public Client(String firstname, String surname) {
         this.firstname = firstname;
         this.surname = surname;
     this.initials = firstname.substring(0,1) + surname.substring(0,1);
     } 
     public String getFirstname() {   return firstname;   }
     public void setFirstname(String firstname) {   this.firstname = firstname;     }
     public String getSurname() {   return surname;     }
     public void setSurname(String surname) {    this.surname = surname;     }
  public String getInitials() { return initials; } 
  public void setInitials(String initials) {   this.initials=initials;} 
}
Source: Client.java

Now, we can run our test again and we will see that the test passes!
ClientTestRunner New output
There is nothing much we can do to "clean up" the code with such a simple example.  This would be done at this point if appropriate.

Let's add a test for the second piece of functionality (Number 2 above) - we want to test for the cases where either the firstname is null or the surname is null.  We don't try to cater for this in the production code (yet!), but instead we write the test.

import org.junit.Test; 
import static org.junit.Assert.assertEquals; 

public class ClientTest { 
  @Test 
  public void testInitialsStandard() {
System.out.println("ClientTest: Inside testIntialsStandard()"); 
  Client client = new Client("David", "Molloy");
    assertEquals("DM",client.getInitials()); 
  } 

  @Test
  public void testGetInitialsWhereNull() {
    System.out.println("ClientTest: Inside testGetInitialsWhereNull()");
         
    Client client = new Client(null, null);
    assertEquals("",client.getInitials());  

    client= new Client("David", null);
    assertEquals("D",client.getInitials()); 

    client= new Client(null, "Molloy");
    assertEquals("M",client.getInitials());   
 }

} 

Similar to before, if we run our ClientTestRunner we expect this test to FAIL. The production code has not yet been written to cater for these scenarios.  The output should look as follows:
ClientTestRunner Expected Output
In line with the diagram, the next step is to write some production code to allow this test to pass.  You can see that the code has changed fairly fundamentally to cater for these new null scenarios.

public class Client{
     private String firstname;
     private String surname;
     private String initials;

     public Client(String firstname, String surname) {
         this.firstname = firstname;
         this.surname = surname;
         initials = "";
     if (firstname!=null) initials = initials + firstname.substring(0,1);
         if (surname!=null) initials = initials + surname.substring(0,1);
     } 
     public String getFirstname() {   return firstname;   }
     public void setFirstname(String firstname) {   this.firstname = firstname;     }
     public String getSurname() {   return surname;     }
     public void setSurname(String surname) {      this.surname = surname;     }
  public String getInitials() { return initials; } 
  public void setInitials(String initials) { this.initials=initials;} 
}
Source: Client.java

Again, running the ClientTestRunner, we get:
ClientTestRunner Output
So far so good.  Normally, at this point we would "clean up" the code, but there is not a lot of improvement that could be made here.

So on to the final test, the handling of blank values for either the firstname or surname.  We create the test first:

import org.junit.Test; 
import static org.junit.Assert.assertEquals; 

public class ClientTest { 
 @Test 
 public void testInitialsStandard() {
System.out.println("ClientTest: Inside testIntialsStandard()"); 
  Client client = new Client("David", "Molloy");
    assertEquals("DM",client.getInitials()); 
 } 
 
 @Test
 public void testGetInitialsWhereNull() {
        System.out.println("ClientTest: Inside testGetInitialsWhereNull()");
        
   Client client = new Client(null, null);
   assertEquals("",client.getInitials());  

   client= new Client("David", null);
   assertEquals("D",client.getInitials()); 

   client= new Client(null, "Molloy");
   assertEquals("M",client.getInitials());   
 }

 @Test
 public void testGetInitialsWhereBlank() {
  System.out.println("ClientTest: Inside testGetInitialsWhereBlank()");
        
  Client client = new Client("", "");
  assertEquals("",client.getInitials()); 

  client = new Client("David", "");
  assertEquals("D",client.getInitials()); 

  client = new Client("", "Molloy");
  assertEquals("M",client.getInitials());    
 }
} 

Running ClientTestRunner we can see that this test FAILS as expected:
ClientTestRunner Output

Let us put in our final bit of production code to satisfy this test:

public class Client{
     private String firstname;
     private String surname;
     private String initials;

     public Client(String firstname, String surname) {
         this.firstname = firstname;
         this.surname = surname;
         initials = "";
     if ((firstname!=null)&&(firstname.length()>0)) initials = initials + firstname.substring(0,1);
         if ((surname!=null)&&(surname.length()>0)) initials = initials + surname.substring(0,1);
     } 
     public String getFirstname() {   return firstname;   }
     public void setFirstname(String firstname) {   this.firstname = firstname;     }
     public String getSurname() {   return surname;     }
     public void setSurname(String surname) {      this.surname = surname;     }
  public String getInitials() { return initials; } 
  public void setInitials(String initials) { this.initials=initials;} 
}
Source: Client.java

And run all of tests again, to make sure that they are satisfied:
ClientTestRunner output

Finally, we should be cleaning up code (not a lot we can do here, short of using the ternary operator for marginal improvement) and adding comments.  
However, what is important is that we now have written our first test-driven development code.  We wrote the test code to rigorously test the functionality requirements and used these tests to help write and test our production code.  

Is this process not considerably slower?  Test-driven development is almost certainly slower in any single developer systems.  However, it does provide more robust and stable code.  It seems somewhat slower than it is in this example, but as the developer gets familiar with writing tests, it becomes a much quicker process.  Initial development tends to be slower, but the benefits are returned at later stages, particularly in multi-developer systems.



Comments