A Simple Test Framework (STF) in Java


The idea for this blog post arose when I decided to implement a simple and basic version of the Untyped Lambda Calculus in Java. As part of the development process, I realised that the simple test code that I was writing was fast ballooning in size. Instead of investing in a relatively heavy framework like the ubiquitous JUnit, I decided that it would be a nice educational experience to convert the testing code into a small test framework of sorts. I call that the STF (Simple Test Framework).

STF is a tiny piece of code that will run without any extra dependencies aside from the standard JDK. I had considered implementing it in a way that would work with any JDK from version 5 and upwards, but in view of the worldwide adoption of Java 8 as also the fact that it is not intended to be a production ready library, I thought it best to implement it using features that would require JDK 7 (or above). That also means that it is implemented as a plain project without using Maven, Gradle, or any other build tool. It can be run directly by running the STFRunner class with the test class as argument (refer to the demo section). I had contemplated putting this code up on GitHub or some such site, but since it is a basic framework that fits well within a blog post, I decided to post it here in its entirety. Full-fledged projects in the future will be posted on some online repository to enable easy access and experimentation, and relevant discussion will be done here in the form of blog posts.

Basic Design

The basic design goals of this small project were:

  • Provide support for defining tests using the @Test annotation.
  • Support @Before and @After annotations for setup and clean-up code.
  • Only these annotations supported (no elements) – @Before, @After, @Test
  • No support for test suites (though it’s easy to extend the framework to support this).
  • Log the output using the built-in Java logging framework.
  • Execute a test class using: java -cp com.z0ltan.stf.STFRunner . A custom class loader loads the test class (compiling the source file first if necessary) and executes the test cases defined therein.
  • No support for line number or line of code where the error occurred i.e., no associated source code information is presented. Only the test method which failed will be logged.
  • Support only for asserts — no support for matching like in JUnit.

The basic layout of STF is amply demonstrated by the following UML diagram:

UML diagram for STF

The STFRunner class in the entry-point to the STF framework. Its responsibility is to use a custom class loader, STFClassLoader to load the test class, create an instance of the STFCore class and pass it the test class.

The STFCore class then creates an instance of the test class, executes any code in the method marked with @Before (only the first method found marked with @Before is executed), run all the test cases in a non-deterministic order logging all the relevant cases which have PASSED or FAILED, and then finally perform any cleanup code as present in the method marked with @After (again, only the first method found marked with this annotation is executed).

The most important class in STF (from a testing perspective) is STFAsserts. This class provides all the assertion facility for STF. There are basic asserts for equality, inequality, conditions, nulls, and for non-nulls not only for basic types, but also for composite types as well as arrays.

Implementation

The entire implementation is copied here for reference. Explanations are provided for the core classes and eschewed for the supporting classes/interfaces.

Before

package com.z0ltan.stf;

import java.lang.annotation.Documented;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;

@Documented
@Retention(RetentionPolicy.RUNTIME)
public @interface Before {
}

After

package com.z0ltan.stf;

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Documented;

@Documented
@Retention(RetentionPolicy.RUNTIME)
public @interface After {
}

Test

package com.z0ltan.stf;

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Documented;

@Documented
@Retention(RetentionPolicy.RUNTIME)
public @interface Test {
}

STFRunner

package com.z0ltan.stf;

public class STFRunner {
        public static void main(String[] args) {
                if (args == null || args.length != 1)
                        throw new RuntimeException("Specify the test file");

                STFCore core = new STFCore(args[0]);
                core.run();
        }
}

STFCore

package com.z0ltan.stf;

import java.util.logging.Logger;

import java.lang.reflect.Method;
import java.lang.reflect.InvocationTargetException;
import java.lang.annotation.Annotation;

public class STFCore {
        private static Logger logger = Logger.getLogger(STFCore.class.getName());
        private String file;
        private STFClassLoader loader;

        public STFCore(String file) {
                this.file = file;
                this.loader = new STFClassLoader();
        }

        public void run() {
                try {
                        Class<?> clazz = loader.findClass(this.file);
                        Object testObj = clazz.getConstructor().newInstance();

                        Method[] methods = clazz.getDeclaredMethods();

                        // Setup code (if present)
                        runBefore(testObj, methods);

                        // Run the test
                        runTests(testObj, methods);

                        // Cleanup code (if present)
                        runAfter(testObj, methods);
                } catch (Throwable ex) {
                        logger.severe("Error encountered. Message = " +
                                ex.getLocalizedMessage());
                        throw new RuntimeException(ex);
                }
        }

        private void runBefore(Object o, Method[] methods) throws Throwable {
                logger.info("Running Setup code");

                for (Method m : methods) {
                        if (containsAnnotation(m, Before.class)) {
                                execute(o, m);
                                break;
                        }
                }                                                        

                logger.info("Finished running Setup code");
        }

        private void runTests(Object o, Method[] methods) throws Throwable {
                for (Method m : methods) {
                        if (containsAnnotation(m, Test.class)) {
                                try {
                                        logger.info("Running Test Case: " + m.getName());
                                        execute(o, m);
                                        logger.info("Test Case: " + m.getName() + " - [PASSED]");
                                } catch (AssertionError err) {
                                        logger.info("Test Case: " + m.getName() +
                                                "- [FAILED]. Reason: " +
                                                err.getLocalizedMessage());
                                        continue;
                                }
                        }
                }
        }

        private void runAfter(Object o, Method[] methods) throws Throwable {
                logger.info("Running Cleanup code");

                for (Method m : methods) {
                        if (containsAnnotation(m, After.class)) {
                                execute(o, m);
                                break;
                        }
                }

                logger.info("Finished running Cleanup code");
        }

        private void execute(Object o, Method m) throws Throwable {
                try {
                        m.invoke(o, (Object[]) null);
                } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException ex) {
                        if ((ex instanceof InvocationTargetException) &&
                                (ex.getCause().getClass() == java.lang.AssertionError.class)) {
                                throw ex.getCause();
                        }
                        throw new RuntimeException("Error while invoking method: " + m.getName() +
                                ". Reason = " + ex.getLocalizedMessage());
                }
        }

        private boolean containsAnnotation(Method m, Class<?> clazz) {
                Annotation[] annotations = m.getDeclaredAnnotations();
                for (Annotation a : annotations) {
                        if (a.annotationType().equals(clazz))
                                return true;
                }

                return false;
        }
}

Explanatory notes: The STFCore class relies heavily on the introspection capabilities of Java. While not as powerful as that of Common Lisp, it is still far more powerful than similar facilities in most mainstream languages such as C++ and Python. The code is pretty much straightforward – simply use Reflection to create an instance of the test class (which has been loaded by STFClassLoader), retrieve the annotations of interest at each stage (Before, Test, and After), and execute all the test cases.

STFAsserts

package com.z0ltan.stf;

import java.util.List;
import java.text.MessageFormat;

public abstract class STFAsserts {
        private static final double EPS = 1e-9;

        private static final MessageFormat equalsFormatter =
                new MessageFormat("{0} is not equal to {1}");

        private static final MessageFormat nullFormatter =
                new MessageFormat("{0} is not null");

        private static final MessageFormat notNullFormatter =
                new MessageFormat("{0} is null");

        private static final MessageFormat trueFormatter =
                new MessageFormat("{0} is false");

        private static final MessageFormat falseFormatter =
                new MessageFormat("{0} is true");

        private static final MessageFormat sameFormatter =
                new MessageFormat("{0} is not the same object as {1}");

        private static final MessageFormat notSameFormatter =
                new MessageFormat("{0} is the same as {1}");

        private STFAsserts() {}

        // Equals
        public static void assertEquals(int x, int y)  {
                if (x != y) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertEquals(long x, long y)  {
                if (x != y) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertEquals(char x, char y)  {
                if (x != y) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertEquals(float x, float y)  {
                if (!(Math.abs(x-y) <= EPS)) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertEquals(double x, double y)  {
                if (!(Math.abs(x-y) <= EPS)) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertEquals(Object x, Object y)  {
                if (!x.equals(y)) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        // Array equals
        public static void assertArrayEquals(char[] x, char[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (x[i] != y[i]) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertArrayEquals(byte[] x, byte[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (x[i] != y[i]) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertArrayEquals(short[] x, short[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (x[i] != y[i]) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertArrayEquals(int[] x, int[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (x[i] != y[i]) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertArrayEquals(long[] x, long[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (x[i] != y[i]) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertArrayEquals(float[] x, float[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (!(Math.abs(x[i]-y[i]) < EPS)) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertArrayEquals(double[] x, double[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (!(Math.abs(x[i]-y[i]) < EPS)) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertArrayEquals(Object[] x, Object[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (!x[i].equals(y[i])) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        // Same and not same
        public static void assertSame(Object x, Object y)  {
                if (x != y) {
                        final String message =
                                sameFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertNotSame(Object x, Object y)  {
                if (x == y) {
                        final String message =
                                notSameFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }                                                

        // True and False
        public static void assertTrue(boolean condition)  {
                if (!condition) {
                        final String message =
                                trueFormatter.format(new Object[] { condition });
                        throw new AssertionError(message);
                }
        }

        public static void assertFalse(boolean condition)  {
                if (condition) {
                        final String message =
                                falseFormatter.format(new Object[] { condition });
                        throw new AssertionError(message);
                }
        }

        // Nulls and Non-nulls
        public static void assertNull(Object x)  {
                if (x != null) {
                        final String message =
                                nullFormatter.format(new Object[] { x });
                        throw new AssertionError(message);
                }
        }

        public static void assertNotNull(Object x)  {
                if (x == null) {
                        final String message =
                                notNullFormatter.format(new Object[] { “ null” });
                        throw new AssertionError(message);
                }
        }
}

STFClassLoader

package com.z0ltan.stf;

import javax.tools.ToolProvider;
import javax.tools.JavaCompiler;
import javax.tools.JavaFileObject;
import javax.tools.StandardJavaFileManager;
import javax.tools.Diagnostic;
import javax.tools.DiagnosticCollector;

import java.util.Locale;
import java.util.Arrays;
import java.util.logging.Logger;
import java.util.regex.Pattern;
import java.util.regex.Matcher;

import java.io.File;
import java.io.FileInputStream;
import java.io.BufferedInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;

import java.nio.charset.Charset;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.nio.file.attribute.FileTime;

import java.text.MessageFormat;
import java.lang.reflect.Method;

public class STFClassLoader extends ClassLoader {
        private static final Logger logger = Logger.getLogger(STFClassLoader.class.getName());
        private static final Pattern p = Pattern.compile("([a-zA-Z0-9\\/]+).?([a-zA-Z0-9]+)?");

        public STFClassLoader() {
                super(STFClassLoader.class.getClassLoader());
        }

        @Override
        public Class<?> findClass(String name) throws ClassNotFoundException {
                if (name == null)
                        throw new RuntimeException("no class name provided");

                logger.info("Loading class: " + name);

                byte[] classBytes = null;
                String className = null;
                Matcher m = p.matcher(name);

                if (m.matches()) {
                        final String baseName = m.group(1);
                        final String extension = m.group(2);
                        final String sourceFileName = baseName + ".java";
                        final String classFileName = baseName + ".class";

                        boolean classFilePresent = Files.exists(Paths.get(classFileName));
                        boolean sourceFilePresent = Files.exists(Paths.get(sourceFileName));
                        boolean forceCompile = classFilePresent? isClassFileOlderThanSourceFile(classFileName, sourceFileName) : false;

                        if (!classFilePresent && !sourceFilePresent) {
                                throw new RuntimeException("Either the source file or the class file must be available!");
                        }

                        if (extension == null || extension.equalsIgnoreCase("class") || extension.equalsIgnoreCase("java")) {
                                if (!classFilePresent || forceCompile)
                                     compileSourceFile(sourceFileName);
                        } else {
                                throw new RuntimeException("invalid class file specified: " + name);
                        }

                        classBytes = findClassBytes(classFileName);

                        char separatorChar = '/';
                        if (baseName.indexOf('\\') != -1)
                                separatorChar = '\\';

                        String definedName = baseName.replace(separatorChar, '.');
                        Class<?> clazz = defineClass(definedName, classBytes, 0, classBytes.length);

                        logger.info("Finished loading class: " + definedName);

                        return clazz;
                } else {
                        throw new ClassNotFoundException();
                }
        }

        private boolean isClassFileOlderThanSourceFile(String classFile, String sourceFile) {
                try {
                        FileTime classTime = Files.getLastModifiedTime(Paths.get(classFile));
                        FileTime sourceTime = Files.getLastModifiedTime(Paths.get(sourceFile));

                        if (classTime.compareTo(sourceTime) < 0)
                                return true;
                } catch (IOException ex) {
                        logger.warning("Error while determining source and class file modified timestamps");
                        return false;
                }

                return false;
        }

        private byte[] findClassBytes(String name) {
                try (BufferedInputStream bin = new BufferedInputStream(new FileInputStream(name));
                     ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
                        byte[] buffer = new byte[4 * 1024];
                        int bytesRead = -1;

                        while ((bytesRead = bin.read(buffer)) != -1) {
                                baos.write(buffer, 0, bytesRead);
                        }

                        return baos.toByteArray();
                } catch (IOException ex) {
                        throw new RuntimeException("Error while loading class file bytes: " +
                                ex.getLocalizedMessage());
                }
        }

        private void compileSourceFile(String fileName) {
                logger.info("Compiling source file: " + fileName);

                File[] files = new File[] { new File(fileName) };

                JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
                DiagnosticCollector<JavaFileObject> collector = new DiagnosticCollector<>();
                StandardJavaFileManager manager =
                        compiler.getStandardFileManager(collector, Locale.getDefault(), Charset.forName("UTF-8"));
                Iterable<? extends JavaFileObject> units = manager.getJavaFileObjectsFromFiles(Arrays.asList(files));

                compiler.getTask(null, manager, collector, null, null, units).call();

	    boolean issuesFound = false;
                for (Diagnostic<? extends JavaFileObject> d : collector.getDiagnostics()) {
                        final String message =
                                MessageFormat.format("Error at line: {0} in source file: {1}. Reason = {2}\n",
                                d.getLineNumber(), d.getSource().toUri(),
                                d.getMessage(Locale.getDefault()));
                        logger.severe(message);
                        issuesFound = true;
                }

               if (issuesFound) {
                        throw new RuntimeException("Aborting testing.... errors found during compilation");
                }
                try {
                        manager.close();
                } catch (IOException ex) {
                        throw new RuntimeException("Compilation failed. Failed to close file manager: " +
                                        ex.getLocalizedMessage());
                }

                logger.info("Finished compiling source file: " + fileName);
        }
}

Explanatory notes: From a purely conceptual viewpoint, this is the most complex class in the whole framework. This custom class loader runs within STF and its main job is to handle any situation that might present itself when a test file is specified – whether the class file exists, is out of date or not, and whether the source file itself is present. If the class file is missing or the class file is older than the source file, the source file is automatically compiled and the byte code loaded into the JVM. The compilation is done using pure Java (more details in the previous blog) and has no dependency on system processes.

Aside from the compilation capabilities of the STFClassLoader, another compelling reason to use a custom class loader for this exercise was to ensure separation of concerns. Due to the visibility rules of class loaders in Java, the loaded class is not visible to the parent class loaders (Bootstrap, Extension, or System class loader). This helps restrict code to within the STF framework. This, in fact, is the way JEE containers as well as OSGi containers work. Java 9 introduces the concept of “modules” into Java, and while it was not designed for the exact same end purposes, there is a whole lot of overlap between the two. It should be interesting to see how Java 9 modules affect the Java ecosystem in the years to come. I have my own reservations on that feature, but overall I think it is a step in the right direction (even thought it doesn’t completely address OSGi hell!).

Demos

First off, let’s create a JAR file out of the str project so that we can run it easily from anywhere that we choose to:

Timmy Jose@WIN-3OCJRNT7NO4 MINGW64 ~/Rabota/Blogs/Java_FP (master)
$ cat manifest.txt
Main-Class: com.z0ltan.stf.STFRunner                                 

Timmy Jose@WIN-3OCJRNT7NO4 MINGW64 ~/Rabota/Blogs/Java_FP (master)
$ jar cmf stf.jar manifest.txt com/z0ltan/stf/*.class

Now stf.jar should have been created in the same directory. Let’s create some test files and test them out!

The sample test class (posiive cases) that will be used for this demo is listed out as follows:

package com.z0ltan.testing;

import com.z0ltan.stf.Before;
import com.z0ltan.stf.Test;
import com.z0ltan.stf.After;
import static com.z0ltan.stf.STFAsserts.*;

import java.util.List;
import java.util.Arrays;

public class PositiveTests {
        private char c1, c2, c3;
        private int i1, i2, i3;
        private long l1, l2, l3;
        private double d1, d2, d3;
        private String s1, s2, s3;
        private List<Integer> li1, li2, li3;
        private int[] ia1, ia2, ia3;
        private Object o1, o2, o3;

        @Before
        public void setup() {
                c1 = 't'; c2 = 'u'; c2 = 't';
                i1 = 100; i2 = 200; i3 = 100;
                l1 = 12345L; l2 = 54321L; l3 = 12345L;
                d1 = 12.34561; d2 = 12.10289; d3 = 12.34561;

                s1 = "Hello"; s2 = "World"; s3 = "Hello";

                li1 = Arrays.asList(1,2,3,4,5);
                li2 = Arrays.asList(1,2,3,4,5,6,7);
                li3 = li1;

                ia1 = new int[] { 1, 2, 3, 4, 5};
                ia2 = new int[] { 1, 2, 3};
                ia3 = new int[] { 1, 2, 3, 4, 5};

                o1 = null;
                o2 = new Object();
                o3 = o1;
        }

        // primitive types
        @Test
        public void testCharEqualSuccess() {
                assertEquals(c1, c2);
        }

        @Test
        public void testIntEqualSuccess() {
                assertEquals(i1, i3);
        }

        @Test
        public void testLongEqualSuccess() {
                assertEquals(l1, l3);
        }

        @Test
        public void testDoubleEqualSuccess() {
                assertEquals(d1, d3);
        }

        // object types
        @Test
        public void testStringEqualSuccess() {
                assertEquals(s1, s3);
        }

        // Sequence types
        @Test
        public void testListEqualSuccess() {
                assertEquals(li1, li3);
        }

        @Test
        public void testListSameSuccess() {
                assertSame(li1, li3);
        }

        // Null and Non-Null checks
        @Test
        public void testNullSuccess() {
                assertNull(o1);
        }

        @Test
        public void testNonNullSuccess() {
                assertNotNull(o2);
        }

        // True and False checks
        @Test
        public void testTrueSucess() {
                assertTrue(1==1);
        }

        @Test
        public void testFalseSuccess() {
                assertFalse(1==2);
        }

        @After
        public void cleanup() {
                li1 = li2 = li3 = null;
                o1 = o2 = o3 = null;
                ia1 = ia2 = ia3 = null;
        }
}

This can be invoked in any of the following ways:

java -jar stf.jar com/z0ltan/testing/PositiveTests.class

or

java -jar stf.jar com/z0ltan/testing/PositiveTests.java

or even

java -jar stf.jar com/z0ltan/testing/PositiveTests

In all of these cases, the STFClassLoader will ensure that if the class file is not present (or if the class file is out of date), the source file will be compiled to get the latest byte code for this test class. The output for this sample test class is as shown below:

Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFClassLoader findClass
INFO: Loading class: com/z0ltan/testing/PositiveTests
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFClassLoader compileSourceFile
INFO: Compiling source file: com/z0ltan/testing/PositiveTests.java
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFClassLoader compileSourceFile
INFO: Finished compiling source file: com/z0ltan/testing/PositiveTests.java
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFClassLoader findClass
INFO: Finished loading class: com.z0ltan.testing.PositiveTests
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runBefore
INFO: Running Setup code
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runBefore
INFO: Finished running Setup code
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testCharEqualSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testCharEqualSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testIntEqualSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testIntEqualSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testStringEqualSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testStringEqualSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testLongEqualSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testLongEqualSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testListSameSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testListSameSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testListEqualSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testListEqualSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testDoubleEqualSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testDoubleEqualSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testFalseSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testFalseSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testNullSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testNullSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testTrueSucess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testTrueSucess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testNonNullSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testNonNullSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runAfter
INFO: Running Cleanup code
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runAfter
INFO: Finished running Cleanup code

As can be seen from the logged output, the test file is compiled, and then the byte code is loaded using the custom class loader first since this is the first run. On subsequent runs, if the source file is newer than the class file, the source will be automatically recompiled and then loaded as well. This is very useful because it allows for faster modifications and checks without the user having to recompile everything manually after every change. This helps maintain mental flow.

And just to check and ensure that negative test cases are caught as expected, let’s simply create a test for some negative cases (pretty much the same cases as those in the positive tests, but with the conditions reversed):

package com.z0ltan.testing;

import com.z0ltan.stf.Before;
import com.z0ltan.stf.Test;
import com.z0ltan.stf.After;
import static com.z0ltan.stf.STFAsserts.*;

import java.util.List;
import java.util.Arrays;

public class NegativeTests {
        private char c1, c2, c3;
        private int i1, i2, i3;
        private long l1, l2, l3;
        private double d1, d2, d3;
        private String s1, s2, s3;
        private List<Integer> li1, li2, li3;
        private int[] ia1, ia2, ia3;
        private Object o1, o2, o3;

        @Before
        public void setup() {
                c1 = 't'; c2 = 'u'; c2 = 't';
                i1 = 100; i2 = 200; i3 = 100;
                l1 = 12345L; l2 = 54321L; l3 = 12345L;
                d1 = 12.34561; d2 = 12.10289; d3 = 12.34561;

                s1 = "Hello"; s2 = "World"; s3 = "Hello";

                li1 = Arrays.asList(1,2,3,4,5);
                li2 = Arrays.asList(1,2,3,4,5,6,7);
                li3 = li1;

                ia1 = new int[] { 1, 2, 3, 4, 5};
                ia2 = new int[] { 1, 2, 3};
                ia3 = new int[] { 1, 2, 3, 4, 5};

                o1 = null;
                o2 = new Object();
                o3 = o1;
        }

        // primitive types
        @Test
        public void testCharEqualFail() {
                assertEquals(c1, c3);
        }

        @Test
        public void testIntEqualFail() {
                assertEquals(i1, i2);
        }

        @Test
        public void testLongEqualFail() {
                assertEquals(l1, l2);
        }

        @Test
        public void testDoubleEqualFail() {
                assertEquals(d1, d2);
        }

        // object types
        @Test
        public void testStringEqualFail() {
                assertEquals(s1, s2);
        }

        // Sequence types
        @Test
        public void testListEqualFail() {
                assertEquals(li1, li2);
        }

        @Test
        public void testListSameFail() {
                assertSame(li1, li2);
        }

        // Null and Non-Null checks
        @Test
        public void testNullFail() {
                assertNull(o2);
        }

        @Test
        public void testNonNullFail() {
                assertNotNull(o1);
        }

        // True and False checks
        @Test
        public void testTrueFail() {
                assertTrue(1==100);
        }

        @Test
        public void testFalseFail() {
                assertFalse(1==1);
        }

        @After
        public void cleanup() {
                li1 = li2 = li3 = null;
                o1 = o2 = o3 = null;
                ia1 = ia2 = ia3 = null;
        }
}

Let’s give it a go and see that the output is as expected:

Aug 25, 2016 6:02:29 PM com.z0ltan.stf.STFClassLoader findClass
INFO: Loading class: com/z0ltan/testing/NegativeTests.java
Aug 25, 2016 6:02:29 PM com.z0ltan.stf.STFClassLoader compileSourceFile
INFO: Compiling source file: com/z0ltan/testing/NegativeTests.java
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFClassLoader compileSourceFile
INFO: Finished compiling source file: com/z0ltan/testing/NegativeTests.java
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFClassLoader findClass
INFO: Finished loading class: com.z0ltan.testing.NegativeTests
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runBefore
INFO: Running Setup code
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runBefore
INFO: Finished running Setup code
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testDoubleEqualFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testDoubleEqualFail- [FAILED]. Reason: 12.346 is not equal to 12.103
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testStringEqualFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testStringEqualFail- [FAILED]. Reason: Hello is not equal to World
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testLongEqualFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testLongEqualFail- [FAILED]. Reason: 12,345 is not equal to 54,321
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testListEqualFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testListEqualFail- [FAILED]. Reason: [1, 2, 3, 4, 5] is not equal to [1, 2, 3, 4, 5, 6, 7]
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testCharEqualFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testCharEqualFail- [FAILED]. Reason: t is not equal to
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testIntEqualFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testIntEqualFail- [FAILED]. Reason: 100 is not equal to 200
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testNullFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testNullFail- [FAILED]. Reason: java.lang.Object@14faf53 is not null
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testFalseFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testFalseFail- [FAILED]. Reason: true is true
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testListSameFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testListSameFail- [FAILED]. Reason: [1, 2, 3, 4, 5] is not the same object as [1, 2, 3, 4, 5, 6, 7]
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testNonNullFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testNonNullFail- [FAILED]. Reason: null is null
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testTrueFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testTrueFail- [FAILED]. Reason: false is false
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runAfter
INFO: Running Cleanup code
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runAfter
INFO: Finished running Cleanup code

Immaculate!

Learnings and Conclusion

The main aim of writing this framework was to see how a small, lightweight, and minimalist test framework might be written that was also sufficiently powerful and generic to be usable in a wide variety of situations. In that respect, I feel that this project has been successful.

In terms of learning, the following observations might be made:

  • A simple test framework can be readily implemented in only a few classes and a few hundred lines of code.
  • Using Reflection in Java is always a very tricky affair. It feels rather stilted, especially after using dynamic languages for much more powerful introspection and runtime modification of objects (Common Lisp, for instance). For instance, as can be seen in STFCore, the behaviour of InvocationTargetException in Java is to gobble up any exception thrown by the reflective call – this applies to both Checked and Unchecked exceptions. This is the reason why the explicit check for the cause of the exception needs to be made. Very clunky indeed!
  • There is a lot of boilerplate code in the STFAsserts file for providing overloaded versions of various methods. Even if some of the common logic were to be abstracted away into some helper methods, that would not really help with the overall code bloat. In Common Lisp, I could have simply written a macro to generate the code and saved dozens of methods of code. Even C++’s metaprogramming capabilities would have been better than Java’s facilities.
  • Implementing a test framework is simple enough. However, making it completely safe (think concurrency) and efficient, however, is an entirely different ball game. This version is safe enough because each test class is run on a different JVM process, but it’s far from being anywhere close to efficient or extensible.
  • Understanding Class Loaders is a crucial part of any well-rounded developer’s repertoire. There is no better way to truly grok it than to implement one yourself!
  • Finally, JShell really help! It seriously saved me a ton of mundane typing to test out various snippets of code and corner-cases.

Potentially trivially implemented improvements:

  • Support annotations with elements: @Test(expected=MyException.class) for instance.
  • Linking with source code to provide granular information about where in the code the specific test failed.
  • Show a summary of how many test cases have passed and how many have failed as well as the total number of test cases executed.
  • Support for full-fledged test suites.
  • Linking with IDEs such as Eclipse, NetBeans, and IntelliJ.

In some future post, I will discuss the implications of creating custom class loaders in Java, and class loaders themselves. This is a very important topic, and creating my own class loader for the STF framework has given me a fresh perspective into this whole esoteric domain of Java (more precisely, the JVM).

For now the next few posts will be related to the aforementioned functional implementations of the Untyped Lambda Calculus in Java, Common Lisp, and in C++. The Java version of this functional library will make extensive use of the STF framework for its testing. Note that the aim will be to not only create a functional library in each of these languages, but also to implement the libraries themselves as functionally as possible. In the future, I might write versions of this library in Python and Haskell as well. Haskell especially is indispensable when studying Functional Programming from a pragmatic viewpoint.

Advertisements
A Simple Test Framework (STF) in Java

Speak your mind!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s