How to build OpenJDK 9 on macOS Sierra

Continuing with my research into Programming Languages (and Compilers in particular), I started looking through the (considerably abstruse) documentation on the OpenJDK site. My patience paid off in the end and I ended up getting a lot of useful information on what I was looking for in particular – the javac source code and documentation. One of my earmarked projects for this year is to modify javac to allow some for extra syntax, and since this involves adding additional semantics, I was keen to download and build OpenJDK for myself.

The steps are surprisingly very well documented and easy to follow. To give a personal perspective on the build process (including a gotcha regarding JDK9), here is the list of steps to download OpenJDK 9 and build the whole system from scratch (for experimentation/contributing to OpenJDK, what have you). Note that the latter steps are specifically for macOS. However, most of the steps are more or less the same across platforms):

  1. First, install Mercurial if you already don’t have it. This was a bit irksome for me at first (being primarily a Git user). If you have Homebrew installed, then it is as trivial as:
    $ brew install hg

    Then, ensure that you have the following settings in your ~/.hgrc file. In case you don’t have a .hgrc file, create one like so:
    $ touch ~/.hgrc

    Add the following lines to the ~/.hgrc file:

    [settings]
    fetch = 
    mq = 
    
  2. Now, clone the OpenJDK 9 repository (for some reason, hg tclone did not work for me (as specified in the documentation), but the following did steps worked just fine:
     $ hg clone http://hg.openjdk.java.net/jdk9/dev 9dev
     $ cd 9dev
     $ sh ./get_source.sh
     

    The last step will retrieve all the necessary code and build environment into your local repository, so this does take some time.

  3. Now, one caveat here – to build OpenJDK, you need an existing JDK installation (the documentation mentions that any standard variant will do). This is called the “boot JDK”. However, I found that the build does not go through with JDK 9 (from the error messages, it looks like Reflection due to the new modules system is the main culprit). In that case, you can use an older version (say, JDK 8) to perform the build instead.

    In the case of macOS, you can use the following command to find the available JDKs on your machine:

     Macushla:RacketMacros z0ltan$ /usr/libexec/java_home -V
     Matching Java Virtual Machines (2):
      9, x86_64:	"Java SE 9-ea"	/Library/Java/JavaVirtualMachines/jdk-
      9.jdk/Contents/Home
      1.8.0_112, x86_64:	"Java SE 8"	
      /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home
    
      /Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home
     

    The last line shows the current default JDK. Before proceeding with the build, you can either change the default version, or you can simply pass in a different “boot JDK” to the configuration step (preferable).

    Now, let’s generate the configuration file:

    bash configure --with-boot-jdk /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home/ --disable-warnings-as-errors
    

    Note that I pass in two flags to bash configure : --with-boot-jdk specifying the JDK that I want to use to build OpenJDK, and --disable-warnings-as-errors (this is required on macOS).

    If this command runs successfully, the build is almost certainly going to succeed.

  4. Finally, trigger the build (this should take a bit of time again):
    $ make images
    

    This will dump the generated OpenJDK images into the build directory.

  5. You can test that the generated image is valid:
    Macushla:9dev z0ltan$ ./build/macosx-x86_64-normal-server-release/jdk/bin/java -version
    openjdk version "9-internal"
    OpenJDK Runtime Environment (build 9-internal+0-adhoc.z0ltan.9dev)
    OpenJDK 64-Bit Server VM (build 9-internal+0-adhoc.z0ltan.9dev, mixed mode)
    

    And there you go. All done in under half an hour!

Note that there is a plethora of build options and flags (including cross-compilation capabilities), and you should definitely check out the two HTML pages in the following location of your repository – ./common/doc/building.html, and ./common/doc/testing.html.

Now the real work begins! 😀

How to build OpenJDK 9 on macOS Sierra

Hoyte’s Line Number closure in a few choice languages

Douglas Hoyte, in his excellent (if a bit fanatical) book, “Let over Lambda” gives a simple and pithy example to demonstrate the concept of closures – little anonymous functions that can capture variables in the environment in which the closure was created.

The example is very simple – create a mini-logging facility by capturing a variable representing the current line number (initially set to 0), incrementing it for every invocation of the closure, and printing it out.

An implementation in Common Lisp might look so –

(defun get-line-logger ()
  (let ((line-num 0))
    #'(lambda (id)
        (incf line-num)
        (format t "[~a] Line number: ~d~%" id line-num))))


(defun logger-demo ()
  (let ((logger-1 (get-line-logger))
        (logger-2 (get-line-logger)))
    (flet ((f (id logger)
             (dotimes (i 5)
               (funcall logger id))
             (terpri)))
      (f "logger-1" logger-1)
      (f "logger-2" logger-2))))

Sample run:

CL-USER> (logger-demo)
[logger-1] Line number: 1
[logger-1] Line number: 2
[logger-1] Line number: 3
[logger-1] Line number: 4
[logger-1] Line number: 5

[logger-2] Line number: 1
[logger-2] Line number: 2
[logger-2] Line number: 3
[logger-2] Line number: 4
[logger-2] Line number: 5

NIL

As expected, we can not only capture local variables in the lexical environment during the time of the closure’s creation, but also modify them independently of any other instances of the closure. This is what makes is a proper closure. Also note that the capture is automatically done (whether we actually use the variables or not is irrelevant).

The Racket version is, unsurprisingly, almost identical not only in syntax, but also semantics:

#lang racket

(define (get-line-logger)
  (let ([line-no 0])
    (lambda (id)
      (set! line-no (+ 1 line-no))
      (fprintf (current-output-port)"[~a] Line number: ~a~%" id line-no))))

(define (logger-demo)
  (let ([logger-1 (get-line-logger)]
        [logger-2 (get-line-logger)])
    (letrec ([f (lambda (id logger)
                  (do
                      ((i 0 (+ i 1)))
                      ((= i 5))
                    (logger id))
                  (newline))])
      (f "logger1" logger-1)
      (f "logger2" logger-2))))

And the behaviour is exactly the same:

hoyte-closure.rkt> (logger-demo)
[logger1] Line number: 1
[logger1] Line number: 2
[logger1] Line number: 3
[logger1] Line number: 4
[logger1] Line number: 5

[logger2] Line number: 1
[logger2] Line number: 2
[logger2] Line number: 3
[logger2] Line number: 4
[logger2] Line number: 5

Racket is, in some ways, more elegant than even Common Lisp. I especially love the part where lambdas don’t need any funcallS or applyS to make them run (unlike in Common Lisp). Still, pretty much a branch off the same family tree.

Moving on, let’s try the same in Java, shall we?

import java.util.function.Function;

public class HoyteClosure {
    private static Function<String, Void> getLineLogger() {
        int lineNum = 0;

        return (String id) -> { lineNum++; System.out.printf("[%s] Line number: %d\n",
                                                                 id, lineNum); return null; };
    }

    private static void invokeLogger(String id, Function<String, Void> logger) {
        for (int i = 0; i < 5; i++) {
            logger.apply(id);
        }
        System.out.println();
    }

    public static void main(String[] args) {
        Function<String, Void> logger1 = getLineLogger();
        Function<String, Void> logger2 = getLineLogger();

        invokeLogger("logger1", logger1);
        invokeLogger("logger2", logger2);
    }
}

Okay, looks good. However, if we try to run it, we run into issues immediately:

Timmys-MacBook-Pro:Java8 z0ltan$ javac HoyteClosure.java 
Timmys-MacBook-Pro:Java8 z0ltan$ javac HoyteClosure.java 
HoyteClosure.java:7: error: local variables referenced from a lambda expression must be final or effectively final
        return (String id) -> { lineNum++; System.out.printf("[%s] Line number: %d\n",
                                ^
HoyteClosure.java:8: error: local variables referenced from a lambda expression must be final or effectively final
                                                                 id, lineNum); return null; };
                                                                     ^
2 errors

The problem is that Java really does not have real closures. The lambda support in Java 8 is just syntactic sugar for the good old anonymous class which can read local variables in the environment, but cannot modify them. So what can we do?

To make Java happy, we can create a new object for every logger instance i.e., use instance variables in lieu of local variables so that the modification of local variables is not an issue any more:

import java.util.function.Function;

public class HoyteClosureModified {
    static class Closure {
        private int lineNum;
        
        private  Function<String, Void> getLineLogger() {
            return (String id) -> { lineNum++; System.out.printf("[%s] Line number: %d\n",
                                                                 id, lineNum); return null; };
        }
    }

    private static void invokeLogger(String id, Function<String, Void> logger) {
        for (int i = 0; i < 5; i++) {
            logger.apply(id);
        }
        System.out.println();
    }

    public static void main(String[] args) {
        Function<String, Void> logger1 = new Closure().getLineLogger();
        Function<String, Void> logger2 = new Closure().getLineLogger();

        invokeLogger("logger1", logger1);
        invokeLogger("logger2", logger2);
    }
}

Taking it for a test spin, we get:

Timmys-MacBook-Pro:Java8 z0ltan$ javac HoyteClosureModified.java
Timmys-MacBook-Pro:Java8 z0ltan$ java -cp . HoyteClosureModified
[logger1] Line number: 1
[logger1] Line number: 2
[logger1] Line number: 3
[logger1] Line number: 4
[logger1] Line number: 5

[logger2] Line number: 1
[logger2] Line number: 2
[logger2] Line number: 3
[logger2] Line number: 4
[logger2] Line number: 5

It’s not the same though, is it? The whole point of using a closure was so that we wouldn’t have to do this explicit management of state ourselves. As such, Java doesn’t really have full-blown closures, just a poor man’
s version of it. Better luck next time, Java.

Finally, the same using Ruby. As I have said before, Ruby feels remarkably like a Lisp despite the syntactic differences.

module HoyteClosure 
    class Demo
        def self.get_line_logger
            line_no = 0

            lambda do |id|
                line_no += 1
                puts "[" + id + "] Line number: " + line_no.to_s
            end
        end

        def self.logger_demo(id, logger)
            5.times {
                logger.call(id)
            }
            puts ""
        end

        def self.main
            logger1 = get_line_logger
            logger2 = get_line_logger

            logger_demo("logger1", logger1)
            logger_demo("logger2", logger2)
        end
    end
end

And the final test run:

irb(main):009:0> load "./hoyte_closure.rb"
load "./hoyte_closure.rb"
=> true
irb(main):010:0> HoyteClosure::Demo.main
HoyteClosure::Demo.main
[logger1] Line number: 1
[logger1] Line number: 2
[logger1] Line number: 3
[logger1] Line number: 4
[logger1] Line number: 5

[logger2] Line number: 1
[logger2] Line number: 2
[logger2] Line number: 3
[logger2] Line number: 4
[logger2] Line number: 5

=> nil

Short, non-idiomatic, and sweet!

Hoyte’s Line Number closure in a few choice languages

A Simple Test Framework (STF) in Java

The idea for this blog post arose when I decided to implement a simple and basic version of the Untyped Lambda Calculus in Java. As part of the development process, I realised that the simple test code that I was writing was fast ballooning in size. Instead of investing in a relatively heavy framework like the ubiquitous JUnit, I decided that it would be a nice educational experience to convert the testing code into a small test framework of sorts. I call that the STF (Simple Test Framework).

STF is a tiny piece of code that will run without any extra dependencies aside from the standard JDK. I had considered implementing it in a way that would work with any JDK from version 5 and upwards, but in view of the worldwide adoption of Java 8 as also the fact that it is not intended to be a production ready library, I thought it best to implement it using features that would require JDK 7 (or above). That also means that it is implemented as a plain project without using Maven, Gradle, or any other build tool. It can be run directly by running the STFRunner class with the test class as argument (refer to the demo section). I had contemplated putting this code up on GitHub or some such site, but since it is a basic framework that fits well within a blog post, I decided to post it here in its entirety. Full-fledged projects in the future will be posted on some online repository to enable easy access and experimentation, and relevant discussion will be done here in the form of blog posts.

Basic Design

The basic design goals of this small project were:

  • Provide support for defining tests using the @Test annotation.
  • Support @Before and @After annotations for setup and clean-up code.
  • Only these annotations supported (no elements) – @Before, @After, @Test
  • No support for test suites (though it’s easy to extend the framework to support this).
  • Log the output using the built-in Java logging framework.
  • Execute a test class using: java -cp com.z0ltan.stf.STFRunner . A custom class loader loads the test class (compiling the source file first if necessary) and executes the test cases defined therein.
  • No support for line number or line of code where the error occurred i.e., no associated source code information is presented. Only the test method which failed will be logged.
  • Support only for asserts — no support for matching like in JUnit.

The basic layout of STF is amply demonstrated by the following UML diagram:

UML diagram for STF

The STFRunner class in the entry-point to the STF framework. Its responsibility is to use a custom class loader, STFClassLoader to load the test class, create an instance of the STFCore class and pass it the test class.

The STFCore class then creates an instance of the test class, executes any code in the method marked with @Before (only the first method found marked with @Before is executed), run all the test cases in a non-deterministic order logging all the relevant cases which have PASSED or FAILED, and then finally perform any cleanup code as present in the method marked with @After (again, only the first method found marked with this annotation is executed).

The most important class in STF (from a testing perspective) is STFAsserts. This class provides all the assertion facility for STF. There are basic asserts for equality, inequality, conditions, nulls, and for non-nulls not only for basic types, but also for composite types as well as arrays.

Implementation

The entire implementation is copied here for reference. Explanations are provided for the core classes and eschewed for the supporting classes/interfaces.

Before

package com.z0ltan.stf;

import java.lang.annotation.Documented;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;

@Documented
@Retention(RetentionPolicy.RUNTIME)
public @interface Before {
}

After

package com.z0ltan.stf;

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Documented;

@Documented
@Retention(RetentionPolicy.RUNTIME)
public @interface After {
}

Test

package com.z0ltan.stf;

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Documented;

@Documented
@Retention(RetentionPolicy.RUNTIME)
public @interface Test {
}

STFRunner

package com.z0ltan.stf;

public class STFRunner {
        public static void main(String[] args) {
                if (args == null || args.length != 1)
                        throw new RuntimeException("Specify the test file");

                STFCore core = new STFCore(args[0]);
                core.run();
        }
}

STFCore

package com.z0ltan.stf;

import java.util.logging.Logger;

import java.lang.reflect.Method;
import java.lang.reflect.InvocationTargetException;
import java.lang.annotation.Annotation;

public class STFCore {
        private static Logger logger = Logger.getLogger(STFCore.class.getName());
        private String file;
        private STFClassLoader loader;

        public STFCore(String file) {
                this.file = file;
                this.loader = new STFClassLoader();
        }

        public void run() {
                try {
                        Class<?> clazz = loader.findClass(this.file);
                        Object testObj = clazz.getConstructor().newInstance();

                        Method[] methods = clazz.getDeclaredMethods();

                        // Setup code (if present)
                        runBefore(testObj, methods);

                        // Run the test
                        runTests(testObj, methods);

                        // Cleanup code (if present)
                        runAfter(testObj, methods);
                } catch (Throwable ex) {
                        logger.severe("Error encountered. Message = " +
                                ex.getLocalizedMessage());
                        throw new RuntimeException(ex);
                }
        }

        private void runBefore(Object o, Method[] methods) throws Throwable {
                logger.info("Running Setup code");

                for (Method m : methods) {
                        if (containsAnnotation(m, Before.class)) {
                                execute(o, m);
                                break;
                        }
                }                                                        

                logger.info("Finished running Setup code");
        }

        private void runTests(Object o, Method[] methods) throws Throwable {
                for (Method m : methods) {
                        if (containsAnnotation(m, Test.class)) {
                                try {
                                        logger.info("Running Test Case: " + m.getName());
                                        execute(o, m);
                                        logger.info("Test Case: " + m.getName() + " - [PASSED]");
                                } catch (AssertionError err) {
                                        logger.info("Test Case: " + m.getName() +
                                                "- [FAILED]. Reason: " +
                                                err.getLocalizedMessage());
                                        continue;
                                }
                        }
                }
        }

        private void runAfter(Object o, Method[] methods) throws Throwable {
                logger.info("Running Cleanup code");

                for (Method m : methods) {
                        if (containsAnnotation(m, After.class)) {
                                execute(o, m);
                                break;
                        }
                }

                logger.info("Finished running Cleanup code");
        }

        private void execute(Object o, Method m) throws Throwable {
                try {
                        m.invoke(o, (Object[]) null);
                } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException ex) {
                        if ((ex instanceof InvocationTargetException) &&
                                (ex.getCause().getClass() == java.lang.AssertionError.class)) {
                                throw ex.getCause();
                        }
                        throw new RuntimeException("Error while invoking method: " + m.getName() +
                                ". Reason = " + ex.getLocalizedMessage());
                }
        }

        private boolean containsAnnotation(Method m, Class<?> clazz) {
                Annotation[] annotations = m.getDeclaredAnnotations();
                for (Annotation a : annotations) {
                        if (a.annotationType().equals(clazz))
                                return true;
                }

                return false;
        }
}

Explanatory notes: The STFCore class relies heavily on the introspection capabilities of Java. While not as powerful as that of Common Lisp, it is still far more powerful than similar facilities in most mainstream languages such as C++ and Python. The code is pretty much straightforward – simply use Reflection to create an instance of the test class (which has been loaded by STFClassLoader), retrieve the annotations of interest at each stage (Before, Test, and After), and execute all the test cases.

STFAsserts

package com.z0ltan.stf;

import java.util.List;
import java.text.MessageFormat;

public abstract class STFAsserts {
        private static final double EPS = 1e-9;

        private static final MessageFormat equalsFormatter =
                new MessageFormat("{0} is not equal to {1}");

        private static final MessageFormat nullFormatter =
                new MessageFormat("{0} is not null");

        private static final MessageFormat notNullFormatter =
                new MessageFormat("{0} is null");

        private static final MessageFormat trueFormatter =
                new MessageFormat("{0} is false");

        private static final MessageFormat falseFormatter =
                new MessageFormat("{0} is true");

        private static final MessageFormat sameFormatter =
                new MessageFormat("{0} is not the same object as {1}");

        private static final MessageFormat notSameFormatter =
                new MessageFormat("{0} is the same as {1}");

        private STFAsserts() {}

        // Equals
        public static void assertEquals(int x, int y)  {
                if (x != y) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertEquals(long x, long y)  {
                if (x != y) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertEquals(char x, char y)  {
                if (x != y) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertEquals(float x, float y)  {
                if (!(Math.abs(x-y) <= EPS)) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertEquals(double x, double y)  {
                if (!(Math.abs(x-y) <= EPS)) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertEquals(Object x, Object y)  {
                if (!x.equals(y)) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        // Array equals
        public static void assertArrayEquals(char[] x, char[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (x[i] != y[i]) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertArrayEquals(byte[] x, byte[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (x[i] != y[i]) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertArrayEquals(short[] x, short[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (x[i] != y[i]) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertArrayEquals(int[] x, int[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (x[i] != y[i]) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertArrayEquals(long[] x, long[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (x[i] != y[i]) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertArrayEquals(float[] x, float[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (!(Math.abs(x[i]-y[i]) < EPS)) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertArrayEquals(double[] x, double[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (!(Math.abs(x[i]-y[i]) < EPS)) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertArrayEquals(Object[] x, Object[] y)  {
                assertNotNull(x);
                assertNotNull(y);

                boolean same = true;

                for (int i = 0; i < x.length; i++) {
                        if (!x[i].equals(y[i])) {
                                same = false;
                                break;
                        }
                }

                if (!same) {
                        final String message =
                                equalsFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        // Same and not same
        public static void assertSame(Object x, Object y)  {
                if (x != y) {
                        final String message =
                                sameFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }

        public static void assertNotSame(Object x, Object y)  {
                if (x == y) {
                        final String message =
                                notSameFormatter.format(new Object[] { x, y });
                        throw new AssertionError(message);
                }
        }                                                

        // True and False
        public static void assertTrue(boolean condition)  {
                if (!condition) {
                        final String message =
                                trueFormatter.format(new Object[] { condition });
                        throw new AssertionError(message);
                }
        }

        public static void assertFalse(boolean condition)  {
                if (condition) {
                        final String message =
                                falseFormatter.format(new Object[] { condition });
                        throw new AssertionError(message);
                }
        }

        // Nulls and Non-nulls
        public static void assertNull(Object x)  {
                if (x != null) {
                        final String message =
                                nullFormatter.format(new Object[] { x });
                        throw new AssertionError(message);
                }
        }

        public static void assertNotNull(Object x)  {
                if (x == null) {
                        final String message =
                                notNullFormatter.format(new Object[] { “ null” });
                        throw new AssertionError(message);
                }
        }
}

STFClassLoader

package com.z0ltan.stf;

import javax.tools.ToolProvider;
import javax.tools.JavaCompiler;
import javax.tools.JavaFileObject;
import javax.tools.StandardJavaFileManager;
import javax.tools.Diagnostic;
import javax.tools.DiagnosticCollector;

import java.util.Locale;
import java.util.Arrays;
import java.util.logging.Logger;
import java.util.regex.Pattern;
import java.util.regex.Matcher;

import java.io.File;
import java.io.FileInputStream;
import java.io.BufferedInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;

import java.nio.charset.Charset;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.nio.file.attribute.FileTime;

import java.text.MessageFormat;
import java.lang.reflect.Method;

public class STFClassLoader extends ClassLoader {
        private static final Logger logger = Logger.getLogger(STFClassLoader.class.getName());
        private static final Pattern p = Pattern.compile("([a-zA-Z0-9\\/]+).?([a-zA-Z0-9]+)?");

        public STFClassLoader() {
                super(STFClassLoader.class.getClassLoader());
        }

        @Override
        public Class<?> findClass(String name) throws ClassNotFoundException {
                if (name == null)
                        throw new RuntimeException("no class name provided");

                logger.info("Loading class: " + name);

                byte[] classBytes = null;
                String className = null;
                Matcher m = p.matcher(name);

                if (m.matches()) {
                        final String baseName = m.group(1);
                        final String extension = m.group(2);
                        final String sourceFileName = baseName + ".java";
                        final String classFileName = baseName + ".class";

                        boolean classFilePresent = Files.exists(Paths.get(classFileName));
                        boolean sourceFilePresent = Files.exists(Paths.get(sourceFileName));
                        boolean forceCompile = classFilePresent? isClassFileOlderThanSourceFile(classFileName, sourceFileName) : false;

                        if (!classFilePresent && !sourceFilePresent) {
                                throw new RuntimeException("Either the source file or the class file must be available!");
                        }

                        if (extension == null || extension.equalsIgnoreCase("class") || extension.equalsIgnoreCase("java")) {
                                if (!classFilePresent || forceCompile)
                                     compileSourceFile(sourceFileName);
                        } else {
                                throw new RuntimeException("invalid class file specified: " + name);
                        }

                        classBytes = findClassBytes(classFileName);

                        char separatorChar = '/';
                        if (baseName.indexOf('\\') != -1)
                                separatorChar = '\\';

                        String definedName = baseName.replace(separatorChar, '.');
                        Class<?> clazz = defineClass(definedName, classBytes, 0, classBytes.length);

                        logger.info("Finished loading class: " + definedName);

                        return clazz;
                } else {
                        throw new ClassNotFoundException();
                }
        }

        private boolean isClassFileOlderThanSourceFile(String classFile, String sourceFile) {
                try {
                        FileTime classTime = Files.getLastModifiedTime(Paths.get(classFile));
                        FileTime sourceTime = Files.getLastModifiedTime(Paths.get(sourceFile));

                        if (classTime.compareTo(sourceTime) < 0)
                                return true;
                } catch (IOException ex) {
                        logger.warning("Error while determining source and class file modified timestamps");
                        return false;
                }

                return false;
        }

        private byte[] findClassBytes(String name) {
                try (BufferedInputStream bin = new BufferedInputStream(new FileInputStream(name));
                     ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
                        byte[] buffer = new byte[4 * 1024];
                        int bytesRead = -1;

                        while ((bytesRead = bin.read(buffer)) != -1) {
                                baos.write(buffer, 0, bytesRead);
                        }

                        return baos.toByteArray();
                } catch (IOException ex) {
                        throw new RuntimeException("Error while loading class file bytes: " +
                                ex.getLocalizedMessage());
                }
        }

        private void compileSourceFile(String fileName) {
                logger.info("Compiling source file: " + fileName);

                File[] files = new File[] { new File(fileName) };

                JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
                DiagnosticCollector<JavaFileObject> collector = new DiagnosticCollector<>();
                StandardJavaFileManager manager =
                        compiler.getStandardFileManager(collector, Locale.getDefault(), Charset.forName("UTF-8"));
                Iterable<? extends JavaFileObject> units = manager.getJavaFileObjectsFromFiles(Arrays.asList(files));

                compiler.getTask(null, manager, collector, null, null, units).call();

	    boolean issuesFound = false;
                for (Diagnostic<? extends JavaFileObject> d : collector.getDiagnostics()) {
                        final String message =
                                MessageFormat.format("Error at line: {0} in source file: {1}. Reason = {2}\n",
                                d.getLineNumber(), d.getSource().toUri(),
                                d.getMessage(Locale.getDefault()));
                        logger.severe(message);
                        issuesFound = true;
                }

               if (issuesFound) {
                        throw new RuntimeException("Aborting testing.... errors found during compilation");
                }
                try {
                        manager.close();
                } catch (IOException ex) {
                        throw new RuntimeException("Compilation failed. Failed to close file manager: " +
                                        ex.getLocalizedMessage());
                }

                logger.info("Finished compiling source file: " + fileName);
        }
}

Explanatory notes: From a purely conceptual viewpoint, this is the most complex class in the whole framework. This custom class loader runs within STF and its main job is to handle any situation that might present itself when a test file is specified – whether the class file exists, is out of date or not, and whether the source file itself is present. If the class file is missing or the class file is older than the source file, the source file is automatically compiled and the byte code loaded into the JVM. The compilation is done using pure Java (more details in the previous blog) and has no dependency on system processes.

Aside from the compilation capabilities of the STFClassLoader, another compelling reason to use a custom class loader for this exercise was to ensure separation of concerns. Due to the visibility rules of class loaders in Java, the loaded class is not visible to the parent class loaders (Bootstrap, Extension, or System class loader). This helps restrict code to within the STF framework. This, in fact, is the way JEE containers as well as OSGi containers work. Java 9 introduces the concept of “modules” into Java, and while it was not designed for the exact same end purposes, there is a whole lot of overlap between the two. It should be interesting to see how Java 9 modules affect the Java ecosystem in the years to come. I have my own reservations on that feature, but overall I think it is a step in the right direction (even thought it doesn’t completely address OSGi hell!).

Demos

First off, let’s create a JAR file out of the str project so that we can run it easily from anywhere that we choose to:

Timmy Jose@WIN-3OCJRNT7NO4 MINGW64 ~/Rabota/Blogs/Java_FP (master)
$ cat manifest.txt
Main-Class: com.z0ltan.stf.STFRunner                                 

Timmy Jose@WIN-3OCJRNT7NO4 MINGW64 ~/Rabota/Blogs/Java_FP (master)
$ jar cmf stf.jar manifest.txt com/z0ltan/stf/*.class

Now stf.jar should have been created in the same directory. Let’s create some test files and test them out!

The sample test class (posiive cases) that will be used for this demo is listed out as follows:

package com.z0ltan.testing;

import com.z0ltan.stf.Before;
import com.z0ltan.stf.Test;
import com.z0ltan.stf.After;
import static com.z0ltan.stf.STFAsserts.*;

import java.util.List;
import java.util.Arrays;

public class PositiveTests {
        private char c1, c2, c3;
        private int i1, i2, i3;
        private long l1, l2, l3;
        private double d1, d2, d3;
        private String s1, s2, s3;
        private List<Integer> li1, li2, li3;
        private int[] ia1, ia2, ia3;
        private Object o1, o2, o3;

        @Before
        public void setup() {
                c1 = 't'; c2 = 'u'; c2 = 't';
                i1 = 100; i2 = 200; i3 = 100;
                l1 = 12345L; l2 = 54321L; l3 = 12345L;
                d1 = 12.34561; d2 = 12.10289; d3 = 12.34561;

                s1 = "Hello"; s2 = "World"; s3 = "Hello";

                li1 = Arrays.asList(1,2,3,4,5);
                li2 = Arrays.asList(1,2,3,4,5,6,7);
                li3 = li1;

                ia1 = new int[] { 1, 2, 3, 4, 5};
                ia2 = new int[] { 1, 2, 3};
                ia3 = new int[] { 1, 2, 3, 4, 5};

                o1 = null;
                o2 = new Object();
                o3 = o1;
        }

        // primitive types
        @Test
        public void testCharEqualSuccess() {
                assertEquals(c1, c2);
        }

        @Test
        public void testIntEqualSuccess() {
                assertEquals(i1, i3);
        }

        @Test
        public void testLongEqualSuccess() {
                assertEquals(l1, l3);
        }

        @Test
        public void testDoubleEqualSuccess() {
                assertEquals(d1, d3);
        }

        // object types
        @Test
        public void testStringEqualSuccess() {
                assertEquals(s1, s3);
        }

        // Sequence types
        @Test
        public void testListEqualSuccess() {
                assertEquals(li1, li3);
        }

        @Test
        public void testListSameSuccess() {
                assertSame(li1, li3);
        }

        // Null and Non-Null checks
        @Test
        public void testNullSuccess() {
                assertNull(o1);
        }

        @Test
        public void testNonNullSuccess() {
                assertNotNull(o2);
        }

        // True and False checks
        @Test
        public void testTrueSucess() {
                assertTrue(1==1);
        }

        @Test
        public void testFalseSuccess() {
                assertFalse(1==2);
        }

        @After
        public void cleanup() {
                li1 = li2 = li3 = null;
                o1 = o2 = o3 = null;
                ia1 = ia2 = ia3 = null;
        }
}

This can be invoked in any of the following ways:

java -jar stf.jar com/z0ltan/testing/PositiveTests.class

or

java -jar stf.jar com/z0ltan/testing/PositiveTests.java

or even

java -jar stf.jar com/z0ltan/testing/PositiveTests

In all of these cases, the STFClassLoader will ensure that if the class file is not present (or if the class file is out of date), the source file will be compiled to get the latest byte code for this test class. The output for this sample test class is as shown below:

Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFClassLoader findClass
INFO: Loading class: com/z0ltan/testing/PositiveTests
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFClassLoader compileSourceFile
INFO: Compiling source file: com/z0ltan/testing/PositiveTests.java
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFClassLoader compileSourceFile
INFO: Finished compiling source file: com/z0ltan/testing/PositiveTests.java
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFClassLoader findClass
INFO: Finished loading class: com.z0ltan.testing.PositiveTests
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runBefore
INFO: Running Setup code
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runBefore
INFO: Finished running Setup code
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testCharEqualSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testCharEqualSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testIntEqualSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testIntEqualSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testStringEqualSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testStringEqualSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testLongEqualSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testLongEqualSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testListSameSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testListSameSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testListEqualSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testListEqualSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testDoubleEqualSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testDoubleEqualSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testFalseSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testFalseSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testNullSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testNullSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testTrueSucess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testTrueSucess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testNonNullSuccess
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testNonNullSuccess - [PASSED]
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runAfter
INFO: Running Cleanup code
Aug 25, 2016 5:47:12 PM com.z0ltan.stf.STFCore runAfter
INFO: Finished running Cleanup code

As can be seen from the logged output, the test file is compiled, and then the byte code is loaded using the custom class loader first since this is the first run. On subsequent runs, if the source file is newer than the class file, the source will be automatically recompiled and then loaded as well. This is very useful because it allows for faster modifications and checks without the user having to recompile everything manually after every change. This helps maintain mental flow.

And just to check and ensure that negative test cases are caught as expected, let’s simply create a test for some negative cases (pretty much the same cases as those in the positive tests, but with the conditions reversed):

package com.z0ltan.testing;

import com.z0ltan.stf.Before;
import com.z0ltan.stf.Test;
import com.z0ltan.stf.After;
import static com.z0ltan.stf.STFAsserts.*;

import java.util.List;
import java.util.Arrays;

public class NegativeTests {
        private char c1, c2, c3;
        private int i1, i2, i3;
        private long l1, l2, l3;
        private double d1, d2, d3;
        private String s1, s2, s3;
        private List<Integer> li1, li2, li3;
        private int[] ia1, ia2, ia3;
        private Object o1, o2, o3;

        @Before
        public void setup() {
                c1 = 't'; c2 = 'u'; c2 = 't';
                i1 = 100; i2 = 200; i3 = 100;
                l1 = 12345L; l2 = 54321L; l3 = 12345L;
                d1 = 12.34561; d2 = 12.10289; d3 = 12.34561;

                s1 = "Hello"; s2 = "World"; s3 = "Hello";

                li1 = Arrays.asList(1,2,3,4,5);
                li2 = Arrays.asList(1,2,3,4,5,6,7);
                li3 = li1;

                ia1 = new int[] { 1, 2, 3, 4, 5};
                ia2 = new int[] { 1, 2, 3};
                ia3 = new int[] { 1, 2, 3, 4, 5};

                o1 = null;
                o2 = new Object();
                o3 = o1;
        }

        // primitive types
        @Test
        public void testCharEqualFail() {
                assertEquals(c1, c3);
        }

        @Test
        public void testIntEqualFail() {
                assertEquals(i1, i2);
        }

        @Test
        public void testLongEqualFail() {
                assertEquals(l1, l2);
        }

        @Test
        public void testDoubleEqualFail() {
                assertEquals(d1, d2);
        }

        // object types
        @Test
        public void testStringEqualFail() {
                assertEquals(s1, s2);
        }

        // Sequence types
        @Test
        public void testListEqualFail() {
                assertEquals(li1, li2);
        }

        @Test
        public void testListSameFail() {
                assertSame(li1, li2);
        }

        // Null and Non-Null checks
        @Test
        public void testNullFail() {
                assertNull(o2);
        }

        @Test
        public void testNonNullFail() {
                assertNotNull(o1);
        }

        // True and False checks
        @Test
        public void testTrueFail() {
                assertTrue(1==100);
        }

        @Test
        public void testFalseFail() {
                assertFalse(1==1);
        }

        @After
        public void cleanup() {
                li1 = li2 = li3 = null;
                o1 = o2 = o3 = null;
                ia1 = ia2 = ia3 = null;
        }
}

Let’s give it a go and see that the output is as expected:

Aug 25, 2016 6:02:29 PM com.z0ltan.stf.STFClassLoader findClass
INFO: Loading class: com/z0ltan/testing/NegativeTests.java
Aug 25, 2016 6:02:29 PM com.z0ltan.stf.STFClassLoader compileSourceFile
INFO: Compiling source file: com/z0ltan/testing/NegativeTests.java
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFClassLoader compileSourceFile
INFO: Finished compiling source file: com/z0ltan/testing/NegativeTests.java
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFClassLoader findClass
INFO: Finished loading class: com.z0ltan.testing.NegativeTests
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runBefore
INFO: Running Setup code
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runBefore
INFO: Finished running Setup code
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testDoubleEqualFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testDoubleEqualFail- [FAILED]. Reason: 12.346 is not equal to 12.103
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testStringEqualFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testStringEqualFail- [FAILED]. Reason: Hello is not equal to World
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testLongEqualFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testLongEqualFail- [FAILED]. Reason: 12,345 is not equal to 54,321
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testListEqualFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testListEqualFail- [FAILED]. Reason: [1, 2, 3, 4, 5] is not equal to [1, 2, 3, 4, 5, 6, 7]
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testCharEqualFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testCharEqualFail- [FAILED]. Reason: t is not equal to
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testIntEqualFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testIntEqualFail- [FAILED]. Reason: 100 is not equal to 200
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testNullFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testNullFail- [FAILED]. Reason: java.lang.Object@14faf53 is not null
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testFalseFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testFalseFail- [FAILED]. Reason: true is true
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testListSameFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testListSameFail- [FAILED]. Reason: [1, 2, 3, 4, 5] is not the same object as [1, 2, 3, 4, 5, 6, 7]
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testNonNullFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testNonNullFail- [FAILED]. Reason: null is null
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Running Test Case: testTrueFail
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runTests
INFO: Test Case: testTrueFail- [FAILED]. Reason: false is false
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runAfter
INFO: Running Cleanup code
Aug 25, 2016 6:02:30 PM com.z0ltan.stf.STFCore runAfter
INFO: Finished running Cleanup code

Immaculate!

Learnings and Conclusion

The main aim of writing this framework was to see how a small, lightweight, and minimalist test framework might be written that was also sufficiently powerful and generic to be usable in a wide variety of situations. In that respect, I feel that this project has been successful.

In terms of learning, the following observations might be made:

  • A simple test framework can be readily implemented in only a few classes and a few hundred lines of code.
  • Using Reflection in Java is always a very tricky affair. It feels rather stilted, especially after using dynamic languages for much more powerful introspection and runtime modification of objects (Common Lisp, for instance). For instance, as can be seen in STFCore, the behaviour of InvocationTargetException in Java is to gobble up any exception thrown by the reflective call – this applies to both Checked and Unchecked exceptions. This is the reason why the explicit check for the cause of the exception needs to be made. Very clunky indeed!
  • There is a lot of boilerplate code in the STFAsserts file for providing overloaded versions of various methods. Even if some of the common logic were to be abstracted away into some helper methods, that would not really help with the overall code bloat. In Common Lisp, I could have simply written a macro to generate the code and saved dozens of methods of code. Even C++’s metaprogramming capabilities would have been better than Java’s facilities.
  • Implementing a test framework is simple enough. However, making it completely safe (think concurrency) and efficient, however, is an entirely different ball game. This version is safe enough because each test class is run on a different JVM process, but it’s far from being anywhere close to efficient or extensible.
  • Understanding Class Loaders is a crucial part of any well-rounded developer’s repertoire. There is no better way to truly grok it than to implement one yourself!
  • Finally, JShell really help! It seriously saved me a ton of mundane typing to test out various snippets of code and corner-cases.

Potentially trivially implemented improvements:

  • Support annotations with elements: @Test(expected=MyException.class) for instance.
  • Linking with source code to provide granular information about where in the code the specific test failed.
  • Show a summary of how many test cases have passed and how many have failed as well as the total number of test cases executed.
  • Support for full-fledged test suites.
  • Linking with IDEs such as Eclipse, NetBeans, and IntelliJ.

In some future post, I will discuss the implications of creating custom class loaders in Java, and class loaders themselves. This is a very important topic, and creating my own class loader for the STF framework has given me a fresh perspective into this whole esoteric domain of Java (more precisely, the JVM).

For now the next few posts will be related to the aforementioned functional implementations of the Untyped Lambda Calculus in Java, Common Lisp, and in C++. The Java version of this functional library will make extensive use of the STF framework for its testing. Note that the aim will be to not only create a functional library in each of these languages, but also to implement the libraries themselves as functionally as possible. In the future, I might write versions of this library in Python and Haskell as well. Haskell especially is indispensable when studying Functional Programming from a pragmatic viewpoint.

A Simple Test Framework (STF) in Java

The true raconteur and I ain’t talking about Feynman!

Feynman has often been described as a raconteur and I am sure he deserves that epithet without me justifying it or otherwise. The reason he was called so was because of his extremely direct manner of explaining, in the simplest of terms, some of the most complex concepts of advanced physics. His video lecture series was one of those myriad events which further reinforced his stature as a clear-headed genius. Now, that is not as trivial as it may seem at first. In my opinion, and arguably in an objective sense as well, the true measure of genuine understanding of any subject is the ability to express it without equivocation or self-doubt. No doubt eloquence, directness of approach and pithiness are characteristics which are often innate (and cultivable to a great extent through practice) but these attributes notwithstanding, the skill to present the matter at hand in a fluid and intelligible manner – caressing the topic with the confidence and prowess of a professional and actually putting forth all the salient points without diluting the content, is true raconteurism. Feynman was adept at that but recently I discovered another genius who actually surpasses him in this regard. Come forth, Joshua Bloch!

To the true technologist, Joshua Bloch requires no further introduction. He is the master and creator of the Java Collections framework (which is one of the finest components of the Java platform), the java.lang.Math package, the ‘assert’ mechanism (amongst others) and indeed has gone on to greater stature and achivements as Principal Engineer (with special emphasis on being the Java guru in-house) at Google. I had heard of him before through his publications such as Effective Java, Java Puzzlers and Concurrency in Java and yet had never had the inclination (call it laziness if you will) to actually read past the titles. The thought that came to my mind was “Yet another Java book… how many of these darned things am I supposed to read before I actually master this constantly mutating platform?”. This bring to mind another powerful concept that made sense to me long after I had been exposed to it. It is called the ‘Paradox of choice’. The video is available here. It is an excellent talk by Barry Schwartz. Quite non-technical but I feel it has begun to be more pertinent to this industry than any other (and will only continue to be more so!). As it so happened, one lazy afternoon when I was casually browsing through some Google Techtalks online, I came across this rather innocuous sounding title – “How to design a good API and why it matters”. It would be a whole six months before I would watch that video!

When I finished watching that hour of pure technological operatic genius, my first reaction was that of dejection. Yes, dejection at seeing the video end so soon! The whole hour had gone off so smoothly and everything in the meantime just made such clear sense that it was nothing short of amazing. I had been deeply engrossed in some major coding work in my current project and wished that I had seen this video two months back! Uncannily though, I discovered, to my own smug contentment, that I had already followed most of the principles that were enunciated in the talk. Some of the pertinent points that he puts forth are as follows and struck me as particularly useful are as follows:

1. Whenever in doubt, leave it out!

This makes perfect sense. As he says, every programmer who develops at least a module is an
API developer because the methods and the public components of his module form the contract
for any clients that actually use his module. This maxim follows the logic that a component can
be added to an API later on, but can never be removed without breaking
existing functionality.This also implies that any ‘doubtful’ functionality can always be harnessed
on later.

2. The ideal size for a method paramater list is three or lesser. If there are more, split
them and/or use helper classes

The example that he provides (of a Win32 API call to draw a simple window) in the video says it
all. I found this principle very useful (in retrospect) when I was designing my modules (I will
have a series of blogs on that very soon) and was often in a bit of a quandary about the exact
scope of my utility methods. The requirements at that point of time were pretty
much solidified and yet the use-cases were not yet finalized. So it was more of a trade-off
between usability and genericity which leads us straight to the next
point.

3. If the API makes everyone equally happy and leaves everyone equally unsatisfied,
it’s done its job

It’s always a tight balance when designing an API – whether to make it extremely powerful (and
invariably for a select group of stakeholders) or to make it powerful enough (and
thus for the majority of the stakeholders). The latter is always preferred irrespective of the
scope of the applicability of the API. The case which is moot in this regard is the VMware API –
generic enough to be almost bloated and yet powerful enough to be used by any client. More
on the VMware APIs in upcoming weeks.

4. Exception handling is not just an escape mechanism but a means of controlling
the system flow

Now this is one part of the talk where Joshua could not allocate any time at all. Considering
that the whole hour-long talk was filled with absolutely no pauses and no wastage of time, this
goes to show how the topic deserved more time than the standard hour allocated for Google
Techtalks. Nonetheless, in my current project, this was the hardest nut to crack. Exception
handling is an even finer balancing act than the design of the contract of the modules (i.e., the
methods) as there is a huge state-space of possible control flow/ processing paths. The sad
truth is that this is often the part which is most often neglected. I, however tried to have a
fairly robust and generic mechanism whereby there was a Exception class
with a corresponding ERROR code interface, which was placed in a common module so as
to allow clients to access and throw specific error codes. The level of granularity is still being
refined as I find more use-cases and corner-cases. In my opinionated opinion, the design
of the exception handling mechanism is the true measure of good or bad design.

5. Document! Document! Document!

Joshua recommends documenting every class (what an instance of it represents), every method
(this forms the contract of the class after all!), every parameter and return type (mentioning the
types and units of the parameters). How important this is can be estimated by this
example – In my current project, I have over 400, 000 lines of Java code modularized into a
dozen projects and hundreds of packages and sub-modules. The real eye-opener is that there is
absolutely no documentation whatsover. Okay, that sounded harsh. Let me rephrase it thus –
There is not a single line of useful documentation. If it were not for my considerable
intellectual gifts and a generally consistent design, it would be impossible to decipher the whole
mish-mash much less augment the functionality!

6. Watch the video paisan!

In closing, I could go on about how profound my experience was watching this video and I could ramble on about about the merits of the whole experience (gosh, it’s almost sounding like I had a technological seance… which wouldn’t be far from the truth! 😉 ) but this post has already gone on too long and I am tired! Just kidding… I am busy in my epiphanic reading of ‘Effective Java’, ‘Java Puzzlers’ and ‘Concurrent Programming in Java’. Peace!

The true raconteur and I ain’t talking about Feynman!