Compiling Java files dynamically using pure Java


Java 6 (and above) provides a wonderful facility to compile Java code using pure Java code. This is done through the tools provided in the javax.tools package. What this allows us to do is to load and compile batches of files without needing to spawn off a separate system process which is not only brittle but highly unreliable in terms of error handling and status reporting.

The idea for this post came about as I was working on creating a custom class loader for use with my STF (Simple Test Framework) testing framework (next post). The main use case that I wanted to support was to allow the user to specify a class name, and then have the class loader load the source file if the class file didn’t exist, compile the source file, and then load that class instead. The advantage of this is that the whole process is completely opaque to the user — in case the test file has not already been compiled, the class loader will take care of that. The onus of linking all the dependencies of the test file via the class path is, of course, on the user.

Dynamic compilation of Java code Pre-Java 6

Before the introduction of the javax.tools package in Java 6, if we wanted to compile Java source files on the fly, we would have to do something like the following:

import java.util.logging.Logger;
import java.text.MessageFormat;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.IOException;


public abstract class CompileJavaFileRuntime {
        private static final Logger logger =  Logger.getLogger(CompileJavaFileRuntime.class.getName());
        private static final MessageFormat formatter = new MessageFormat("javac -cp . {0}”);

        private CompileJavaFileRuntime() {}

        public static void main(String[] args) {
                if (args == null || args.length == 0)
                        throw new RuntimeException("specify at least one source file");

                compileSourceFiles(args);
        }

        public static void compileSourceFiles(String[] files) {
                for (String file : files) {
                        compileSourceFile(file);
                }
        }

        public static void compileSourceFile(String file) {
                logger.info("Compiling file: " + file);

                String command = formatter.format(new String[] { file });

                Process p = null;
                try {
                       p = Runtime.getRuntime().exec(command);
                       if (p != null) {
                               p.waitFor();
                               logger.info("Diagnostic info for file: " + file);
                               try (BufferedReader reader = 
                                         new BufferedReader(new InputStreamReader(p.getInputStream()))) {
                                        logger.info(reader.readLine());
                               } catch (IOException ex) {
                               }
                       }
                } catch (InterruptedException | IOException ex) {
                        logger.severe("Error while compiling source file: " + file + 
                           ". Reason = " + ex.getLocalizedMessage());
                        throw new RuntimeException(ex);
                } finally {
                        if (p != null && p.isAlive()) {
                                p.destroy();
                        }
                }
                logger.info("Finished compiling file: " + file);
        }
}

Aug 23, 2016 11:19:25 PM CompileJavaFileRuntime compileSourceFile
INFO: Compiling file: Bar.java
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: Diagnostic info for file: Bar.java
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: null
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: Finished compiling file: Bar.java
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: Compiling file: ClassClient.java
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: Diagnostic info for file: ClassClient.java
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: null
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: Finished compiling file: ClassClient.java
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: Compiling file: ClassPreamble.java
Aug 23, 2016 11:19:27 PM CompileJavaFileRuntime compileSourceFile
INFO: Diagnostic info for file: ClassPreamble.java
Aug 23, 2016 11:19:27 PM CompileJavaFileRuntime compileSourceFile
INFO: null
Aug 23, 2016 11:19:27 PM CompileJavaFileRuntime compileSourceFile
INFO: Finished compiling file: ClassPreamble.java

or, better still,

import java.util.logging.Logger;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.IOException;


public abstract class CompileJavaFileProcessBuilder {
        private static final Logger logger = Logger.getLogger(CompileJavaFileProcessBuilder.class.getName());

        private CompileJavaFileProcessBuilder() {}

        public static void main(String[] args) {
                if (args == null || args.length == 0)
                        throw new RuntimeException("specify at least one source file");

                compileSourceFiles(args);
        }

        public static void compileSourceFiles(String[] files) {
                for (String file : files) {
                        compileSourceFile(file);
                }
        }

        public static void compileSourceFile(String file) {
                logger.info("Compiling file: " + file);

                Process p = null;
                try {
                       p = new ProcessBuilder("javac", “-cp”, “.”, file).start();
                       if (p != null) {
                               p.waitFor();
                               logger.info("Diagnostic info for file: " + file);
                               try (BufferedReader reader = 
                                         new BufferedReader(new InputStreamReader(p.getInputStream()))) {
                                        logger.info(reader.readLine());
                               } catch (IOException ex) {
                               }
                       }
                } catch (InterruptedException | IOException ex) {
                        logger.severe("Error while compiling source file: " + file + 
                           ". Reason = " + ex.getLocalizedMessage());
                        throw new RuntimeException(ex);
                } finally {
                        if (p != null && p.isAlive()) {
                                p.destroy();
                        }
                }
                logger.info("Finished compiling file: " + file);
        }
}

Aug 23, 2016 11:18:08 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: Compiling file: Bar.java
Aug 23, 2016 11:18:09 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: Diagnostic info for file: Bar.java
Aug 23, 2016 11:18:09 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: null
Aug 23, 2016 11:18:09 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: Finished compiling file: Bar.java
Aug 23, 2016 11:18:09 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: Compiling file: ClassClient.java
Aug 23, 2016 11:18:10 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: Diagnostic info for file: ClassClient.java
Aug 23, 2016 11:18:10 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: null
Aug 23, 2016 11:18:10 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: Finished compiling file: ClassClient.java

In both cases, we create native OS processes to compile each Java file in turn. Both of these version are essentially the same, but the ProcessBuilder approach allows much more customisation that is possible with the barebones Runtime.getRuntime().exec() approach. For this trivial case, however, the difference is inconsequential.

Dynamic compilation of Java code from Java 6 onwards

Consider the following class that can compile the source files passed in as command-line arguments:

import javax.tools.ToolProvider;
import javax.tools.JavaCompiler;
import javax.tools.StandardJavaFileManager;
import javax.tools.JavaFileObject;
import javax.tools.Diagnostic;
import javax.tools.DiagnosticCollector;

import java.io.File;
import java.io.IOException;

import java.util.logging.Logger;
import java.util.Arrays;
import java.util.Locale;

import java.nio.charset.Charset;
import java.text.MessageFormat;


public abstract class CompileJavaFileTools {
        private static final Logger logger = Logger.getLogger(CompileJavaFileTools.class.getName());

        private CompileJavaFileTools() {}

        public static void main(String[] args) {
                if (args == null || args.length == 0)
                        throw new RuntimeException("specify at least one source file");

                compileSourceFiles(args);
        }

        public static void compileSourceFiles(String[] files) {
                final String fileNames = ToolsHelper.stringArrayToString(files);
                logger.info("Compiling source files: " + fileNames);

                File[] fileArray = new File[files.length];
                for (int i = 0; i < fileArray.length; i++) {
                        fileArray[i] = new File(files[i]);
                }
                                                        
                JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
                DiagnosticCollector<JavaFileObject> collector = new DiagnosticCollector<>();
                StandardJavaFileManager manager = 
                	compiler.getStandardFileManager(collector, Locale.getDefault(),      								        Charset.forName("UTF-8"));
                Iterable<? extends JavaFileObject> compilationUnits = 	          
                	manager
			.getJavaFileObjectsFromFiles(Arrays.asList(fileArray));

                compiler.getTask(null, manager, collector, null, null, compilationUnits).call();

                for (Diagnostic<? extends JavaFileObject> d : collector.getDiagnostics()) {
                        final String message = MessageFormat.format("Error at line: {0}, in file: {1}\n", 
                                d.getLineNumber(), d.getSource().toUri());
                        logger.severe(message);
                }

                try {
                        manager.close();
                } catch (IOException ex) {
                        logger.severe("Error while closing file manager. Message = " 
                                + ex.getLocalizedMessage());
                }                        

                logger.info("Finished compiling source files: " + fileNames);
        }

        static class ToolsHelper {
                public static String stringArrayToString(String[] strings) {
                        StringBuffer buffer = new StringBuffer();

                        if (strings == null)
                                return null;

                        for (String string : strings) {
                                buffer.append(string);
                                buffer.append(" ");
                        }

                        return "[ " + buffer.toString().trim() + " ]";
                }
        }
}
                                                        
                                
Aug 23, 2016 11:53:59 PM CompileJavaFileTools compileSourceFiles
INFO: Compiling source files: [ Bar.java ClassPreamble.java ]
Aug 23, 2016 11:54:00 PM CompileJavaFileTools compileSourceFiles
INFO: Finished compiling source files: [ Bar.java ClassPreamble.java ]

Explanation:

The sequence of steps to compile source files using the javax.tools package is as follows:

  • Create an instance of the Java Compiler:
    JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
    
  • Create an instance of a DiagnosticCollector (which implements DiagnosticListener). This class is used to log any compilation issues:
    DiagnosticCollector<JavaFileObject> collector = new DiagnosticCollector<>();
    
  • Get a handle to the standard file handler to handle source files:
     StandardJavaFileManager manager = 
                    	compiler.getStandardFileManager(collector, Locale.getDefault(),      								        Charset.forName("UTF-8"));
    
  • Create an internal representation of the compilation units (one per source file):
    Iterable<? extends JavaFileObject> compilationUnits = 	          
                    	manager
    			.getJavaFileObjectsFromFiles(Arrays.asList(fileArray));
    
  • Finally perform the actual compilation. The parameters passed to the getTask method are – output stream writer, the file manager, an instance of DiagnosticListener, options, classes, and the compilation units:
    compiler.getTask(null, manager, collector, null, null, compilationUnits).call();
    
  • Finally, don’t forget to close the file manager handle to prevent any leaks:
    try {
                            manager.close();
                    } catch (IOException ex) {
                            logger.severe("Error while closing file manager. Message = " 
                                    + ex.getLocalizedMessage());
                    }          
    

As simple as it gets! This is not only more reliable, but also more flexible that spawning separate processes to compile Java source files dynamically. Also note that there is better error reporting in the form of Diagnostic Listeners. This is infinitely superior to parsing error messages from a process’ output stream.

Closing thoughts

The classes and interfaces provided in the javax.tools package supply the developer with powerful tools to generate byte code dynamically. This can be not only, as in the example mentioned, develop efficient and simple ways to generate compiling class loaders, but also build more complex tools that depend on dynamic generation of byte code such as compilers. All in all, the main advantage of this approach is that there is no dependency on any external tools. The JDK provides all the functionality that one needs to write platform-independent, dependable byte code generation code.

Advertisements
Compiling Java files dynamically using pure Java

One thought on “Compiling Java files dynamically using pure Java

Speak your mind!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s