How to compile a Java program embedded in a String? Here’s how! (Better)

I am not quite satisfied with the way I left things in the last post.

There was just a bit too much hand-waving and magic about. Since then, I’ve experimented a bit more, and found out that some things that were claimed in the last couple of posts on this topic are also blatantly wrong – for instance, this approach of dynamic compilation does require a JDK (even though it does not need javac). The javax.tools package depends on tools.jar, which does not come bundled with the JRE.

This post is meant to make amends for that mistake, plus show a better way of handling compilation of Java code stored inside a String object without resorting to creating directories dynamically. In this attempt, we will try and make the following improvements:

  • The client code now invokes a custom compiling class loader.
  • The class loader then compiles the code stored inside the string.
  • The code is then loaded by the class loader into the current JVM.
  • Finally, the code is executed by the Client using Reflection.

Contents

  1. Code Listing
    1. The Class Loader
    2. The Compiler
  2. Testing
    1. The Client
    2. Test Run
  3. Conclusion

The Code

The code is organised as follows:

Timmys-MacBook-Pro:Better z0ltan$ tree
.
├── com
│   └── z0ltan
│   ├── compilers
│   │   └── JavaStringCompiler.java
│   └── loaders
│   └── CompilingClassLoader.java
└── org
└── z0ltan
└── client
└── Client.java

The Class Loader

Top

package com.z0ltan.loaders;

import java.io.File;
import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;

import java.nio.file.Files;
import java.nio.file.Paths;

import java.util.regex.Pattern;
import java.util.regex.Matcher;
import java.util.logging.Logger;

import com.z0ltan.compilers.JavaStringCompiler;

public class CompilingClassLoader extends ClassLoader {
    private static final CompilingClassLoader __instance
	= new CompilingClassLoader();

    private static final Logger logger =
        Logger.getLogger(CompilingClassLoader.class.getName());

    private Pattern namePattern;
    private Pattern packagePattern;

    private CompilingClassLoader() {
        this.namePattern =
	Pattern.compile(".*class[ ]+([a-zA-Z0-9$_]+).*");
        this.packagePattern =
	Pattern.compile(".*package[ ]+([a-zA-Z0-9$_.]+).*");
    }

    public static CompilingClassLoader getInstance() {
        return __instance;
    }

    // load the class file after compiling the code
    public Class<?> loadClassFromString(final String program) throws ClassNotFoundException {
        final String className = getClassName(program);
        final String packagePath = getPackagePath(program);

        final String fullClassName;
        if (packagePath != null) {
            fullClassName = packagePath + '.' + className;
        } else {
            fullClassName = className;
        }

        logger.info("Loading " + fullClassName);

        // compile it!
        boolean result =
		JavaStringCompiler.INSTANCE
		.compileStringCode(fullClassName, program);

        if (result) {
            byte[] classBytes = getClassBytes(className);
            if (classBytes != null) {
                logger.info("Loaded " + fullClassName);
                return defineClass(fullClassName, classBytes, 0, classBytes.length);
            } else
                throw new ClassNotFoundException("Unable to load: " + fullClassName +
                                                 ". Reason = failed to load class bytes.");
        } else
            throw new ClassNotFoundException("Unable to load: " + fullClassName +
                                             ". Reason = compilation failed.");
    }

    private String getClassName(final String program) {
        Matcher m = namePattern.matcher(program);

        if (m.matches() && (m.groupCount() == 1)) {
            return m.group(1);
        }
        throw new RuntimeException("Could not find main class to load!");
    }

    private String getPackagePath(final String program) {
        Matcher m = packagePattern.matcher(program);

        if (m.matches() && (m.groupCount() == 1)) {
            return m.group(1);
        }
        return null;
    }        

    private byte[] getClassBytes(final String className) {
        final String classFilePath =
            className.replace('.', File.separatorChar) + ".class";

        try (BufferedInputStream bin = new BufferedInputStream(new FileInputStream(classFilePath));
             ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
            byte[] buffer = new byte[4 * 1024];
            int bytesRead = -1;

            while ((bytesRead = bin.read(buffer)) != -1) {
                baos.write(buffer, 0, bytesRead);
            }

            // delete the class file before returning
            try {
                Files.deleteIfExists(Paths.get(classFilePath));
            } catch (IOException ex) {
                //
            }

            return baos.toByteArray();
        } catch (IOException ex) {
            return null;
        }
    }
}

The code is as simple as it gets. The loadClassFromString method takes in the Java String containing our program, and constructs the full class name by appending the package path (if any).

It then invokes the compiler to generate the .class file. Finally, it deletes the .class file once the bytes have been read from it, thus cleaning up the unnecessary file.

The Compiler

Top

package com.z0ltan.compilers;

import java.net.URI;

import java.io.File;
import java.io.IOException;

import java.util.Collections;
import java.util.Arrays;
import java.util.logging.Logger;

import javax.tools.ToolProvider;
import javax.tools.JavaCompiler;
import javax.tools.DiagnosticCollector;
import javax.tools.Diagnostic;
import javax.tools.DiagnosticCollector;
import javax.tools.StandardJavaFileManager;
import javax.tools.SimpleJavaFileObject;
import javax.tools.JavaFileObject;

public enum JavaStringCompiler {
    INSTANCE;

    private JavaCompiler compiler;
    private DiagnosticCollector<JavaFileObject> collector;
    private StandardJavaFileManager manager;

    private static final Logger logger =
        Logger.getLogger(JavaStringCompiler.class.getName());

    private JavaStringCompiler() {
        this.compiler = ToolProvider.getSystemJavaCompiler();
        this.collector  = new DiagnosticCollector<JavaFileObject>();
        this.manager = compiler.getStandardFileManager(collector, null, null);
    }

    // class to represent a string object as a source file
    class StringCodeObject extends SimpleJavaFileObject {
        private String code;

        StringCodeObject(final String name, final String code) {
            super(URI.create("string:///" + name.replace('.', File.separatorChar) +
                             Kind.SOURCE.extension),
                  Kind.SOURCE);
            this.code = code;
        }

        @Override
        public CharSequence getCharContent(boolean ignoreEncodingErrors) {
            return this.code;
        }
    }

    // Compile the Java code stored inside the string
    public boolean compileStringCode(final String name, final String code) {
        logger.info("Compiling: " + name);

        boolean result = false;
        StringCodeObject source = new StringCodeObject(name, code);

        result = compiler.getTask(null, manager, null, null, null,
                                  Collections.unmodifiableList(Arrays.asList(source))).call();

        // display errors, if any
        for (Diagnostic<? extends JavaFileObject> d : collector.getDiagnostics()) {
            System.err.format("Error at line: %d, in file: %s\n",
                              d.getLineNumber(),
                              d.getSource().toUri());
        }

        try {
            manager.close();
        } catch (IOException ex) {
            //
        }

        logger.info("Finished compiling: " + name);

        return result;
    }
}

This again makes use of the same facilities available in the javax.tools package. However, unlike last time, this time we make sure that we use the StandardJavaFileManager to output any compilation errors correctly. Apparently, even explicitly specifying a DiagnosticCollector instance in the compiler.getTask() call does not work. We need to have a file manager to be able to catch errors.

The rest of the is pretty much the same as last time.

Testing

Top

The testing code agains covers the two scenarios – using the default package, and using a more realistic package structure.

The Client

Top

Here is the sample client that we will use to test the code:

// The Client code

package org.z0ltan.client;

import java.lang.reflect.Method;
import java.util.logging.Logger;

import com.z0ltan.loaders.CompilingClassLoader;

public class Client {
    private static final Logger logger =
        Logger.getLogger(Client.class.getName());

    public static void main(String[] args) throws Exception {
        final String simpleProgram = "public class SimpleProgram {" +
            " public static void main(String[] args) {" +
            "    System.out.println(\"Hello from SimpleProgram!\");}}";

        testSimpleProgram(simpleProgram);

        final String complexProgram = "package foo.bar.baz.quux;" +
            "import java.util.Random;" +
            "public class ComplexProgram {"+
            "  public static void main(String[] args) {" +
            "   System.out.println(\"'Sup from Fubar\");}" +
            "public int getRandomNumber() {" +
            "  return (new Random()).nextInt(100);}}";

        testComplexProgram(complexProgram);
    }

    private static void testSimpleProgram(final String simpleProgram) throws Exception {
        logger.info("Testing SimpleProgram");

        Class<?> simpleClazz =
            CompilingClassLoader.getInstance().loadClassFromString(simpleProgram);

        if (simpleClazz != null) {
            Method main = simpleClazz.getDeclaredMethod("main", String[].class);

            if (main != null) {
                main.invoke(null, (Object)null);
            }
        }
        logger.info("Finished testing SimpleProgram");
    }

    private static void testComplexProgram(final String complexProgram) throws Exception {
        logger.info("Testing ComplexProgram");

        Class<?> complexClazz =
            CompilingClassLoader.getInstance().loadClassFromString(complexProgram);

        if (complexClazz != null) {
            Object obj = complexClazz.getConstructor().newInstance();
            if (obj != null) {
                Method main = complexClazz.getDeclaredMethod("main", String[].class);

                if (main != null) {
                    main.invoke(null, (Object)null);
                }

                Method getRandomNumber = complexClazz.getDeclaredMethod("getRandomNumber");
                if (getRandomNumber != null) {
                    int n = (int)getRandomNumber.invoke(obj);
                    System.out.format("Random number = %d\n", n);
                }
            }
        }
        logger.info("Finished testing ComplexProgram");
    }
}

Test Run

Top

Timmys-MacBook-Pro:Better z0ltan$ javac -cp . com/z0ltan/compilers/JavaStringCompiler.java 

Timmys-MacBook-Pro:Better z0ltan$ javac -cp . com/z0ltan/loaders/CompilingClassLoader.java 

Timmys-MacBook-Pro:Better z0ltan$ javac -cp . org/z0ltan/client/Client.java 

Timmys-MacBook-Pro:Better z0ltan$ java -cp . org.z0ltan.client.Client

Sep 16, 2016 2:17:49 PM org.z0ltan.client.Client testSimpleProgram
INFO: Testing SimpleProgram
Sep 16, 2016 2:17:49 PM com.z0ltan.loaders.CompilingClassLoader loadClassFromString
INFO: Loading SimpleProgram
Sep 16, 2016 2:17:50 PM com.z0ltan.compilers.JavaStringCompiler compileStringCode
INFO: Compiling: SimpleProgram
Sep 16, 2016 2:17:50 PM com.z0ltan.compilers.JavaStringCompiler compileStringCode
INFO: Finished compiling: SimpleProgram
Sep 16, 2016 2:17:50 PM com.z0ltan.loaders.CompilingClassLoader loadClassFromString
INFO: Loaded SimpleProgram
<strong>Hello from SimpleProgram!</strong>
Sep 16, 2016 2:17:50 PM org.z0ltan.client.Client testSimpleProgram
INFO: Finished testing SimpleProgram
Sep 16, 2016 2:17:50 PM org.z0ltan.client.Client testComplexProgram
INFO: Testing ComplexProgram
Sep 16, 2016 2:17:50 PM com.z0ltan.loaders.CompilingClassLoader loadClassFromString
INFO: Loading foo.bar.baz.quux.ComplexProgram
Sep 16, 2016 2:17:50 PM com.z0ltan.compilers.JavaStringCompiler compileStringCode
INFO: Compiling: foo.bar.baz.quux.ComplexProgram
Sep 16, 2016 2:17:50 PM com.z0ltan.compilers.JavaStringCompiler compileStringCode
INFO: Finished compiling: foo.bar.baz.quux.ComplexProgram
Sep 16, 2016 2:17:50 PM com.z0ltan.loaders.CompilingClassLoader loadClassFromString
INFO: Loaded foo.bar.baz.quux.ComplexProgram
<strong>'Sup from Fubar
Random number = 63</strong>
Sep 16, 2016 2:17:50 PM org.z0ltan.client.Client testComplexProgram
INFO: Finished testing ComplexProgram

And just to verify that no .class files are left in the file system:

Timmys-MacBook-Pro:Better z0ltan$ ls
com		compiler.jar	loader.jar	org

Excellent! That’s much better.

Conclusion

Top

Well, that was much more satisfying that the unnecessary complexity (and extra work!) of the previous version of this program. Now there is better separation of concerns, better diagnostics, and is much easier to read and understand.

Till next time then, folks!

How to compile a Java program embedded in a String? Here’s how! (Better)

How to compile a Java program embedded in a String? Here’s how!

Taking a small break from the mini-series on interop between languages (and also working on the main project – embedding a JVM instance in a Common Lisp image), I thought I’d share something that I frankly found amazing – how to compile and run a complete Java program contained inside a Java String! Who needs files any more, eh?

The implications of this are enormous – this means that on any machine where there is a JVM (version 6 or above) installed, even without the JDK itself*, not only can we compile and run arbitrary Java code, but also pass around Java code in strings, and compile and run them on the fly!

*Of course, it’s understood that the compiler code itself must be available in some form – as a class file, or as a Jar file.

Content

  1. A little background
  2. Implementation
    1. Compile Java code inside a String
    2. Compile Java code inside a String – fixed
  3. References

Background

I had previously written a post on how to compile Java code using the classes in the javax.tools package. That deals with the first part of the equation – how to dynamically compile Java code without resorting to brittle approaches like spawning OS processes , and without any dependency on javac.

You can find that discussion here – Dynamic Compilation of Java code (no JDK).

In this post, we will focus on the second aspect – how to store an entire Java program inside a Java String instance, and then compile and run the code.

Note: Since the javax.tools package was introduced only in Java 6, it’s only natural to deduce that these examples will work only in Java 6 (or above).

All right then, let’s get on with it!

Implementation

Top

The implementation will be in two parts. We’ll first implement a solution that appears to do the job – it can compile a Java program contained inside a string, and in many cases, even run it perfectly.

However, there is a major problem with the first implementation that we will then propose to fix by taking care of the various scenarios that might crop up in real-world use.

A first go

Top

The way we implement this solution is quite similar to the one that we used to compile Java source files. I recommend checking out the earlier post (mentioned in the Background section), but in the case of compiling a Java source string, we just have the following steps:

  • Get an instance of JavaCompiler using ToolProvider.
  • Construct the “compilation units” (which are instances of Iterable. In this specific case, we will need to construct a sub-type of SimpleJavaFileObject to pass in as the “compilation unit”.
  • Finally, create a compilation task using the compiler, and invoke call.

Well, that’s the general idea anyway. Let’s code that up now:

import java.io.File;
import java.io.IOException;

import java.util.Collections;
import java.util.Arrays;
import java.util.regex.Pattern;
import java.util.regex.Matcher;
import java.util.logging.Logger;

import java.net.URI;

import javax.tools.ToolProvider;
import javax.tools.JavaCompiler;
import javax.tools.SimpleJavaFileObject;
import javax.tools.JavaFileObject;
import javax.tools.JavaFileObject.Kind;


public enum JavaStringCompilerBasic {
	INSTANCE;
	
	private final JavaCompiler compiler;

	private Pattern namePattern;
		
	private JavaStringCompilerBasic() {
		this.compiler = ToolProvider.getSystemJavaCompiler();
		this.namePattern = Pattern.compile(".*class[ ]+([a-zA-Z0-9$_]+).*");
	}
	
	// class that defines the Java String object as a valid source file
	class EmbeddedJavaSource extends SimpleJavaFileObject {
		private String code;
		
		EmbeddedJavaSource(final String name, final String code) {
			super(URI.create("string:///" + 
							name.replace('.', File.separatorChar) + 
							Kind.SOURCE.extension),
					Kind.SOURCE);
		this.code = code;
		}
		
		@Override
		public CharSequence getCharContent(boolean ignoreEncodingErrors) {
			return this.code;
		}
	}

	/**
	  * Compile the Java code embedded inside the String.
	  * 
	  * @param program Java code
	  */	
	public void compileJavaString(final String program) {
		final String className = getClassName(program);
		
		final EmbeddedJavaSource sourceString 
				= new EmbeddedJavaSource(className, program);
		;
		
		compiler.getTask(null, null, null, null, null, 
			Collections.unmodifiableList(Arrays.asList(sourceString))).call();
	}
	
	private String getClassName(final String program) {
		final Matcher m = namePattern.matcher(program);
		
		if (m.matches() && (m.groupCount() == 1)) {
			return m.group(1);
		} else {
			throw new RuntimeException("Could not extract class name");
		}
	}

	public static void main(String[] args) {
		final String demoProgram 
			= "public class DemoProgram { " +
			  "   public static void main(String[] args) { " +
			  "       System.out.println(\"Hello from DemoProgram!\");" +
			  "    }" +
			  "}"; 	
	
		JavaStringCompilerBasic.INSTANCE.compileJavaString(demoProgram);
	}
}

As you can see, our little demo program is stored inside the demoProgram variable in the main method.


final String demoProgram
= "public class DemoProgram { " +
" public static void main(String[] args) { " +
" System.out.println(\"Hello from DemoProgram!\");" +
" }" +
"}";

Notes:: The main point of interest here is the EmbeddedJavaSource class which extends SimpleJavaFileObject. In this case, it doesn’t do much apart from creating a pseudo-URI pertaining to the fully-qualified name of the class (within the Java string).

Take it for a little spin:

Timmys-MacBook-Pro:Basic z0ltan$ javac -cp . JavaStringCompilerBasic.java 

Timmys-MacBook-Pro:Basic z0ltan$ java -cp . JavaStringCompilerBasic

Timmys-MacBook-Pro:Basic z0ltan$ java -cp . DemoProgram
Hello from DemoProgram!

Brilliant! Everything seems hunky-dory. Well, not quite. Let’s clean out the class files and try out another example to see if it is indeed working as we expect it to:

Here is the snippet we add to main:


final String problemProgram
= "package com.foo.bar;" +
"public class ProblemDemoProgram {" +
" public static void main(String[] args) {" +
" System.out.println(\"Hello from ProblemDemoProgram!\");" +
" }" +
"}";

So what’s the problem? Let’s run it:

Timmys-MacBook-Pro:Basic z0ltan$ java -cp . ProblemDemoProgram
Error: Could not find or load main class ProblemDemoProgram

Ah! That doesn’t look too good! What really happened is this – The compilation went through fine because the compiler doesn’t really care what the package definition inside the embedded Java program is (because the file doesn’t really exist on the file system now, does it?).

So it goes ahead and generates the .class file anyway. Now, when we try to run it, the java tool reads the byte code, sees a package declaration com.foo.bar, and tries to load up com/foo/bar/ProblemDemoProgram.class which, clearly, doesn’t exist!

So how do we fix it? The solution I propose it to ensure that the use case does succeed instead of failing. I propose reading the package information from the embedded Java program, parsing out the package details (if any), and creating that entire directory structure, and copying the .class file into that directory structure.

This will ensure that when java goes looking for DemoProgram.class, it really does find it, load it, and run it.

We’ll see how that solution might look like in the next section.

Final version

Top

import java.io.File;
import java.io.IOException;

import java.nio.file.Files;
import java.nio.file.Paths;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;

import java.util.Collections;
import java.util.Arrays;
import java.util.regex.Pattern;
import java.util.regex.Matcher;
import java.util.logging.Logger;

import java.net.URI;

import javax.tools.ToolProvider;
import javax.tools.JavaCompiler;
import javax.tools.SimpleJavaFileObject;
import javax.tools.JavaFileObject;
import static javax.tools.JavaFileObject.Kind;


public enum JavaStringCompilerFinal {
	INSTANCE;
	
	private final JavaCompiler compiler;

	private Pattern namePattern;
	private Pattern pkgPattern;
		
	private JavaStringCompilerFinal() {
		this.compiler = ToolProvider.getSystemJavaCompiler();
		this.namePattern = Pattern.compile(".*class[ ]+([a-zA-Z0-9$_]+).*");
		this.pkgPattern = Pattern.compile(".*package[ ]+([a-zA-Z0-9$_.]+).*");
	}
	
	// class that defines the Java String object as a valid source file
	class EmbeddedJavaSource extends SimpleJavaFileObject {
		private String code;
		
		EmbeddedJavaSource(final String name, final String code) {
			super(URI.create("string:///" + 
							name.replace('.', File.separatorChar) + 
							Kind.SOURCE.extension),
					Kind.SOURCE);
			this.code = code;
		}
		
		@Override
		public CharSequence getCharContent(boolean ignoreEncodingErrors) {
			return this.code;
		}
	}

	/**
	  * Compile the Java code embedded inside the String.
	  * 
	  * @param program Java code
	  */	
	public void compileJavaString(final String program) {
		String className = getClassName(program);
		
		final String packagePath = getPackagePath(program);
		
		if (packagePath != null) {
			makePackagePaths(packagePath);
			className = packagePath + '.' + className;
		} 
		
		final EmbeddedJavaSource sourceString 
				= new EmbeddedJavaSource(className, program);

		compiler.getTask(null, null, null, null, null, 
				Collections.unmodifiableList(Arrays.asList(sourceString))).call();
				
		// move the compiled class into the created folder
		if (packagePath != null) {
			moveClassIntoPackagePath(className);
		}
	}
	
	private void moveClassIntoPackagePath(final String className) {
		final String sourceFile
			= className.substring(className.lastIndexOf('.') +1)
				+ ".class";

		final String targetFile 
			= className.substring(0, className.lastIndexOf('.'))
						.replace('.', File.separatorChar)
						+ File.separatorChar
						+ sourceFile;
						
		try {
			Files.move(Paths.get(sourceFile), 
					   Paths.get(targetFile), 
					   StandardCopyOption.REPLACE_EXISTING);
		} catch (IOException ex) {
			throw new RuntimeException("Error while moving file: " + sourceFile + " to "
				+ targetFile + ". Message = " + ex.getLocalizedMessage());
		}
	}
	
	private void makePackagePaths(final String pkgPath) {
		final String pkgFilePath = pkgPath.replace('.', File.separatorChar);
		
		try {
			if (!Files.exists(Paths.get(pkgFilePath))) {
				Files.createDirectories(Paths.get(pkgFilePath));
			}
		} catch (IOException ex) {
			throw new RuntimeException("Could not create directories: " +
						pkgFilePath);
		}
	}
	
	private String getPackagePath(final String program) {
		final Matcher m = pkgPattern.matcher(program);
		
		if (m.matches() && (m.groupCount() == 1)) {
			return m.group(1);
		} 
		return null;
	}
			 
	private String getClassName(final String program) {
		final Matcher m = namePattern.matcher(program);
		
		if (m.matches() && (m.groupCount() == 1)) {
			return m.group(1);
		} else {
			throw new RuntimeException("Could not extract class name");
		}
	}
	
	
	public static void main(String[] args) {
		final String demoProgram 
			= "public class DemoProgram { " +
			  "   public static void main(String[] args) { " +
			  "       System.out.println(\"Hello from DemoProgram!\");" +
			  "    }" +
			  "}"; 	
	
		JavaStringCompilerFinal.INSTANCE.compileJavaString(demoProgram);
		
		final String problemProgram
			= "package com.foo.bar;" +
			  "public class ProblemDemoProgram {" +
			  "   public static void main(String[] args) {" +
			  "      System.out.println(\"Hello from ProblemDemoProgram!\");" +
			  "  }" +
			  "}";
			  
		JavaStringCompilerFinal.INSTANCE.compileJavaString(problemProgram);
	}
}

And a sample test run with the same example as in the previous section:

Timmys-MacBook-Pro:Final z0ltan$ javac -cp . JavaStringCompilerFinal.java 

Timmys-MacBook-Pro:Final z0ltan$ java -cp . JavaStringCompilerFinal

Timmys-MacBook-Pro:Final z0ltan$ ls
DemoProgram.class					JavaStringCompilerFinal.class		com
JavaStringCompilerFinal$EmbeddedJavaSource.class	JavaStringCompilerFinal.java

Timmys-MacBook-Pro:Final z0ltan$ java -cp . DemoProgram
Hello from DemoProgram!

Timmys-MacBook-Pro:Final z0ltan$ java -cp . com.foo.bar.ProblemDemoProgram
Hello from ProblemDemoProgram!

Timmys-MacBook-Pro:Final z0ltan$ tree com/
com/
└── foo
    └── bar
        └── ProblemDemoProgram.class

2 directories, 1 file

Woo-hoo! As we can see, the directory structure is created, the .class file moved into the directory, and the code runs beautifully.

Some observations:

The only difference from the first version is that we have added extra code to create the directories (using regex to extract the relevant details from the program code), compiled the Java code stored in the strings, and copied the class file to the relevant directories. The program is then invoked normally by specifying the full class path of the Java class.

Of course, the javax.tools package is primarily aimed at compiler writers who want to create custom compilers on top of Java, but covering that use case is beyond the scope of this blog.

For instance, the better approach would have been to implement a custom JavaFileManager that then implemented the creation of the directories (if needed), or we could have used a Class Loader to load the compiled class (in case we don’t need the source, etc. The possibilities are endless!

References

Top

Here is basically the only reference that you need (really) to get started with this topic, and run with it!

Till next time then, folks!

How to compile a Java program embedded in a String? Here’s how!

Compiling Java files dynamically using pure Java

Java 6 (and above) provides a wonderful facility to compile Java code using pure Java code. This is done through the tools provided in the javax.tools package. What this allows us to do is to load and compile batches of files without needing to spawn off a separate system process which is not only brittle but highly unreliable in terms of error handling and status reporting.

The idea for this post came about as I was working on creating a custom class loader for use with my STF (Simple Test Framework) testing framework (next post). The main use case that I wanted to support was to allow the user to specify a class name, and then have the class loader load the source file if the class file didn’t exist, compile the source file, and then load that class instead. The advantage of this is that the whole process is completely opaque to the user — in case the test file has not already been compiled, the class loader will take care of that. The onus of linking all the dependencies of the test file via the class path is, of course, on the user.

Dynamic compilation of Java code Pre-Java 6

Before the introduction of the javax.tools package in Java 6, if we wanted to compile Java source files on the fly, we would have to do something like the following:

import java.util.logging.Logger;
import java.text.MessageFormat;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.IOException;


public abstract class CompileJavaFileRuntime {
        private static final Logger logger =  Logger.getLogger(CompileJavaFileRuntime.class.getName());
        private static final MessageFormat formatter = new MessageFormat("javac -cp . {0}”);

        private CompileJavaFileRuntime() {}

        public static void main(String[] args) {
                if (args == null || args.length == 0)
                        throw new RuntimeException("specify at least one source file");

                compileSourceFiles(args);
        }

        public static void compileSourceFiles(String[] files) {
                for (String file : files) {
                        compileSourceFile(file);
                }
        }

        public static void compileSourceFile(String file) {
                logger.info("Compiling file: " + file);

                String command = formatter.format(new String[] { file });

                Process p = null;
                try {
                       p = Runtime.getRuntime().exec(command);
                       if (p != null) {
                               p.waitFor();
                               logger.info("Diagnostic info for file: " + file);
                               try (BufferedReader reader = 
                                         new BufferedReader(new InputStreamReader(p.getInputStream()))) {
                                        logger.info(reader.readLine());
                               } catch (IOException ex) {
                               }
                       }
                } catch (InterruptedException | IOException ex) {
                        logger.severe("Error while compiling source file: " + file + 
                           ". Reason = " + ex.getLocalizedMessage());
                        throw new RuntimeException(ex);
                } finally {
                        if (p != null && p.isAlive()) {
                                p.destroy();
                        }
                }
                logger.info("Finished compiling file: " + file);
        }
}

Aug 23, 2016 11:19:25 PM CompileJavaFileRuntime compileSourceFile
INFO: Compiling file: Bar.java
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: Diagnostic info for file: Bar.java
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: null
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: Finished compiling file: Bar.java
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: Compiling file: ClassClient.java
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: Diagnostic info for file: ClassClient.java
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: null
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: Finished compiling file: ClassClient.java
Aug 23, 2016 11:19:26 PM CompileJavaFileRuntime compileSourceFile
INFO: Compiling file: ClassPreamble.java
Aug 23, 2016 11:19:27 PM CompileJavaFileRuntime compileSourceFile
INFO: Diagnostic info for file: ClassPreamble.java
Aug 23, 2016 11:19:27 PM CompileJavaFileRuntime compileSourceFile
INFO: null
Aug 23, 2016 11:19:27 PM CompileJavaFileRuntime compileSourceFile
INFO: Finished compiling file: ClassPreamble.java

or, better still,

import java.util.logging.Logger;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.IOException;


public abstract class CompileJavaFileProcessBuilder {
        private static final Logger logger = Logger.getLogger(CompileJavaFileProcessBuilder.class.getName());

        private CompileJavaFileProcessBuilder() {}

        public static void main(String[] args) {
                if (args == null || args.length == 0)
                        throw new RuntimeException("specify at least one source file");

                compileSourceFiles(args);
        }

        public static void compileSourceFiles(String[] files) {
                for (String file : files) {
                        compileSourceFile(file);
                }
        }

        public static void compileSourceFile(String file) {
                logger.info("Compiling file: " + file);

                Process p = null;
                try {
                       p = new ProcessBuilder("javac", “-cp”, “.”, file).start();
                       if (p != null) {
                               p.waitFor();
                               logger.info("Diagnostic info for file: " + file);
                               try (BufferedReader reader = 
                                         new BufferedReader(new InputStreamReader(p.getInputStream()))) {
                                        logger.info(reader.readLine());
                               } catch (IOException ex) {
                               }
                       }
                } catch (InterruptedException | IOException ex) {
                        logger.severe("Error while compiling source file: " + file + 
                           ". Reason = " + ex.getLocalizedMessage());
                        throw new RuntimeException(ex);
                } finally {
                        if (p != null && p.isAlive()) {
                                p.destroy();
                        }
                }
                logger.info("Finished compiling file: " + file);
        }
}

Aug 23, 2016 11:18:08 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: Compiling file: Bar.java
Aug 23, 2016 11:18:09 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: Diagnostic info for file: Bar.java
Aug 23, 2016 11:18:09 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: null
Aug 23, 2016 11:18:09 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: Finished compiling file: Bar.java
Aug 23, 2016 11:18:09 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: Compiling file: ClassClient.java
Aug 23, 2016 11:18:10 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: Diagnostic info for file: ClassClient.java
Aug 23, 2016 11:18:10 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: null
Aug 23, 2016 11:18:10 PM CompileJavaFileProcessBuilder compileSourceFile
INFO: Finished compiling file: ClassClient.java

In both cases, we create native OS processes to compile each Java file in turn. Both of these version are essentially the same, but the ProcessBuilder approach allows much more customisation that is possible with the barebones Runtime.getRuntime().exec() approach. For this trivial case, however, the difference is inconsequential.

Dynamic compilation of Java code from Java 6 onwards

Consider the following class that can compile the source files passed in as command-line arguments:

import javax.tools.ToolProvider;
import javax.tools.JavaCompiler;
import javax.tools.StandardJavaFileManager;
import javax.tools.JavaFileObject;
import javax.tools.Diagnostic;
import javax.tools.DiagnosticCollector;

import java.io.File;
import java.io.IOException;

import java.util.logging.Logger;
import java.util.Arrays;
import java.util.Locale;

import java.nio.charset.Charset;
import java.text.MessageFormat;


public abstract class CompileJavaFileTools {
        private static final Logger logger = Logger.getLogger(CompileJavaFileTools.class.getName());

        private CompileJavaFileTools() {}

        public static void main(String[] args) {
                if (args == null || args.length == 0)
                        throw new RuntimeException("specify at least one source file");

                compileSourceFiles(args);
        }

        public static void compileSourceFiles(String[] files) {
                final String fileNames = ToolsHelper.stringArrayToString(files);
                logger.info("Compiling source files: " + fileNames);

                File[] fileArray = new File[files.length];
                for (int i = 0; i < fileArray.length; i++) {
                        fileArray[i] = new File(files[i]);
                }
                                                        
                JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
                DiagnosticCollector<JavaFileObject> collector = new DiagnosticCollector<>();
                StandardJavaFileManager manager = 
                	compiler.getStandardFileManager(collector, Locale.getDefault(),      								        Charset.forName("UTF-8"));
                Iterable<? extends JavaFileObject> compilationUnits = 	          
                	manager
			.getJavaFileObjectsFromFiles(Arrays.asList(fileArray));

                compiler.getTask(null, manager, collector, null, null, compilationUnits).call();

                for (Diagnostic<? extends JavaFileObject> d : collector.getDiagnostics()) {
                        final String message = MessageFormat.format("Error at line: {0}, in file: {1}\n", 
                                d.getLineNumber(), d.getSource().toUri());
                        logger.severe(message);
                }

                try {
                        manager.close();
                } catch (IOException ex) {
                        logger.severe("Error while closing file manager. Message = " 
                                + ex.getLocalizedMessage());
                }                        

                logger.info("Finished compiling source files: " + fileNames);
        }

        static class ToolsHelper {
                public static String stringArrayToString(String[] strings) {
                        StringBuffer buffer = new StringBuffer();

                        if (strings == null)
                                return null;

                        for (String string : strings) {
                                buffer.append(string);
                                buffer.append(" ");
                        }

                        return "[ " + buffer.toString().trim() + " ]";
                }
        }
}
                                                        
                                
Aug 23, 2016 11:53:59 PM CompileJavaFileTools compileSourceFiles
INFO: Compiling source files: [ Bar.java ClassPreamble.java ]
Aug 23, 2016 11:54:00 PM CompileJavaFileTools compileSourceFiles
INFO: Finished compiling source files: [ Bar.java ClassPreamble.java ]

Explanation:

The sequence of steps to compile source files using the javax.tools package is as follows:

  • Create an instance of the Java Compiler:
    JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
    
  • Create an instance of a DiagnosticCollector (which implements DiagnosticListener). This class is used to log any compilation issues:
    DiagnosticCollector<JavaFileObject> collector = new DiagnosticCollector<>();
    
  • Get a handle to the standard file handler to handle source files:
     StandardJavaFileManager manager = 
                    	compiler.getStandardFileManager(collector, Locale.getDefault(),      								        Charset.forName("UTF-8"));
    
  • Create an internal representation of the compilation units (one per source file):
    Iterable<? extends JavaFileObject> compilationUnits = 	          
                    	manager
    			.getJavaFileObjectsFromFiles(Arrays.asList(fileArray));
    
  • Finally perform the actual compilation. The parameters passed to the getTask method are – output stream writer, the file manager, an instance of DiagnosticListener, options, classes, and the compilation units:
    compiler.getTask(null, manager, collector, null, null, compilationUnits).call();
    
  • Finally, don’t forget to close the file manager handle to prevent any leaks:
    try {
                            manager.close();
                    } catch (IOException ex) {
                            logger.severe("Error while closing file manager. Message = " 
                                    + ex.getLocalizedMessage());
                    }          
    

As simple as it gets! This is not only more reliable, but also more flexible that spawning separate processes to compile Java source files dynamically. Also note that there is better error reporting in the form of Diagnostic Listeners. This is infinitely superior to parsing error messages from a process’ output stream.

Closing thoughts

The classes and interfaces provided in the javax.tools package supply the developer with powerful tools to generate byte code dynamically. This can be not only, as in the example mentioned, develop efficient and simple ways to generate compiling class loaders, but also build more complex tools that depend on dynamic generation of byte code such as compilers. All in all, the main advantage of this approach is that there is no dependency on any external tools. The JDK provides all the functionality that one needs to write platform-independent, dependable byte code generation code.

Compiling Java files dynamically using pure Java