Interop mini-series – Calling C and C++ Callbacks from Java (Part 4)

Continuing from the last post, I will show how we can use JNA to allow C (and C++) code to invoke callback functions implemented in pure Java.

In case you are a bit rusty on callbacks, I have a whole series of posts on that, as part of this series. Feel free to check those out, starting with Callbacks.

Let’s jump right into it then!

Contents

  1. Demo
  2. Conclusion

Demo

For this demo, let’s pick a very simple example. (This is the same example as covered in the post covering Common Lisp callbacks from C/C++ code.).

We have a person type which has the following slots/fields – name, gender, and age. From our Java code, we want to instantiate an instance of person, and then use a function in a native library, prefix_name to append either “Mr.” or “Miss” in front of the person’s name, depending on the value of the gender slot (0 for female, anything else for male).

First we define the interface for the native library (in callback_demo.h):

#ifndef __CALLBACK_DEMO_H__
#define __CALLBACK_DEMO_H__ "callback_demo.h"

typedef struct person {
    char* name;
    int gender;
    int age;
} person;

#ifdef __cplusplus
extern "C" {
#endif
    void prefix_name(person*, void (*)(person*));
#ifdef __cplusplus
}
#endif
#endif

We then write the code containing the prefix_name function that will invoke our callback function (in callback_demo.c):

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "callback_demo.h"

#define MAXSIZE 50

char* concatenate_names(const char* prefix, char* name)
{
    int len = strlen(prefix) + strlen(name) + 1;

    char* full_name = (char*)malloc(len * sizeof(char));

    if (full_name != NULL) {
        char* cp = full_name;

        while (*prefix != '\0')
            *cp++ = *prefix++;

        *cp++ = 0x20;

        while (*name != '\0')
            *cp++ = *name++;
   
         *cp = '\0';

        return full_name;
    }
   return name;
}
   

void prefix_name(person* p, void (*cb)(person*))
{
    const char* MISTER = "Mr.";
    const char* MISS = "Ms.";
    char* res = NULL;

    // 0 - female, anything else male
    res = p->gender == 0 ? concatenate_names(MISS, p->name) :
                           concatenate_names(MISTER, p->name);
    strcpy(p->name, res);
    
    (*cb)(p);
}

void sample_callback(person* p)
{
    printf("%s, %s, %d\n", p->name, p->gender == 0 ? "Female" : "Male", p->age);
}

int main()
{
    person rich;

    rich.name = (char*)malloc(MAXSIZE * sizeof(char));
    strcpy(rich.name, "Rich");
    rich.gender = 1;
    rich.age = 49;

    prefix_name(&rich, &sample_callback);
    
    return 0;
}

Explanation: The code is relatively straightforward. As can be seen from the header file, prefix_name is the entry point to the library (and the one which gets invoked from the Java code).

The prefix_name function takes an instance of the person structure as well as a callback function. Note the signature of the callback function:

void (*)(person*).

This callback function expects to be passed a modified instance of the person instance that is the first parameter of the prefix_name function.

The logic is very simple – simply check for the gender field, and then depending on whether it is 0 or some other way, update the name field of the person instance by prepending “Miss” or “Mr.” respectively.

Finally, the callback function cb is invoked, passing control back to the client code.

All right, now we compile the code into a library, libcallbackdemo.dylib:

Timmys-MacBook-Pro:Demo z0ltan$ clang -dynamiclib -o libcallbackdemo.dylib callback_demo.c

Timmys-MacBook-Pro:Demo z0ltan$ ls
callback_demo.c		callback_demo.h		libcallbackdemo.dylib

Excellent!

The Java code to plug into the native libraries surprisingly concise and simple (in CallbackDemo.java):

import com.sun.jna.Library;
import com.sun.jna.Native;
import com.sun.jna.Callback;
import com.sun.jna.Platform;
import com.sun.jna.Structure;

import java.util.List;
import java.util.Arrays;

public class CallbackDemo {
    private static final String lib;

    static {
        if (Platform.isMac())
            lib = "libcallbackdemo.dylib";
        else if (Platform.isWindows())
            lib = "libcallbackdemo.dll";
        else
            lib = "libcallbackdemo.so";
    }

    public interface CallbackDemoLib extends Library {
        CallbackDemoLib INSTANCE = (CallbackDemoLib)Native
                                    .loadLibrary(lib, CallbackDemoLib.class);

        interface PrefixCallback extends Callback {
            void print(Person person);
        }

        void prefix_name(Person person, PrefixCallback func);
    }

    public static class Person extends Structure {
       public String name;
       public int gender;
       public int age;

        @Override
        public List<String> getFieldOrder() {
            return Arrays.asList(new String[] {"name", "gender", "age"});
        }
    }

    public static void main(String[] args) {
        CallbackDemoLib.PrefixCallback callback = new CallbackDemoLib.PrefixCallback() {
                                                        @Override
                                                        public void print(Person person) {
                                                            System.out.format("Name: %s, Gender: %s, Age: %d\n",
                                                                person.name, person.gender == 0 ? "Female" : "Male",
                                                                person.age);
                                                        }
                                                    };

        Person rich = new Person();
        rich.name = "Rich";
        rich.gender = 1;
        rich.age = 49;

        Person vigdis = new Person();
        vigdis.name = "Vigdis";
        vigdis.gender = 0;
        vigdis.age = 28;

        CallbackDemoLib.INSTANCE.prefix_name(rich, callback);
        CallbackDemoLib.INSTANCE.prefix_name(vigdis, callback);
    }
}

And the output:

Timmys-MacBook-Pro:Java-to-C z0ltan$ javac -cp "./:./jna.jar" JavaToC.java

Timmys-MacBook-Pro:Java-to-C z0ltan$ java -cp "./:./jna.jar" JavaToC
System information:
Arch: x86_64, Model: MacBookPro11,2, Memory: 16GB, CPUs: 8, Logical CPUs: 8

10 25 -100 199 0 1 1 98 99 100 
-100 0 1 1 10 25 98 99 100 199 

Perfect! Now let’s do a brief rundown on the Java code.

Explanation: The modus operandi is very similar to that used for normal interaction with native libraries.

We define an interface CallbackDemoLib that holds proxies for the native functions in the libcallbackdemo.dylib shared library. This library exposes a single function prefix_name that expects a pointer to a person struct instance, and a callback function.

To implement the callback function, we declare another interface PrefixCallback that contains a single method print. Note that this name can be any name that you desire. The only contract is that the signature of this method should match that of the callback defined in the native function.

The native prefix_name function expects a callback that takes a pointer to a person instance. In Java, this maps nicely to a Person class. This class does have to extend the JNA class Structure.

The Person class, which maps exactly onto the native person struct needs to have one overridden method getFieldOrder(). This is important because other JNA cannot determine the layout of the object to map onto the native struct. So, in this example, we take care to ensure that the field order is specified exactly in this order – name, gender, and age. What would happen if we had mixed the order around? Let’s try that and see.

First let’s swap out the name and age fields:

public static class Person extends Structure {
       public String name;
       public int gender;
       public int age;

        @Override
        public List<String> getFieldOrder() {
            return Arrays.asList(new String[] {"age", "gender", "name"});
        }
    }

And the output:

Timmys-MacBook-Pro:C-to-Java z0ltan$ java -cp "./:./jna.jar" CallbackDemo
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fff94886132, pid=1232, tid=2823
#
# JRE version: Java(TM) SE Runtime Environment (9.0+131) (build 9-ea+131)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (9-ea+131, mixed mode, tiered, compressed oops, g1 gc, bsd-amd64)
# Problematic frame:
# C  [libsystem_c.dylib+0x1132]  strlen+0x12
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/z0ltan/Rabota/Blogs/Cffi/JNA/C-to-Java/hs_err_pid1232.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Abort trap: 6

It crashed the whole darned JVM! Hmmm… before we speculate on what’s happening, let’s try another variation – swapping the gender and age fields:

public static class Person extends Structure {
       public String name;
       public int gender;
       public int age;

        @Override
        public List<String> getFieldOrder() {
            return Arrays.asList(new String[] {"name", "age", "gender"});
        }
    }

And the output:

Timmys-MacBook-Pro:C-to-Java z0ltan$ java -cp "./:./jna.jar" CallbackDemo
Name: Mr. Rich, Gender: Male, Age: 49
Name: Mr. Vigdis, Gender: Female, Age: 28

Eh? So what’s going on here? My deduction is that this is because of the memory layout that JNA needed to do to accommodate the native person struct. In the first case, we swapped out two fields of different types whereas in the second case, the swapped fields were the same size, and so it did not affect the memory layout – it was still in the same order – String, int, and int.

So it’s appears like the name of the fields don’t matter as much as the actual data types they represent (makes sense) even if JNA does do checks to ensure we can’t give names different from the actual field names. To test out this hypothesis, let’s try out one more variation and see if that crashes the JVM again – let’s swap out name and gender instead this time:

public static class Person extends Structure {
       public String name;
       public int gender;
       public int age;

        @Override
        public List<String> getFieldOrder() {
            return Arrays.asList(new String[] {"gender", "name", "age"});
        }
    }

And the output?

Timmys-MacBook-Pro:C-to-Java z0ltan$ java -cp "./:./jna.jar" CallbackDemo
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fff94886132, pid=1247, tid=2823
#
# JRE version: Java(TM) SE Runtime Environment (9.0+131) (build 9-ea+131)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (9-ea+131, mixed mode, tiered, compressed oops, g1 gc, bsd-amd64)
# Problematic frame:
# C  [libsystem_c.dylib+0x1132]  strlen+0x12
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/z0ltan/Rabota/Blogs/Cffi/JNA/C-to-Java/hs_err_pid1247.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Abort trap: 6

And just taking a peek at the crash file:

---------------  S U M M A R Y ------------

Command Line: CallbackDemo

Host: MacBookPro11,2 x86_64 2200 MHz, 8 cores, 16G, Darwin 15.6.0
Time: Wed Sep  7 10:41:53 2016 IST elapsed time: 0 seconds (0d 0h 0m 0s)

---------------  T H R E A D  ---------------

Current thread (0x00007f9e3080c000):  JavaThread "main" [_thread_in_native, id=2823, stack(0x000070000011a000,0x000070000021a000)]

Stack: [0x000070000011a000,0x000070000021a000],  sp=0x0000700000218b00,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libsystem_c.dylib+0x1132]  strlen+0x12
C  [libcallbackdemo.dylib+0xcc6]  concatenate_names+0x26
C  [libcallbackdemo.dylib+0xe28]  prefix_name+0x68
C  [jna5393012570626767002.tmp+0xe134]  ffi_call_unix64+0x4c
C  0x00007000002195c8

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  com.sun.jna.Native.invokeVoid(JI[Ljava/lang/Object;)V+0
j  com.sun.jna.Function.invoke([Ljava/lang/Object;Ljava/lang/Class;Z)Ljava/lang/Object;+29
j  com.sun.jna.Function.invoke(Ljava/lang/reflect/Method;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Object;Ljava/util/Map;)Ljava/lang/Object;+249
j  com.sun.jna.Library$Handler.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object;+348
j  com.sun.proxy.$Proxy0.prefix_name(LCallbackDemo$Person;LCallbackDemo$CallbackDemoLib$PrefixCallback;)V+20
j  CallbackDemo.main([Ljava/lang/String;)V+63
v  ~StubRoutines::call_stub

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000

Indeed – it looks like the strlen is being attempted on an int! No wonder it crashed and burnt.

However, I have a suspicion that this behaviour might again differ depending on the platform. So the takeaway here is – always use the field names (with the correct types) in the same order as in the native struct.

Conclusion

Top

The JNA library is extremely useful because it is pure Java (unlike JNI, which is hardly convenient to use). In addition, as we have seen in the last post and in the current post, the APIs for JNA are very well-designed indeed.

I would recommend exploring further using the resources mentioned in the References section of the last post.

Next up, a small mini-project as the conclusion of this interop mini-series – a less-than-trivial essay at embedding a JVM instance within Common Lisp! Stay tuned.

Interop mini-series – Calling C and C++ Callbacks from Java (Part 4)

Interop mini-series – Calling C and C++ code from Java (Part 3)

In this post and the next, I will show how we can use the JNA (Java Native Access) library to perform the same interop operations as demonstrated in the first three parts of this mini-series (check them out!).

Contents

  1. The JNA library
    1. Installation
  2. Demo
    1. Building the C library
    2. Building the C++ library
    3. Demo run
  3. References

The JNA library

The JNA (Java Native Access) library is an alternative to using JNI in order to interop with C and C++ shared libraries. While JNI is a real pain to get right, JNA abstracts away all the gory details so that the developer can focus on the functionality instead of boilerplate.

Interestingly, JNA itself uses JNI stubs internally, connecting with the libffi library. (A link has been provided in the References section). This is why the library works across multiple platforms, and is still relatively lightweight.

Installation

JNA comprises of two Jar files – jna.jar and jna-platform.jar. jna.jar provides the core APIs while jna-platform contains the aforementioned stubs to interact with libffi.

For instance, the Mac OS X specific bits are secreted away in com.sun.jna.darwin/libjnidispatch.jnilib inside jna-platform.jar. Likewise for other platforms.

You can either download the JNA jars directly and use them, or use use Maven to integrate JNA functionality in your project. Again, I have provided all the links in the References section.

Demo

Top

For this demo, we will first see how the two native libraries (which our Java code will be using) are created, and then we will use them both in the same example.

The first library, libsysteminfo.dylib uses pure C code, whereas the other library, libnumbersorting.dylib uses C++ code with a C wrapper (the reason will be explained in the relevant section).

Building the C library

Top

For the C example, I decided to use use the native library to get some useful system information – architecture type, model name, memory, number of cpus, and the number of logical cpus (cores).

Note that this example works only in Mac OS X. For Linux, the sysctlbyname function can be replaced by sysctl with appropriate changes. For Windows, you will have to check which kernel call provides the same functionality.

We will use the sysctlbyname function to extract these values.

Let’s define the header file first (in system_info.h):

#ifndef __SYSTEM_INFO_H__
#define __SYSTEM_INFO_H__ "system_info.h"

#include <sys/types.h>
#include <sys/sysctl.h>

#ifdef __cplusplus
extern "C" {
#endif

char* get_machine();
char* get_model();
int64_t get_memory();
int32_t get_ncpu();
int32_t get_nlogicalcpu();

#ifdef __cplusplus
}
#endif
#endif

And the corresponding C implementation (in system_info.c:

#include <stdio.h>
#include "system_info.h"

#define MAXSIZE 210

char* get_machine()
{
    static char machine[MAXSIZE];
    size_t len = sizeof(machine);

    sysctlbyname("hw.machine", &machine, &len, NULL, 0);

    return machine;
}

char* get_model()
{
    static char model[MAXSIZE];
    size_t len = sizeof(model);

    sysctlbyname("hw.model", &model, &len, NULL, 0);

    return model;
}

int64_t get_memory()
{
    int64_t mem;
    size_t len = sizeof(mem);

    sysctlbyname("hw.memsize", &mem, &len, NULL, 0);

    return mem;
}

int32_t get_ncpu()
{
    int32_t cpu;
    size_t len = sizeof(cpu);

    sysctlbyname("hw.ncpu", &cpu, &len, NULL, 0);

    return cpu;
}


int32_t get_nlogicalcpu()
{
    int32_t logical_cpu;
    size_t len = sizeof(logical_cpu);

    sysctlbyname("hw.logicalcpu", &logical_cpu, &len, NULL, 0);

    return logical_cpu;
}

int main(int argc, char* argv[])
{
    printf("%s, %s, %lld, %d, %d\n", 
            get_machine(),
            get_model(),
            get_memory(),
            get_ncpu(),
            get_nlogicalcpu());

    return 0;
}

Let’s compile it into a shared library (in this case, Clang + LLVM on Mac OS X. For other compilers such as gcc proper, check the relevant documentation):

Timmys-MacBook-Pro:c_demo_system_info z0ltan$ clang -dynamiclib -o libsysteminfo.lib system_info.c

Timmys-MacBook-Pro:c_demo_system_info z0ltan$ ls
libsysteminfo.lib	system_info.c		system_info.h

Building the C++ library

Top

This is the more interesting bit for more than one reason! In this example, let’s try and sort an array of integers using the native library.

Again, let’s write the interface out first (in number_sorting.h):

#ifndef __NUMBER_SORTING_H__ 
#define __NUMBER_SORTING_H__ "number_sorting.h"

void callback_function(int[], int);

extern "C" {
    void sort_numbers(int[], int);
}
#endif

Looks good, but what’s the deal with the callback_function? We’ll get to that in just a monent. For now, let’s flesh out the functionality for this interface (in number_sorting.cpp:

#include <iostream>
#include <vector>
#include <algorithm>
#include <functional>

void sort_vector(std::vector<int>&, int[], int);

void callback_function(int array[], int size)
{
    std::vector<int> vec(size);

    sort_vector(vec, array, size);

    int i = 0;
    for (std::vector<int>::const_iterator it = vec.begin(); it != vec.end(); it++)
        array[i++] = *it;
}
    

template <typename T>
void display_elements(const std::vector<T>& vec)
{
    for (std::vector<int>::const_iterator it = vec.begin(); it != vec.end(); it++)
        std::cout << *it << " ";
    std::cout << std::endl;
}


void sort_vector(std::vector<int>& v, int numbers[], int count)
{
    for (int i = 0; i < count; i++)
        v[i] = numbers[i];

    display_elements(v);

    std::sort(v.begin(), v.end(), [](int x, int y) { return x < y; });
}

int main()
{
    std::ios_base::sync_with_stdio(false);

    int sample[] = { 1, 2, 0, -1, 3, 199, 200, 110, -234, 12345 };

    callback_function(sample, sizeof(sample)/sizeof(sample[0]));

    for (int i = 0; i < (int)sizeof(sample)/sizeof(sample[0]); i++)
        std::cout << sample[i] << " ";
    std::cout << std::endl;

    return 0;
}

Hmmm, this seems a bit too convoluted for this simple example? Why all the indirection? The reason will become crystal clear once we define the corresponding C file (in number_sorting.c) as well:

#include "number_sorting.h"

void sort_numbers(int numbers[], int n)
{
    callback_function(numbers, n);
}

Explanation: The reasons why we need both number_sorting.c and number_sorting.cpp, both of which implement the same interface, number_sorting.h
are two-fold:

  1. Since we are using some C++-only features such std::vector , std::sort, and C++11 lambdas, we need to invoke them in a separate function
  2. And the more important reason – C++’s pernicious name-mangling

Now, if we had simply written the entire sorting functionality using integer arrays and sorted using with C-like constructs (say, qsort, or a manually written sorting function), we wouldn’t need all this indirection, and we could have simply written the header as:

#ifndef __NUMBER_SORTING_H__ 
#define __NUMBER_SORTING_H__ "number_sorting.h"

extern "C" {
    void sort_numbers(int[], int);
}
#endif

and provided the implementation in number_sorting.cpp alone. That would have worked out fine. However, because we use all those C++ template constructs as well as functional constructs, if we had used this same header file, we would have got a name-mangling issue, and the function would not be visible to the Java client! (note: there are tools such as JNAerator which can help resolve name mangling issues, but that is beyond the scope of this tutorial).

To get around this, we write a C wrapper (number_sorting.c) which simply invokes the C++ function callback_function defined in number_sorting.cpp. Now you may think that we could have simply embedded callback_function inside the definition of sort_numbers in the C++ file alone, but that would not work either. Check out the reference “How to mix C and C++” in the “References section” for more details.

All right, let’s compile the code and generate the shared library:

Timmys-MacBook-Pro:c++_demo_sorting z0ltan$ clang++ -std=c++11 -stdlib=libc++ -dynamiclib -o libnumbersorting.dylib number_sorting.c number_sorting.cpp
clang: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated

Timmys-MacBook-Pro:c++_demo_sorting z0ltan$ ls
libnumbersorting.dylib	number_sorting.c	number_sorting.cpp	number_sorting.h

Timmys-MacBook-Pro:c++_demo_sorting z0ltan$ nm -gU libnumbersorting.dylib 
0000000000000a10 T __Z11sort_vectorRNSt3__16vectorIiNS_9allocatorIiEEEEPii
0000000000000730 T __Z17callback_functionPii
0000000000000c00 T _main
0000000000000700 T _sort_numbers

We can also see that the function sort_numbers has not been subjected to name-mangling.

Demo Run

Top

The Java code to plug into the native libraries is surprisingly concise and simple (in JavaToC.java):

import com.sun.jna.Native;
import com.sun.jna.Platform;
import com.sun.jna.Library;

public class JavaToC {
    public interface SystemInfoLib extends Library {
        SystemInfoLib INSTANCE = (SystemInfoLib)Native.loadLibrary("libsysteminfo.dylib",
                                    SystemInfoLib.class);

        String get_machine();
        String get_model();
        long get_memory();
        int get_ncpu();
        int get_nlogicalcpu();
    }

    public static native void sort_numbers(int[] numbers, int count);

    static {
        Native.register("libnumbersorting.dylib");
    }

    public static void main(String[] args) {
        System.out.println("System information:");
        System.out.format("Arch: %s, Model: %s, Memory: %dGB, CPUs: %d, Logical CPUs: %d\n\n",
            SystemInfoLib.INSTANCE.get_machine(),
            SystemInfoLib.INSTANCE.get_model(),
            SystemInfoLib.INSTANCE.get_memory()/ (1024*1024*1024),
            SystemInfoLib.INSTANCE.get_ncpu(),
            SystemInfoLib.INSTANCE.get_nlogicalcpu());

        int[] numbers = new int[]{10, 25, -100, 199, 0, 1, 1, 98, 99, 100};
        
        sort_numbers(numbers, numbers.length);
           
        for (int n : numbers) {
            System.out.format("%d ", n);
        }
        System.out.println();
    }
}

And the output:

Timmys-MacBook-Pro:Java-to-C z0ltan$ javac -cp "./:./jna.jar" JavaToC.java

Timmys-MacBook-Pro:Java-to-C z0ltan$ java -cp "./:./jna.jar" JavaToC
System information:
Arch: x86_64, Model: MacBookPro11,2, Memory: 16GB, CPUs: 8, Logical CPUs: 8

10 25 -100 199 0 1 1 98 99 100 
-100 0 1 1 10 25 98 99 100 199 

Perfect! Now let’s do a brief rundown on the Java code.

Explanation: There are basically two main ways in which to use JNA to load the native library:

    1. Implement an interface that extends the Library interface, use Native.loadLibrary() to load the specific library for your platform. The JNA class Platform provides some helper methods and static fields for this purpose.
      You can then include the declarations for the native methods inside your interface. Note that the names of the methods must exactly match those in the native library.

 

  1. The other way (preferred if you are only interested in a few functions) is to load the native library statically using a static block and the Native.register method.
    Along with this, you also need to declare the functions of interest as static native methods. Take care to ensure that the names again match with those defined inside the native library.

In the given example, both approaches have been amply demonstrated – libsysteminfo.dylib is loaded using the first approach since there are quite a few functions that I am interested in using.

On the other hand, since there is only a single function the libnumbersorting.dylib native library, it’s more convenient to load it a static native method instead.

Pay special attention to the data types used in the declarations of the methods. Just like with cffi, the data types match quite nicely with the native types in most cases. Refer to the documentation for more specific type declarations.

In the next post, we will see how to write callback functions in Java that can be invoked by functions from inside native libraries.

References

Top

Some useful links for the JNA library:

Interop mini-series – Calling C and C++ code from Java (Part 3)

Interop mini-series – Calling C and C++ code from Common Lisp using CFFI (Part 1)

Starting with this post, the new few posts will dig into interoperability between various languages. The next couple of posts will cover C and C++ code from Common Lisp, and how to write callback functions in Common Lisp that plug into code residing in a shared library. This will make use of the cffi library.

Then the following two posts will cover the analogue in Java-world. To this end, we will make use of the JNA project to indicate interop between C/C++ and Java.

Finally, this series will (hopefully) conclude with a mini-project of sorts – a completely embedded JVM instance inside a Common Lisp image! A number of demos will illustrate different uses of embedding Java within Common Lisp. This is a bit of an undertaking though, and will definitely take some time to implement least of all due to the fact that I want to extract the maximum amount of learning from this activity!

Contents

  1. Introducing the cffi library
  2. Demos
    1. Interop with C
    2. Interop with C++
  3. Summary of useful functions
  4. References

Setup used for this tutorial series

In order to keep things sane, I will be sticking to a single platform (unless otherwise noted) with the following configuration for this whole mini-series:

  • Mac OS X El Capitan system
  • 8 cores, 16GB RAM, 1600MHz processor
  • SBCL as my Common Lisp implementation
  • JDK 9
  • Apple LLVM 6.1.0 (with Clang as the frontend) as my C and C++ compiler

Note that even though the compiler used is LLVM, the behaviour is more or less the same as that of standard gcc/g++. The same flags also work for compilation, and the only difference vis-a-vis this tutorial will be how the shared (dynamic) library is created.

Introduction to the CFFI library

Let’s install the cffi library using QuickLisp (if you haven’t done so already) first:

LISP-TO-C-USER> (ql:quickload :cffi)
To load "cffi":
  Load 4 ASDF systems:
    alexandria babel trivial-features uiop
  Install 1 Quicklisp release:
    cffi
; Fetching #<URL "http://beta.quicklisp.org/archive/cffi/2016-03-18/cffi_0.17.1.tgz">
; 234.48KB
==================================================
240,107 bytes in 0.33 seconds (712.70KB/sec)
; Loading "cffi"
[package cffi-sys]................................
[package cffi]....................................
..................................................
[package cffi-features]
(:CFFI)

Now, let’s talk a bit about this library and the features that it provides. Links to the download site and manual are provided in the “References” section.

The cffi library is a cross-platform (across Common Lisp implementations that is) library that supports interop with C (and with C++, but we’ll talk more about that later). What this means is that you can load a native library (dylib, so file, DLL, etc.) and use the functions defined therein within your Lisp code.

The interop is two-ways – the general case is that you want to invoke C functions from Lisp code, or you may want to invoke a function in the library that expects a callback, and you can define this callback as pure Common Lisp code! Nifty, isn’t it?

The library is very well-designed and personally I find the Lispy nature of the APIs (and generated functions) an extra bonus.

The best way to learn the library is to see it in action, so let’s get on with it!

Note: Platform support for different features varies according to the quirks of the specific Common Lisp implementation. Refer to the cffi documentation for specifics).

Demos

Top

These demos are aimed to be simple and small, and yet somewhat useful in terms of real-world applicability.

I personally feel that purely contrived demos are best avoided since they hardly teach anything well, most of all for the reason that they are extremely boring!

In the first demo, we will see how a C library may be loaded and run from Common Lisp. This will be the most common use case.

In the second demo, we will do the same, but for a C++ library with a C wrapper around the C++ functionality.

Since both demos are defined in the same package, let’s define the package first:

(require 'cffi)

(defpackage :lisp-to-c-user
  (:use :cl :cffi))

(in-package :lisp-to-c-user)

Interop with C

Top

For the C example, I decided to use use the native library to get some useful system information – architecture type, model name, memory, number of cpus, and the number of logical cpus (cores).

Note that this example works only in Mac OS X. For Linux, the sysctlbyname function can be replaced by sysctl with appropriate changes. For Windows, you will have to check which kernel call provides the same functionality.

We will use the sysctlbyname function to extract these values.

Let’s define the header file first (in system_info.h):

#ifndef __SYSTEM_INFO_H__
#define __SYSTEM_INFO_H__ "system_info.h"

#include <sys/types.h>
#include <sys/sysctl.h>

#ifdef __cplusplus
extern "C" {
#endif

char* get_machine();
char* get_model();
int64_t get_memory();
int32_t get_ncpu();
int32_t get_nlogicalcpu();

#ifdef __cplusplus
}
#endif
#endif

And the corresponding C implementation (in system_info.c:

#include <stdio.h>
#include "system_info.h"

#define MAXSIZE 210

char* get_machine()
{
    static char machine[MAXSIZE];
    size_t len = sizeof(machine);

    sysctlbyname("hw.machine", &machine, &len, NULL, 0);

    return machine;
}

char* get_model()
{
    static char model[MAXSIZE];
    size_t len = sizeof(model);

    sysctlbyname("hw.model", &model, &len, NULL, 0);

    return model;
}

int64_t get_memory()
{
    int64_t mem;
    size_t len = sizeof(mem);

    sysctlbyname("hw.memsize", &mem, &len, NULL, 0);

    return mem;
}

int32_t get_ncpu()
{
    int32_t cpu;
    size_t len = sizeof(cpu);

    sysctlbyname("hw.ncpu", &cpu, &len, NULL, 0);

    return cpu;
}


int32_t get_nlogicalcpu()
{
    int32_t logical_cpu;
    size_t len = sizeof(logical_cpu);

    sysctlbyname("hw.logicalcpu", &logical_cpu, &len, NULL, 0);

    return logical_cpu;
}

int main(int argc, char* argv[])
{
    printf("%s, %s, %lld, %d, %d\n", 
            get_machine(),
            get_model(),
            get_memory(),
            get_ncpu(),
            get_nlogicalcpu());

    return 0;
}

Let’s compile it into a shared library (in this case, Clang + LLVM on Mac OS X. For other compilers such as gcc proper, check the relevant documentation):

Timmys-MacBook-Pro:c_demo_system_info z0ltan$ clang -dynamiclib -o libsysteminfo.lib system_info.c

Timmys-MacBook-Pro:c_demo_system_info z0ltan$ ls
libsysteminfo.lib	system_info.c		system_info.h

Excellent! Finally, let’s write the Common Lisp client code to use this library:

;;; C-demo

(define-foreign-library libsysteminfo
  (:darwin "libsysteminfo.dylib")
  (:unix "libsysteminfo.so")
  (t (:default "libsysteminfo.dylib")))

(load-foreign-library 'libsysteminfo)

(defcfun "get_machine" :string)
(defcfun "get_model" :string)
(defcfun "get_memory" :long-long)
(defcfun "get_ncpu" :int)
(defcfun "get_nlogicalcpu" :int)

(defun print-system-info ()
  (let ((arch (get-machine))
        (model (get-model))
        (mem (/ (get-memory) (* 1024 1024 1024)))
        (ncpu (get-ncpu))
        (nlogicalcpu (get-nlogicalcpu)))
    (format t "System Information~%")
    (format t "Arch: ~a, Model: ~a, Mem = ~dGB, CPUs = ~d, Logical CPUs = ~d~%"
            arch model mem ncpu nlogicalcpu)))

(print-system-info)

(close-foreign-library 'libsysteminfo)

And the output:

LISP-TO-C-USER> (print-system-info)
System Information
Arch: x86_64, Model: MacBookPro11,2, Mem = 16GB, CPUs = 8, Logical CPUs = 8
NIL

Explanation: We define the native library by using the cffi:define-foreign-library macro. This macro also allows us to define the specific name of the shared library depending on the OS.

Then we can load the specified library using the cffi:load-foreign-library macro. Take care to observed that the name of the library is quoted. This can save you a lot of anguish later on.

The next part is interesting – we use the cffi:decfun macro to define the C functions present in the library as Lispy function. For instance, the C function “get_machine” which is defined in the libsysteminfo.dylib library, is proxied into the current Lisp image as “get-machine”. There are ways to perform such name mangling automatically, but letting the cffi library take care of this is my recommendation.

The general syntax of the defcfun macro is:

	(cffi:defcun <C-function-name> &optional<return-type> 
		<arg with types>*) 

So the first defcfun indicates that get_machine is a C function that returns a character array (represented by cffi’s local type, :string), and that it doesn’t take any parameter(s), The cffi library defines a huge set of types that map to C’s primitive, pointer, and structure types extremely well.

Now that we have create proxies for the C functions, we can invoke them as seen in the print-system-info function by passing in the appropriate return type and parameters.

Finally, we unload the native library using another macro, cffi:close-foreign-library, which also takes a quoted library representation.

Interop with C++

Top

This is the more interesting demo for more than one reason! In this example, let’s try and sort an array of integers using the native library.

Again, let’s write the interface out first (in number_sorting.h):

#ifndef __NUMBER_SORTING_H__ 
#define __NUMBER_SORTING_H__ "number_sorting.h"

void callback_function(int[], int);

extern "C" {
    void sort_numbers(int[], int);
}
#endif

Looks good, but what’s the deal with the callback_function? We’ll get to that in just a moment. For now, let’s flesh out the functionality for this interface (in number_sorting.cpp:

#include <iostream>
#include <vector>
#include <algorithm>
#include <functional>

void sort_vector(std::vector<int>&, int[], int);

void callback_function(int array[], int size)
{
    std::vector<int> vec(size);

    sort_vector(vec, array, size);

    int i = 0;
    for (std::vector<int>::const_iterator it = vec.begin(); it != vec.end(); it++)
        array[i++] = *it;
}
    

template <typename T>
void display_elements(const std::vector<T>& vec)
{
    for (std::vector<int>::const_iterator it = vec.begin(); it != vec.end(); it++)
        std::cout << *it << " ";
    std::cout << std::endl;
}


void sort_vector(std::vector<int>& v, int numbers[], int count)
{
    for (int i = 0; i < count; i++)
        v[i] = numbers[i];

    display_elements(v);

    std::sort(v.begin(), v.end(), [](int x, int y) { return x < y; });
}

int main()
{
    std::ios_base::sync_with_stdio(false);

    int sample[] = { 1, 2, 0, -1, 3, 199, 200, 110, -234, 12345 };

    callback_function(sample, sizeof(sample)/sizeof(sample[0]));

    for (int i = 0; i < (int)sizeof(sample)/sizeof(sample[0]); i++)
        std::cout << sample[i] << " ";
    std::cout << std::endl;

    return 0;
}

Hmmm, this seems a bit too convoluted for this simple example? Why all the indirection? The reason will become crystal clear once we define the corresponding C file (in number_sorting.c) as well:

#include "number_sorting.h"

void sort_numbers(int numbers[], int n)
{
    callback_function(numbers, n);
}

Explanation: The reasons why we need both number_sorting.c and number_sorting.cpp, both of which implement the same interface, number_sorting.h
are two-fold:

  1. Since we are using some C++-only features such std::vector , std::sort, and C++11 lambdas, we need to invoke them in a separate function
  2. And the more important reason – C++’s pernicious name-mangling

Now, if we had simply written the entire sorting functionality using integer arrays and sorted using with C-like constructs (say, qsort, or a manually written sorting function), we wouldn’t need all this indirection, and we could have simply written the header as:

#ifndef __NUMBER_SORTING_H__ 
#define __NUMBER_SORTING_H__ "number_sorting.h"

extern "C" {
    void sort_numbers(int[], int);
}
#endif

and provided the implementation in number_sorting.cpp alone. That would have worked out fine. However, because we use all those C++ templated constructs as well as functional constructs, if we had used this same header file, we would have got a name-mangling issue, and the function would not be visible to the Common Lisp client!

To get around this, we write a C wrapper (number_sorting.c) which simply invokes the C++ function callback_function defined in number_sorting.cpp. Now you may think that we could have simply embedded callback_function inside the definition of sort_numbers in the C++ file alone, but that would not work either. Check out the reference “How to mix C and C++” in the “References section” for more details.

All right, let’s compile the code and generate the shared library:

Timmys-MacBook-Pro:c++_demo_sorting z0ltan$ clang++ -std=c++11 -stdlib=libc++ -dynamiclib -o libnumbersorting.dylib number_sorting.c number_sorting.cpp
clang: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated

Timmys-MacBook-Pro:c++_demo_sorting z0ltan$ ls
libnumbersorting.dylib	number_sorting.c	number_sorting.cpp	number_sorting.h

Timmys-MacBook-Pro:c++_demo_sorting z0ltan$ nm -gU libnumbersorting.dylib 
0000000000000a10 T __Z11sort_vectorRNSt3__16vectorIiNS_9allocatorIiEEEEPii
0000000000000730 T __Z17callback_functionPii
0000000000000c00 T _main
0000000000000700 T _sort_numbers

We can also see that the function sort_numbers has not been subjected to name-mangling. Now that we’ve resolved that, let’s flesh out the Common Lisp client, and run the demo!

;;; C++-demo

(define-foreign-library libnumbersorting
  (:darwin "libnumbersorting.dylib")
  (:unix "libnumbersorting.so")
  (t (:default "libnumbersorting.dylib")))

(use-foreign-library libnumbersorting)

(defun sort-some-numbers (&optional (n 10))
  (with-foreign-object (numbers :int n)
    (dotimes (i n)
      (setf (mem-aref numbers :int i) (random 100)))
    (let ((before (loop for i below n
                     collect (mem-aref numbers :int i))))
      (format t "Before: ~{~d ~}~%" before))
    (foreign-funcall "sort_numbers" :pointer numbers :int n :void)
    (let ((after (loop for j below n
                    collect (mem-aref numbers :int j))))
      (format t "After: ~{~d ~}~%" after))))

(sort-some-numbers)

(close-foreign-library 'libnumbersorting)

And the output:

LISP-TO-C-USER> (sort-some-numbers 15)
Before: 52 11 18 62 39 89 2 48 48 66 73 89 73 26 97 
After: 2 11 18 26 39 48 48 52 62 66 73 73 89 89 97 
NIL

Cool!

Explanation: This demo differs only slightly from the C demo in terms of Common Lisp code. We define the native library in the same manner, but we use another macro,
use-foreign-library instead this time. This is my preferred way of loading a native library since I always forget the quoting with load-foreign-library!

Jokes aside, we can see another way of executing a function defined in a native library: cffi:foreign-funcall.

This macro has the following syntax:

	(cffi:foreign-funcall <C-function-name> &optional<args with types>* 
		<return-type>)

I tend to prefer foreign-funcall for functions with only side-effects (as in this case), and use defcfun when I need to use the function in the Common Lisp part more than once. YMMV.

The most interesting bit, of course, in the with-foreign-object macro. I won’t bother to show its general syntax, but suffice to say that this macro is used to allocate, set, and use foreign memory (i.e., from the native library) with encapsulation within its body.

In this case, we simply generate a C integer array (not the usage of the type specifier, :int), set the values of the elements of this array using cffi:mem-aref, and read the values of the array using the same accessor function.

Note that value of the var numbersis a pointer type, and also that this is available only within the body of the macro. In the next post, we will see how we can work with custom C-style structs.

Useful basic functions

Top

Here is a summarised list of the functions used in the demos in this blog post.

  • cffi:define-foreign-library
  • cffi:load-foreign-library
  • cffi:close-foreign-library
  • cffi:use-foreign-library
  • cffi:defcfun
  • cffi:foreign-funcall
  • cffi:with-foreign-object
  • cffi:mem-aref

References

Top

Some references that you might find useful on this subject matter:

Interop mini-series – Calling C and C++ code from Common Lisp using CFFI (Part 1)