Interop mini-series – Calling C and C++ Callbacks from Java (Part 4)

Continuing from the last post, I will show how we can use JNA to allow C (and C++) code to invoke callback functions implemented in pure Java.

In case you are a bit rusty on callbacks, I have a whole series of posts on that, as part of this series. Feel free to check those out, starting with Callbacks.

Let’s jump right into it then!

Contents

  1. Demo
  2. Conclusion

Demo

For this demo, let’s pick a very simple example. (This is the same example as covered in the post covering Common Lisp callbacks from C/C++ code.).

We have a person type which has the following slots/fields – name, gender, and age. From our Java code, we want to instantiate an instance of person, and then use a function in a native library, prefix_name to append either “Mr.” or “Miss” in front of the person’s name, depending on the value of the gender slot (0 for female, anything else for male).

First we define the interface for the native library (in callback_demo.h):

#ifndef __CALLBACK_DEMO_H__
#define __CALLBACK_DEMO_H__ "callback_demo.h"

typedef struct person {
    char* name;
    int gender;
    int age;
} person;

#ifdef __cplusplus
extern "C" {
#endif
    void prefix_name(person*, void (*)(person*));
#ifdef __cplusplus
}
#endif
#endif

We then write the code containing the prefix_name function that will invoke our callback function (in callback_demo.c):

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "callback_demo.h"

#define MAXSIZE 50

char* concatenate_names(const char* prefix, char* name)
{
    int len = strlen(prefix) + strlen(name) + 1;

    char* full_name = (char*)malloc(len * sizeof(char));

    if (full_name != NULL) {
        char* cp = full_name;

        while (*prefix != '\0')
            *cp++ = *prefix++;

        *cp++ = 0x20;

        while (*name != '\0')
            *cp++ = *name++;
   
         *cp = '\0';

        return full_name;
    }
   return name;
}
   

void prefix_name(person* p, void (*cb)(person*))
{
    const char* MISTER = "Mr.";
    const char* MISS = "Ms.";
    char* res = NULL;

    // 0 - female, anything else male
    res = p->gender == 0 ? concatenate_names(MISS, p->name) :
                           concatenate_names(MISTER, p->name);
    strcpy(p->name, res);
    
    (*cb)(p);
}

void sample_callback(person* p)
{
    printf("%s, %s, %d\n", p->name, p->gender == 0 ? "Female" : "Male", p->age);
}

int main()
{
    person rich;

    rich.name = (char*)malloc(MAXSIZE * sizeof(char));
    strcpy(rich.name, "Rich");
    rich.gender = 1;
    rich.age = 49;

    prefix_name(&rich, &sample_callback);
    
    return 0;
}

Explanation: The code is relatively straightforward. As can be seen from the header file, prefix_name is the entry point to the library (and the one which gets invoked from the Java code).

The prefix_name function takes an instance of the person structure as well as a callback function. Note the signature of the callback function:

void (*)(person*).

This callback function expects to be passed a modified instance of the person instance that is the first parameter of the prefix_name function.

The logic is very simple – simply check for the gender field, and then depending on whether it is 0 or some other way, update the name field of the person instance by prepending “Miss” or “Mr.” respectively.

Finally, the callback function cb is invoked, passing control back to the client code.

All right, now we compile the code into a library, libcallbackdemo.dylib:

Timmys-MacBook-Pro:Demo z0ltan$ clang -dynamiclib -o libcallbackdemo.dylib callback_demo.c

Timmys-MacBook-Pro:Demo z0ltan$ ls
callback_demo.c		callback_demo.h		libcallbackdemo.dylib

Excellent!

The Java code to plug into the native libraries surprisingly concise and simple (in CallbackDemo.java):

import com.sun.jna.Library;
import com.sun.jna.Native;
import com.sun.jna.Callback;
import com.sun.jna.Platform;
import com.sun.jna.Structure;

import java.util.List;
import java.util.Arrays;

public class CallbackDemo {
    private static final String lib;

    static {
        if (Platform.isMac())
            lib = "libcallbackdemo.dylib";
        else if (Platform.isWindows())
            lib = "libcallbackdemo.dll";
        else
            lib = "libcallbackdemo.so";
    }

    public interface CallbackDemoLib extends Library {
        CallbackDemoLib INSTANCE = (CallbackDemoLib)Native
                                    .loadLibrary(lib, CallbackDemoLib.class);

        interface PrefixCallback extends Callback {
            void print(Person person);
        }

        void prefix_name(Person person, PrefixCallback func);
    }

    public static class Person extends Structure {
       public String name;
       public int gender;
       public int age;

        @Override
        public List<String> getFieldOrder() {
            return Arrays.asList(new String[] {"name", "gender", "age"});
        }
    }

    public static void main(String[] args) {
        CallbackDemoLib.PrefixCallback callback = new CallbackDemoLib.PrefixCallback() {
                                                        @Override
                                                        public void print(Person person) {
                                                            System.out.format("Name: %s, Gender: %s, Age: %d\n",
                                                                person.name, person.gender == 0 ? "Female" : "Male",
                                                                person.age);
                                                        }
                                                    };

        Person rich = new Person();
        rich.name = "Rich";
        rich.gender = 1;
        rich.age = 49;

        Person vigdis = new Person();
        vigdis.name = "Vigdis";
        vigdis.gender = 0;
        vigdis.age = 28;

        CallbackDemoLib.INSTANCE.prefix_name(rich, callback);
        CallbackDemoLib.INSTANCE.prefix_name(vigdis, callback);
    }
}

And the output:

Timmys-MacBook-Pro:Java-to-C z0ltan$ javac -cp "./:./jna.jar" JavaToC.java

Timmys-MacBook-Pro:Java-to-C z0ltan$ java -cp "./:./jna.jar" JavaToC
System information:
Arch: x86_64, Model: MacBookPro11,2, Memory: 16GB, CPUs: 8, Logical CPUs: 8

10 25 -100 199 0 1 1 98 99 100 
-100 0 1 1 10 25 98 99 100 199 

Perfect! Now let’s do a brief rundown on the Java code.

Explanation: The modus operandi is very similar to that used for normal interaction with native libraries.

We define an interface CallbackDemoLib that holds proxies for the native functions in the libcallbackdemo.dylib shared library. This library exposes a single function prefix_name that expects a pointer to a person struct instance, and a callback function.

To implement the callback function, we declare another interface PrefixCallback that contains a single method print. Note that this name can be any name that you desire. The only contract is that the signature of this method should match that of the callback defined in the native function.

The native prefix_name function expects a callback that takes a pointer to a person instance. In Java, this maps nicely to a Person class. This class does have to extend the JNA class Structure.

The Person class, which maps exactly onto the native person struct needs to have one overridden method getFieldOrder(). This is important because other JNA cannot determine the layout of the object to map onto the native struct. So, in this example, we take care to ensure that the field order is specified exactly in this order – name, gender, and age. What would happen if we had mixed the order around? Let’s try that and see.

First let’s swap out the name and age fields:

public static class Person extends Structure {
       public String name;
       public int gender;
       public int age;

        @Override
        public List<String> getFieldOrder() {
            return Arrays.asList(new String[] {"age", "gender", "name"});
        }
    }

And the output:

Timmys-MacBook-Pro:C-to-Java z0ltan$ java -cp "./:./jna.jar" CallbackDemo
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fff94886132, pid=1232, tid=2823
#
# JRE version: Java(TM) SE Runtime Environment (9.0+131) (build 9-ea+131)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (9-ea+131, mixed mode, tiered, compressed oops, g1 gc, bsd-amd64)
# Problematic frame:
# C  [libsystem_c.dylib+0x1132]  strlen+0x12
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/z0ltan/Rabota/Blogs/Cffi/JNA/C-to-Java/hs_err_pid1232.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Abort trap: 6

It crashed the whole darned JVM! Hmmm… before we speculate on what’s happening, let’s try another variation – swapping the gender and age fields:

public static class Person extends Structure {
       public String name;
       public int gender;
       public int age;

        @Override
        public List<String> getFieldOrder() {
            return Arrays.asList(new String[] {"name", "age", "gender"});
        }
    }

And the output:

Timmys-MacBook-Pro:C-to-Java z0ltan$ java -cp "./:./jna.jar" CallbackDemo
Name: Mr. Rich, Gender: Male, Age: 49
Name: Mr. Vigdis, Gender: Female, Age: 28

Eh? So what’s going on here? My deduction is that this is because of the memory layout that JNA needed to do to accommodate the native person struct. In the first case, we swapped out two fields of different types whereas in the second case, the swapped fields were the same size, and so it did not affect the memory layout – it was still in the same order – String, int, and int.

So it’s appears like the name of the fields don’t matter as much as the actual data types they represent (makes sense) even if JNA does do checks to ensure we can’t give names different from the actual field names. To test out this hypothesis, let’s try out one more variation and see if that crashes the JVM again – let’s swap out name and gender instead this time:

public static class Person extends Structure {
       public String name;
       public int gender;
       public int age;

        @Override
        public List<String> getFieldOrder() {
            return Arrays.asList(new String[] {"gender", "name", "age"});
        }
    }

And the output?

Timmys-MacBook-Pro:C-to-Java z0ltan$ java -cp "./:./jna.jar" CallbackDemo
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fff94886132, pid=1247, tid=2823
#
# JRE version: Java(TM) SE Runtime Environment (9.0+131) (build 9-ea+131)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (9-ea+131, mixed mode, tiered, compressed oops, g1 gc, bsd-amd64)
# Problematic frame:
# C  [libsystem_c.dylib+0x1132]  strlen+0x12
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/z0ltan/Rabota/Blogs/Cffi/JNA/C-to-Java/hs_err_pid1247.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Abort trap: 6

And just taking a peek at the crash file:

---------------  S U M M A R Y ------------

Command Line: CallbackDemo

Host: MacBookPro11,2 x86_64 2200 MHz, 8 cores, 16G, Darwin 15.6.0
Time: Wed Sep  7 10:41:53 2016 IST elapsed time: 0 seconds (0d 0h 0m 0s)

---------------  T H R E A D  ---------------

Current thread (0x00007f9e3080c000):  JavaThread "main" [_thread_in_native, id=2823, stack(0x000070000011a000,0x000070000021a000)]

Stack: [0x000070000011a000,0x000070000021a000],  sp=0x0000700000218b00,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libsystem_c.dylib+0x1132]  strlen+0x12
C  [libcallbackdemo.dylib+0xcc6]  concatenate_names+0x26
C  [libcallbackdemo.dylib+0xe28]  prefix_name+0x68
C  [jna5393012570626767002.tmp+0xe134]  ffi_call_unix64+0x4c
C  0x00007000002195c8

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  com.sun.jna.Native.invokeVoid(JI[Ljava/lang/Object;)V+0
j  com.sun.jna.Function.invoke([Ljava/lang/Object;Ljava/lang/Class;Z)Ljava/lang/Object;+29
j  com.sun.jna.Function.invoke(Ljava/lang/reflect/Method;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Object;Ljava/util/Map;)Ljava/lang/Object;+249
j  com.sun.jna.Library$Handler.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object;+348
j  com.sun.proxy.$Proxy0.prefix_name(LCallbackDemo$Person;LCallbackDemo$CallbackDemoLib$PrefixCallback;)V+20
j  CallbackDemo.main([Ljava/lang/String;)V+63
v  ~StubRoutines::call_stub

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000

Indeed – it looks like the strlen is being attempted on an int! No wonder it crashed and burnt.

However, I have a suspicion that this behaviour might again differ depending on the platform. So the takeaway here is – always use the field names (with the correct types) in the same order as in the native struct.

Conclusion

Top

The JNA library is extremely useful because it is pure Java (unlike JNI, which is hardly convenient to use). In addition, as we have seen in the last post and in the current post, the APIs for JNA are very well-designed indeed.

I would recommend exploring further using the resources mentioned in the References section of the last post.

Next up, a small mini-project as the conclusion of this interop mini-series – a less-than-trivial essay at embedding a JVM instance within Common Lisp! Stay tuned.

Interop mini-series – Calling C and C++ Callbacks from Java (Part 4)

Interop mini-series – Calling C and C++ Callbacks from Common Lisp (Part 2c)

This post picks up on the first part of this interop mini-series (Calling C and C++ from Common Lisp). I recommend checking out that post first in order to make sense of this one.

Contents

  1. Intent
  2. Demo
  3. Useful functions
  4. Conclusion

Intent

The scope of this post is to cover interop with C and C++ code from Common Lisp using callbacks. In case you are not sure about what callbacks are, please check the first part of this post out – Callbacks special.

We will continue to use the cffi library for our demo here as well.

Demo

Top

For this demo, let’s pick a very simple example.

We have a person type which has the following slots/fields – name, gender, and age. From our Common Lisp code, we want to instantiate an instance of person, and then use a function in a native library, prefix_name to append either “Mr.” or “Miss” in front of the person’s name, depending on the value of the gender slot (0 for female, anything else for male).

First we define the interface for the native library (in callback_demo.h:

#ifndef __CALLBACK_DEMO_H__
#define __CALLBACK_DEMO_H__ "callback_demo.h"

typedef struct person {
    char* name;
    int gender;
    int age;
} person;

#ifdef __cplusplus
extern "C" {
#endif
    void prefix_name(person*, void (*)(person*));
#ifdef __cplusplus
}
#endif
#endif

We then write the code containing the prefix_name function that will invoke our callback function (in callback_demo.c:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "callback_demo.h"

#define MAXSIZE 50

char* concatenate_names(const char* prefix, char* name)
{
    int len = strlen(prefix) + strlen(name) + 1;

    char* full_name = (char*)malloc(len * sizeof(char));

    if (full_name != NULL) {
        char* cp = full_name;

        while (*prefix != '\0')
            *cp++ = *prefix++;

        *cp++ = 0x20;

        while (*name != '\0')
            *cp++ = *name++;
   
         *cp = '\0';

        return full_name;
    }
   return name;
}
   

void prefix_name(person* p, void (*cb)(person*))
{
    const char* MISTER = "Mr.";
    const char* MISS = "Ms.";
    char* res = NULL;

    // 0 - female, anything else male
    res = p->gender == 0 ? concatenate_names(MISS, p->name) :
                           concatenate_names(MISTER, p->name);
    strcpy(p->name, res);
    
    (*cb)(p);
}

void sample_callback(person* p)
{
    printf("%s, %s, %d\n", p->name, p->gender == 0 ? "Female" : "Male", p->age);
}

int main()
{
    person rich;

    rich.name = (char*)malloc(MAXSIZE * sizeof(char));
    strcpy(rich.name, "Rich");
    rich.gender = 1;
    rich.age = 49;

    prefix_name(&rich, &sample_callback);
    
    return 0;
}

Explanation: The code is relatively straightforward. As can be seen from the header file, prefix_name is the entry point to the library (and the one which gets invoked from the Common Lisp code).

The prefix_name function takes an instance of the person structure as well as a callback function. Note the signature of the callback function:

void (*)(person*).

This callback function expects to be passed a modified instance of the person instance that is the first parameter of the prefix_name function.

The logic is very simple – simply check for the gender field, and then depending on whether it is 0 or some other way, update the name field of the person instance by prepending “Miss” or “Mr.” respectively.

Finally, the callback function cb is invoked, passing control back to the client code.

All right, now we compile the code into a library, libcallbackdemo.dylib:

Timmys-MacBook-Pro:Demo z0ltan$ clang -dynamiclib -o libcallbackdemo.dylib callback_demo.c

Timmys-MacBook-Pro:Demo z0ltan$ ls
callback_demo.c		callback_demo.h		libcallbackdemo.dylib

Excellent!

Now we focus on the Common Lisp bit. This part is relatively straight forward. Let’s see the code in action first, and then a bit of explanation.

First the code that calls the native library function, prefix_name (in c-to-lisp.lisp):

;;;; Demonstrating how Common Lisp can invoke functions in C or C++ code, which then themselves invoke a callback function written in Common Lisp.
;;;; This helps in those cases when Common Lisp needs to make use of some 
;;;; functionality present in a native library which is written using callbacks.

(require 'cffi)

(defpackage :c-to-lisp-user
  (:use :cl :cffi))

(in-package :c-to-lisp-user)


;;; Callback demo - first define the foreign library
;;; containing the function which takes a callback function.

(define-foreign-library libcallbackdemo
  (:darwin "libcallbackdemo.dylib")
  (:unix "libcallbackdemo.so")
  (t (:default "libcallbackdemo.dylib")))

(use-foreign-library libcallbackdemo)

;;; define Common Lisp equivalent of the C structure
(defcstruct person
  (name :string)
  (gender :int)
  (age :int))


;;; define the implementation of the callback
(defcallback print-prefixed-person :void
    ((ptr (:pointer (:struct person))))
  (with-foreign-slots ((name gender age) ptr (:struct person))
    (format t "Name: ~a, Gender: ~a, Age: ~d~%"
            name
            (if (zerop gender) "Female" "male")
            age)))


;;; invoke the callback in the C library with a new instance of
;;; a person object
(defun test-callback ()
  (with-foreign-object (rich '(:struct person))
    (setf (foreign-slot-value rich '(:struct person) 'name) "Rich"
          (foreign-slot-value rich '(:struct person) 'gender) 1
          (foreign-slot-value rich '(:struct person) 'age) 49)
    (foreign-funcall "prefix_name"
                     :pointer rich
                     :pointer (callback print-prefixed-person)
                     :void))
  (with-foreign-object (vigdis '(:struct person))
    (setf (foreign-slot-value vigdis '(:struct person) 'name) "Vigdis"
          (foreign-slot-value vigdis '(:struct person) 'gender) 0
          (foreign-slot-value vigdis '(:struct person) 'age) 28)
    (foreign-funcall "prefix_name"
                     :pointer vigdis
                     :pointer (callback print-prefixed-person)
                     :void)))

;;; unload the foreign library
(close-foreign-library 'libcallbackdemo)

And the output:

C-TO-LISP-USER> (test-callback)
Name: Mr. Rich, Gender: male, Age: 49
Name: Ms. Vigdis, Gender: Female, Age: 28
; No value

Explanation: This code is also quite simple. We begin by defining the native library, and then loading it.

Next, we define the callback function using the cffi:defcallback macro. The defined callback function, print-prefixed-person uses a pointer to a person instance (which is returned by the prefix_name function inside libcallbackdemo.dylib), and so need to define the person structure first.

For that, we use another macro, cffi:defcstruct. As you can see, there is simply an exact representation of the structure defined in callback_demo.h albeit in a Lispy manner.

cffi:with-foreign-slots is a very important macro that destructures its pointer argument into the supplied slots. Note that the slot names must be the same as that provided in the person structure defined in the Common Lisp code. Note the use of cffi:foreign-slot-value instead of cffi:mem-aref as in the previous post. The rule of thumb is this – use cffi:foreign-slot-value when accessing slots, and use cffi:mem-aref when accessing atomic types.

Finally, we actually invoke the prefix_name function from test_callback. We create two instances of the person structure, and then we pass the callback function in the foreign-funcall invocation using the macro cffi:callback.

cffi:callback simply returns a pointer which is what the prefix_name function in libcallbackdemo.dylib requires. The cycle is complete!

As we can see from the output, the names are prepended with the correct suffix.

Basic useful functions

Top

Here is the summarised list of the additional functions that were used in the demo:

  • cffi:defcstruct
  • cffi:defcallback
  • cffi:with-foreign-slots
  • cffi:foreign-slot-value
  • cffi:callback

Conclusion

Top

The cffi library is a very powerful and well-designed library for dealing with native libraries. It is also quite vast, and I would most definitely recommend browsing through the official manual for further examples, and also for usage patterns for your specific needs.

Next up, I will demonstrate interop between C (and C++) and Java using the JNA library, which is far superior to the alternative of using pure JNI. That will be also be in two parts.

Interop mini-series – Calling C and C++ Callbacks from Common Lisp (Part 2c)

Interop mini-series – Callbacks special! (Common Lisp special) (Part 2b)

This is a continuation of the previous post callbacks interlude. I decided to give the section pertaining to Common Lisp its own post as I think there is some good educational value in this part itself.

We carry on from where we left off last time. We continue with the same squaring number callback example.

As a quick refresher, the idea is to implement a synchronous callback scenario. The function client invokes another function squarify which squares the passed value and invokes a callback function callback.

How it’s done in Common Lisp

Let’s start off with our first attempt to implement the solution in Common Lisp.

;;;; Callback demo using the squarify example.

(defpackage :callback-demo-user
  (:use :cl))

(in-package :callback-demo-user)

(defun callback(n)
  (format t "Received: ~d~%" n))

(defun squarify(n cb)
  (funcall cb (* n n)))

(defun client ()
  (let ((n (progn
             (princ "Enter a number: ")
             (read))))
    (squarify n #'callback)))
CALLBACK-DEMO-USER> (client)
Enter a number: 19
Received: 361
NIL

That’s the direct equivalent of all the demos shown so far. However, since Common Lisp is a functional language (albeit not as pure as, say, Scheme or Haskell), we can certainly do better!

In most Functional Programming languages, higher order functions are usually deployed to do the job. So let’s see if we can cook up something nicely functional like function composition.
Here’s a first attempt:

(defun client()
  (funcall #'(lambda (n)
               (format t "Received: ~d~%" n))
           (funcall #'(lambda (n)
                        (* n n))
                    (funcall #'(lambda ()
                                 (princ "Enter number: ")
                                 (read))))))

Which produces:

CALLBACK-DEMO-USER> (client)
Enter number: 19
Received: 361
NIL

As expected! Now, as you may know, funcall simply takes a function and some arguments (optional), and applies the function to those arguments. In this case, we simply compose them in the proper order so that the types match up: read a number -> square it -> print message.

However, let’s work our way to a generic compose function that simulates the behaviour of Haskell’s composition operator. The previous function can be improved by defining a new version that composes the three functions in the mentioned order (so as to match types):

The compose function:

(defun compose (fn gn hn)
  #'(lambda (&rest args)
      (funcall fn (funcall gn (apply hn args)))))

And the client to test it:

(defun client ()
  (funcall (compose #'(lambda (x)
                        (format t "Received: ~d~%" x))
                    #'(lambda (x)
                        (* x x))
                    #'(lambda ()
                        (princ "Enter a number: ")
                        (read)))))

And the output is the same:

CALLBACK-DEMO-USER> (client)
Enter a number: 19
Received: 361
NIL

So what’s changed? Well, taking inspiration from the nested funcall function, we defined compose to invoke the functions in the proper order – first read the number, and then square it, and then finally print it! (Remember that the functions are composed in reverse order in which they are entered).

Note that the last function invocation is done using apply instead of funcall because &rest args produces a list of arguments, and funcall does not work with that (unless the function definition takes a list itself as a parameter, but that is not the general case, and apply works very well with lists and destructures them correctly.

How can we make this generic enough though? We notice the pattern – we invoke apply on the innermost function call, but we use funcall for the rest of the function call chain. This means that we must handle two cases – if there is a single function passed in, we should simply use apply on that, and if not, we should take care to chain them up as discussed. This lends itself to a nice recursive definition as shown next.

The updated compose function:

(defun compose (&rest funcs)
  (labels ((f (funcs args)
             (if (null (cdr funcs))
                 (apply (car funcs) args)
                 (funcall (car funcs) (f (cdr funcs) args)))))
    #'(lambda (&rest args)
        (f funcs args))))
)

And the test client for it:

(defun client ()
  (funcall (compose #'(lambda (x)
                        (format t "Received: ~d~%" x))
                    #'(lambda (x)
                        (* x x))
                    #'(lambda ()
                        (princ "Enter number: ")
                        (read)))))

And the output:

CALLBACK-DEMO-USER> (client)
Enter number: 19
Received: 361
NIL

Explanation: What we do is simply generalise the three-function version of compose into a generic function. For this we, define an internal function f that takes the supplied functions and the arguments as input.

f then recursively decomposes the function applications. The base condition (stopping condition) is when there is only one function left. The (if (null (cdr funcs)) bit then takes care to return the only apply call that we need, and that is of course, applied to the args argument.

As the recursion unwinds the call stack, successive funcallS are applied at each stage. This is exactly in line with the algorithm discussed at the end of the last section.

Now we are almost home and dry! Pay special attention to the order in which the lambda equivalents of the functions are entered in the client function. They are applied in the following order – callback, squarify, and then client.

We could stop here, but there’s one more change that we can make. The current version of compose works absolutely as expected, but the intuitive order of supplying functions is the opposite of what we could expect as a user. The expected order would be, in English, “read in the number, square it, and then print out a message indicating that the number was received”.

Let’s fix that last bit for out final version of compose.

Final version of compose:

;;; final version of compose
(defun compose(&rest funcs)
  (labels ((f (funcs args)
             (if (null (cdr funcs))
                 (apply (car funcs) args)
                 (funcall (car funcs) (f (cdr funcs) args)))))
    #'(lambda (&rest args)
        (f (reverse funcs) args)))))

And the corresponding test code:

;;; test out the final version of compose
(defun client ()
  (funcall (compose #'(lambda ()
                        (princ "Enter a number: ")
                        (read))
                    #'(lambda (x)
                        (* x x))
                    #'(lambda (x)
                        (format t "Received: ~d~%" x)))))

And now let’s test out and see if it works!

CALLBACK-DEMO-USER> (client)
Enter a number: 19
Received: 361
NIL

Success!

The only difference is this line: (f (reverse funcs) args). We simply reverse the order of the received functions while passing it to the recursive function f, and the rest of the code remains exactly the same!

And, of course, this is purely functional! Sweet, ain’t it?

The compose function could be optimised in multiple ways – converting it to an iterative version for instance, but conceptually, this works exactly as advertised.

Conclusion

This post illustrates why I love Common Lisp! Even as I make my journey through the world of Common Lisp, my admiration for it only grows. If there is some feature that we would like to incorporate into the language, it can be done in a just a few lines of code! No other language truly comes close in terms of expressiveness and extensibility.

Interop mini-series – Callbacks special! (Common Lisp special) (Part 2b)

Interop mini-series – Callbacks special! (Part 2a)

This post was actually meant to be part of the previous post(Calling C and C++ from Common Lisp).

However, as I began writing the section of “callbacks”, it started growing to such an extent that I decided to give its own post with a slightly more comprehensive treatment than originally planned!

Contents

  1. What is a callback?
    1. Uses of callbacks
    2. Methods of implementation
  2. Demos
    1. How it’s done in C
    2. How it’s done in C++
    3. How it’s done in Java
    4. How its’s done in Common Lisp
    5. How it’s done in other languages
      1. JavaScript
      2. Python
  3. References

What exactly is a callback?

A callback, in essence, is simply a function that is executed by another function which has a reference of sorts to the first function. Yes, that’s really it!

Uses

One major use is to ensure proper separation of concerns.

Suppose we are writing some client code that makes use of a library, and say that our client function wishes to invoke a library function. Now, this library function executes code that might result in some form of platform specific signalling that will need to be handled in disparate ways depending on the specific signal received. The library writer could not possibly have imagined all the scenarios for such signalling when he was writing the library. So how can this work? Callbacks to the rescue!

So what the library writer did was to hold a reference to a callback function in his own function, and then his function invokes this callback function as and when the need arises (say an error condition or an OS interrupt). The callback function then takes care of all the handling and bookkeeping involved.

This callback function is, of course, expected to be supplied by the client code. This makes sense since the client has the best knowledge of its own domain. This then means that the library writer can make his code as generic as possible, leaving the specifics for the client to manage.

Another common use of callbacks is asynchronous programming. For example, suppose we have a function that needs to be activated when some specific conditions have arisen, and those conditions are decided by some other code. This is a good case to use a callback.

The current function can “register” itself with the condition-generating code, and then that code can invoke a callback in the current function’s module, which can then proceed to completion. Node, in particular, makes extensive use of this approach. The general Observer pattern is, in essence, the generalisation of a callback.

Implementation

Top

Callbacks may be implemented through various means – function pointers, function objects, or lambda abstractions. The important bit is to understand the concept and defer the specifics of the modes of implementation to the language at hand.

Callbacks can be both synchronous or asynchronous (think Node).

So much for the concept. As far as the terminology goes, it is important to remember that the callback itself is the actual function that is invoked by the function that takes the callback as the parameter. A lot of confusion arises precisely for the reason that some people tend to assume that the function taking the function parameter is the callback function. Quite contrary, as we have just surmised. The one mnemonic that always works for me is to remember that both the client function and the callback function are in the same conceptual module.

Finally, a caveat – extensive use of callbacks can lead to what is known as “callback hell” (check the Reference section – there is a whole site dedicated to it!). The rule of thumb is to use a callback only when it is absolutely needed. Otherwise, it can lead to code which is both unreadable and unmaintainable.

Demos

Top

Let’s now take a brief look at the functionality offered by callbacks is implemented in various languages. Of course, there may be different mechanisms for doing so, but I have chosen what I feel to be the idiomatic form in each language under discussion.

For all these examples, we will consider the same example – we have a function (squarify) which takes two parameters – a number and a callback function (callback). squarify simply squares the parameter, and then invokes callback with the squared value.

callback simply prints out the received value with a small message. The whole chain is triggered by another function client, which invokes squarify.

Note that all the examples here are, for the sake of simplicity, synchronous.

How it’s done in C

Top

In C and C++, we make use of function pointers like so:

#include <stdio.h>

void squarify(int, void(*)(int));
void callback(int);
void client();

int main()
{
    client();

    return 0;
}

void client()
{
    int n;
    
    printf("Enter a number: ");
    scanf("%d", &n);

    squarify(n, &callback);
}

void squarify(int n, void (*cb)(int))
{
    (*cb)(n*n);
}

void callback(int n)
{
    printf("Received %d\n", n);
}

And the output:

Timmys-MacBook-Pro:C z0ltan$ gcc -Wall -o callbackdemo callbackdemo.c
 
Timmys-MacBook-Pro:C z0ltan$ ./callbackdemo 
Enter a number: 19
Received 361

Notice how we pass the address of callback to squarify using &callback inside the client function.

How it’s done in C++

Top

The technique used in the C example (declaring the callback as a function pointer parameter to squarify and then passing it the address of callback at runtime will work just the same way in C++ as well.

However, in addition, C++ offers a whole lot more ways of achieving the same result. Let’s explore three of these in the same demo – lambda abstractions, function objects, and functors.

To this end, we use a std::function object to hold a reference to the callback in squarify. This class type is specified in the header.

The logic remains unchanged from that used in the C demo.

Note that this code only works in C++11 (or above).

//C++11 or above
#include <iostream>
#include <functional>

// Define the functor class
typedef struct {
    public:
        void operator()(int n)
        {
            std::cout << "Received: " << n << std::endl;
        }
} backcall;

void squarify(int, std::function<void(int)>);
void callback(int);
void client();

int main()
{
    client();

    return 0;
}

void client()
{
    int n;

    std::cout << "Enter a number: ";
    std::cin >> n;

    // simply pass in a lambda abstraction!
    squarify(n, [](int x) 
				{ std::cout << "Received: " 
					<< x << std::endl; 
			});
    
    // or specify a function explicitly
    squarify(n, callback);

    // or pass in a functor!
    squarify(n, backcall());
}

void squarify(int n, std::function<void(int)> cb)
{
    cb(n*n);
}

void callback(int n)
{
    std::cout << "Received: " << n << std::endl;
}

And the output:

Timmys-MacBook-Pro:C++ z0ltan$ g++ -std=c++11 -Wall -o callbackdemo callbackdemo.cpp 

Timmys-MacBook-Pro:C++ z0ltan$ ./callbackdemo 
Enter a number: 19
Received: 361
Received: 361
Received: 361

Et voila!

How it’s done in Java

Top

In Java, the situation is a bit more complicated than usual for many reasons – lack of pure function objects, extreme verboseness, lack of pure generic functions, etc.

However, the code below demonstrates how we would do it pre-Java 8 (and frankly, most code written today still follow this idiomatic approach).

import java.io.InputStreamReader;
import java.io.BufferedReader;
import java.io.IOException;

interface Callback {
    void call(int x);
}

public class CallbackDemo {
    public static void main(String[] args) {
            client(); 
    }

    public static void client() {
        int n;

        Callback cb = new Callback() {
                        @Override
                        public void call(int n) {
                            System.out.println("Received: " + n);
                        }
                    };

        try (BufferedReader reader = new BufferedReader(new InputStreamReader(System.in))) {
                System.out.print("Enter a number: ");
                n = Integer.parseInt(reader.readLine());
                squarify(n, cb);
        } catch (NumberFormatException |IOException ex) {
            ex.printStackTrace();
        }
    }

    public static void squarify(int n, Callback callback) {
        callback.call(n*n);
    }
}
Timmys-MacBook-Pro:Java z0ltan$ javac CallbackDemo.java 
Timmys-MacBook-Pro:Java z0ltan$ java -cp . CallbackDemo
Enter a number: 19
Received: 361

The code is mostly self-explanatory. To simulate function pointers/function objects, we simply make use of essentially what’s equivalent to the C++ functor (backcall) used in the previous demo.

The Callback interface declares a single abstract method called call which takes a single int parameter, and prints out a small message onto the console.

The squarify function takes an int parameter along with an instance of Callback, and then calls that instance’s call function. (On a side note, this is precisely why even C++’s functors are superior to Java’s. C++ has operator overloading, Java unfortunately does not).

Now, let’s take a look at how it would be done using Java 8 (and above). The Java 8 version is a marked improvement in terms of readability and conciseness.

Here’s the code:

import java.io.InputStreamReader;
import java.io.BufferedReader;
import java.io.IOException;

import java.util.function.Function;

public class CallbackDemo8 {
    public static void main(String[] args) {
        client();
    }

    public static void client() {
        try (BufferedReader reader = new BufferedReader(new InputStreamReader(System.in))) {
                System.out.print("Enter a number: ");
                int n = Integer.parseInt(reader.readLine());
                
                squarify(n, (x) -> { System.out.println("Received: " + x); return null; });
        } catch (NumberFormatException | IOException ex) {
            ex.printStackTrace();
        }
    }

    public static void squarify(int n, Function<Integer,Void> cb) {
        cb.apply(n*n);
    }
}


And here’s the output:


Timmys-MacBook-Pro:Java z0ltan$ java -cp . CallbackDemo8
Enter a number: 19
Received: 361

We observe a few things that differentiate it from the pre-Java 8 version:

  • The Callback interface is gone, having been replaced by the built-in Function function interface.
  • The callback function is also gone, and a lambda abstraction replaces it instead.

The lambda expression (x) -> { System.out.println(“Received: “ + x); return null; } still looks ugly with that return null; call. It is clearly redundant, but because of the way the Function functional interface is defined, this statement is mandatory.

We could fix that by creating our own functional interface like so:

@FunctionalInterface
interface Function<T> {
	void apply(T o);
}

However, it would reintroduce a custom interface in our code. So, not much gained there!

How it’s done in Common Lisp

Top

A full post containing a detailed discussion on this topic (along with the relevant demos) is available here in the next part of this series. Make sure to check that out!

How it’s done in other languages

Top

Let’s implement the same example in a few other languages for our own edification! For the sake of brevity (this post is already quite long!), we will stick to very commonly used languages - JavaScript and Python.

I feel these should be representative of most of the mainstream languages. Haskell is a bit of a different beast, but that is worthy of its own series of posts!

JavaScript

Top

Since client-side JavaScript does not provide any means of taking input in from the command line, we will use Node.js for this demo. For I/O from the console, we will use the readline module that now comes bundled with Node.

const readline = require('readline');

const stream = readline.createInterface({
                    input: process.stdin,
                    output: process.stdout
                });

function callback(n) {
    console.log("Received: " + n);
}


function squarify(n, cb) {
    cb(n*n);
}

function client() {
    stream.question("Enter a number: ", function(n) {
        squarify(n, callback);
        stream.close();
        process.stdin.destroy();
    });
}

// run!
client();

And the output:

Timmys-MacBook-Pro:JavaScript z0ltan$ node callback.js 
Enter a number: 19
Received: 361

Again, this is equivalent to the C version. We simply pass the function name (which is a reference to the function object) to the squarify function as the callback function.

However, we could do it more idiomatically using a lambda abstraction as follows:

const readline = require('readline');

const stream = readline.createInterface({
                    input: process.stdin,
                    output: process.stdout
                });

function squarify(n, cb) {
    cb(n*n);
}

function client() {
    stream.question("Enter a number: ", function(n) {
        squarify(n, function(x) {
            console.log("Received: " + x);
        });

        stream.close();
        process.stdin.destroy();
    });
}

// run!
client()

Note how the callback function has now been replaced by a a lambda abstraction that does the same operation.

And the output:

Timmys-MacBook-Pro:JavaScript z0ltan$ node callback_demo_lambda.js 
Enter a number: 19
Received: 361

Nice!

Python

Top

In Python, just like in JavaScript, the function name itself is an intrinsic reference to the function object. Functions are, after all, first-class objects in Python, and we can simply pass it around like so:

def callback(n):
    print("Received: " + str(n))

def squarify(n, cb):
    cb(n*n)

def client():
    print("Enter a number: ", end='')
    n = int(input())
    
    squarify(n, callback)

if __name__ == '__main__':
    client()

Note that the code was written in Python 3. However, it will easily work with minimal changes with Python 2.x as well.

And the output:

Timmys-MacBook-Pro:Python z0ltan$ python3 callback_demo.py 
Enter a number: 19
Received: 361

However, since Python also supports a crude form of lambda abstractions, we could rewrite the demo like so:

def squarify(n, cb):
    cb(n*n)

def client():
    print("Enter a number: ", end='')
    n = int(input())

    squarify(n, lambda x: print("Received: " + str(x)))

if __name__ == '__main__':
    client()

So now we have simply passed the callback function as a lambda abstraction to the squarify function.

And just to verify, the output:

Timmys-MacBook-Pro:Python z0ltan$ python3 callback_demo_lambda.py 
Enter a number: 19
Received: 361

So that’s all for now! Next up, callbacks in Common Lisp, and how we can write a simple function to perform function composition.

References

Top

Here are a few references related to the topics discussed in this blog post that you might find useful:

Interop mini-series – Callbacks special! (Part 2a)