Basic Concurrency and Parallelism in Common Lisp – Part 2 (Bootstrapping a threading-capable Mac OS X SBCL instance from source)


The Problem

The main (and practically only) issue with the default binary distribution available for SBCL on Mac OS X is that it does not come with built-in support for threads:

CL-USER> (format t "~a, ~a~%" (lisp-implementation-type)
                 (lisp-implementation-version))
SBCL, 1.2.11
NILCL-USER> (member :thread-support *FEATURES*)
NIL

As we can see, the SBCL version is 1.2.11 and there is no support for threads.

Assuming that the Bordeaux library has already been installed, load it up:

CL-USER> (ql:quickload :bt-semaphore)
To load "bt-semaphore":
  Load 1 ASDF system:
    bt-semaphore
; Loading "bt-semaphore"

(:BT-SEMAPHORE)

This is a bit surprising – if there is no thread support on the platform, shouldn’t the Bordeaux library fail during loading itself? Well, it does not, and it is precisely this fact the one has to be careful about – just because a library is successfully downloaded and loaded does not mean that it is going to run as expected:

CL-USER> bt:*supports-threads-p*
NIL

The Solution

The reason why threading is off by default is that thread support on the Mac OS platform is an experimental feature in SBCL. In order to work with threads on Mac OS X, we need to build SBCL from source with thread support.

The way we do this is by bootstrapping the new compiled version of the source code using the existing SBCL installation. The following steps should make it amply clear:

  • First, we need to download the latest source for SBCL:
    Timmys-MacBook-Pro:Software z0ltan$ wget https://sourceforge.net/projects/sbcl/files/sbcl/1.3.8/sbcl-1.3.8-source.tar.bz2
    --2016-08-29 05:16:40--  https://sourceforge.net/projects/sbcl/files/sbcl/1.3.8/sbcl-1.3.8-source.tar.bz2
    Resolving sourceforge.net... 216.34.181.60
    Connecting to sourceforge.net|216.34.181.60|:443... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: https://sourceforge.net/projects/sbcl/files/sbcl/1.3.8/sbcl-1.3.8-source.tar.bz2/download [following]
    --2016-08-29 05:16:41--  https://sourceforge.net/projects/sbcl/files/sbcl/1.3.8/sbcl-1.3.8-source.tar.bz2/download
    Connecting to sourceforge.net|216.34.181.60|:443... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: http://downloads.sourceforge.net/project/sbcl/sbcl/1.3.8/sbcl-1.3.8-source.tar.bz2?r=&ts=1472428003&use_mirror=liquidtelecom [following]
    --2016-08-29 05:16:43--  http://downloads.sourceforge.net/project/sbcl/sbcl/1.3.8/sbcl-1.3.8-source.tar.bz2?r=&ts=1472428003&use_mirror=liquidtelecom
    Resolving downloads.sourceforge.net... 216.34.181.59
    Connecting to downloads.sourceforge.net|216.34.181.59|:80... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: http://liquidtelecom.dl.sourceforge.net/project/sbcl/sbcl/1.3.8/sbcl-1.3.8-source.tar.bz2 [following]
    --2016-08-29 05:16:45--  http://liquidtelecom.dl.sourceforge.net/project/sbcl/sbcl/1.3.8/sbcl-1.3.8-source.tar.bz2
    Resolving liquidtelecom.dl.sourceforge.net... 197.155.77.8
    Connecting to liquidtelecom.dl.sourceforge.net|197.155.77.8|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 5754599 (5.5M) [application/octet-stream]
    Saving to: ‘sbcl-1.3.8-source.tar.bz2’
    
    sbcl-1.3.8-source.tar.bz2                          100%[==============================================================================================================>]   5.49M   600KB/s    in 17s     
    
    2016-08-29 05:17:02 (333 KB/s) - ‘sbcl-1.3.8-source.tar.bz2’ saved [5754599/5754599]
    

    Version 1.3.8 is the latest stable version of the source code distribution. If you want to stay bleeding edge, you can download the latest development version from here – https://sourceforge.net/p/sbcl/sbcl/ci/master/tree/

  • Now we unzip, untar, and compile the source code by passing in two flags – —fancy and —with-sb-thread:
    Timmys-MacBook-Pro:Software z0ltan$ bzip2 -cd sbcl-1.3.8-source.tar.bz2 | tar xf -
    Timmys-MacBook-Pro:Software z0ltan$ ls
    ASDF				sbcl-1.2.11-x86-64-darwin	sbcl-1.3.8			sbcl-1.3.8-source.tar.bz2
    Timmys-MacBook-Pro:Software z0ltan$ 
    

    Excellent! Now we have the sbcl-1.3.8 directory which contains the source code and scripts that we need to use next.

    A brief note on the mentioned flags – when you pass in —fancy, you actually enable all extended features of SBCL. In our case, we’re only interested in threads, so we could try passing in only the —with-sb-thread flag. However, that would possibly not work (at least one my local system it didn’t!).

    The reason is that —fancy installs the extended features even on platforms where it might be unstable which is precisely the case here. So my recommendation is to pass in both flags as shown:

    Timmys-MacBook-Pro:sbcl-1.3.8 z0ltan$ sudo sh make.sh —fancy —with-sb-thread
    
    <lots of build messages - elided>
    
    The build seems to have finished successfully, including       18 (out of       18)
    contributed modules. If you would like to run more extensive tests on
    the new SBCL, you can try:
    
      cd tests && sh ./run-tests.sh
    
      (All tests should pass on x86/Linux, x86/FreeBSD4, and ppc/Darwin. On other platforms some failures are currently expected; patches welcome as always.)
    
    To build documentation:
    
      cd doc/manual && make
    
    To install SBCL (more information in INSTALL):
    
      sh install.sh
    
    //build started:  Mon Aug 29 05:38:52 IST 2016
    //build finished: Mon Aug 29 05:42:42 IST 2016
    

    The command should take a minute or two to run.

  • Finally, we can install the binaries into the default locations (which can be changed using the INSTALL_ROOT environment variable — refer to the documentation for specifics):
    Timmys-MacBook-Pro:sbcl-1.3.8 z0ltan$ sudo sh install.sh
    Password:
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//uiop.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//asdf.fasl "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-aclrepl.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-aclrepl.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-bsd-sockets.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-bsd-sockets.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-cltl2.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-cltl2.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-concurrency.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-concurrency.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-cover.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-cover.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-executable.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-executable.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-gmp.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-gmp.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-grovel.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-grovel.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-introspect.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-introspect.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-md5.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-md5.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-mpfr.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-mpfr.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-posix.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-posix.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-queue.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-queue.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-rotate-byte.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-rotate-byte.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-rt.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-rt.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-simple-streams.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-simple-streams.asd "/usr/local/lib/sbcl/contrib/"
    cp /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-sprof.fasl /Users/z0ltan/Software/sbcl-1.3.8/obj/sbcl-home/contrib//sb-sprof.asd "/usr/local/lib/sbcl/contrib/"
    
    SBCL has been installed:
     binary /usr/local/bin/sbcl
     core and contribs in /usr/local/lib/sbcl/
    
    Documentation:
     man /usr/local/share/man/man1/sbcl.1
    

Now that we have a working SBCL instance compiled from source, let’s fire up SLIME and test it out!

Checking the SBCL version:

CL-USER> (lisp-implementation-type)
"SBCL"
CL-USER> (lisp-implementation-version)
"1.3.8"

Load the Bordeaux library and check if threading support is now enabled:

CL-USER> (ql:quickload :bt-semaphore)
To load "bt-semaphore":
  Load 1 ASDF system:
    bt-semaphore
; Loading "bt-semaphore"

(:BT-SEMAPHORE)
CL-USER> bt:*supports-threads-p*
T
CL-USER> (member :thread-support *FEATURES*)
(:THREAD-SUPPORT :SWANK :QUICKLISP :ASDF-PACKAGE-SYSTEM :ASDF3.1 :ASDF3 :ASDF2
 :ASDF :OS-MACOSX :OS-UNIX :NON-BASE-CHARS-EXIST-P :ASDF-UNICODE :64-BIT
 :64-BIT-REGISTERS :ALIEN-CALLBACKS :ANSI-CL :ASH-RIGHT-VOPS :BSD
 :C-STACK-IS-CONTROL-STACK :COMMON-LISP :COMPARE-AND-SWAP-VOPS
 :COMPLEX-FLOAT-VOPS :CYCLE-COUNTER :DARWIN :DARWIN9-OR-BETTER :FLOAT-EQL-VOPS
 :FP-AND-PC-STANDARD-SAVE :GENCGC :IEEE-FLOATING-POINT :INLINE-CONSTANTS
 :INODE64 :INTEGER-EQL-VOP :LINKAGE-TABLE :LITTLE-ENDIAN
 :MACH-EXCEPTION-HANDLER :MACH-O :MEMORY-BARRIER-VOPS :MULTIPLY-HIGH-VOPS
 :OS-PROVIDES-BLKSIZE-T :OS-PROVIDES-DLADDR :OS-PROVIDES-DLOPEN
 :OS-PROVIDES-PUTWC :OS-PROVIDES-SUSECONDS-T :PACKAGE-LOCAL-NICKNAMES
 :PRECISE-ARG-COUNT-ERROR :RAW-INSTANCE-INIT-VOPS :SB-DOC :SB-EVAL :SB-LDB
 :SB-PACKAGE-LOCKS :SB-SIMD-PACK :SB-SOURCE-LOCATIONS :SB-TEST :SB-THREAD
 :SB-UNICODE :SBCL :STACK-ALLOCATABLE-CLOSURES :STACK-ALLOCATABLE-FIXED-OBJECTS
 :STACK-ALLOCATABLE-LISTS :STACK-ALLOCATABLE-VECTORS
 :STACK-GROWS-DOWNWARD-NOT-UPWARD :SYMBOL-INFO-VOPS :UD2-BREAKPOINTS :UNIX
 :UNWIND-TO-FRAME-AND-CALL-VOP :X86-64)

Brilliant! Finally, let’s test out a simple example to make sure that threads actually do work now:

CL-USER> (defun test-sbcl-thread ()
           (let ((top-level *query-io*))
             (sb-thread:make-thread
              (lambda ()
                (format top-level "Hello from ~a~%"
                        (sb-thread:thread-name sb-thread:*current-thread*))))))
TEST-SBCL-THREAD
CL-USER> (test-sbcl-thread)
Hello from NIL
#<SB-THREAD:THREAD FINISHED values: NIL {1005132733}>

and the same example using the Bordeaux library:

CL-USER> (defun test-bordeaux-thread ()
           (let ((top-level *query-io*))
             (bt:make-thread
              (lambda ()
                (format top-level "Hello from ~a~%"
                        (bt:thread-name (bt:current-thread)))))))
TEST-BORDEAUX-THREAD
CL-USER> (test-bordeaux-thread)
Hello from Anonymous thread
#<SB-THREAD:THREAD "Anonymous thread" FINISHED values: NIL {1002C1A7B3}>

Success! As you can see, it’s really rather simple to build SBCL from source (even if you simply want to upgrade). There are a few other tweaks that you can perform when installing SBCL from source whereas with a binary installation, you are pretty much stuck in terms of functionality.

One advantage of this approach is that we can even bootstrap the SBCL client if we have an existing installation of most of the major Common Lisp implementations. Again, the documentation is the final word on the matter.

Next up in this mini-series of blogs, basic concurrency in Common Lisp using SBCL threads and the Bordeaux threads library.

Advertisements
Basic Concurrency and Parallelism in Common Lisp – Part 2 (Bootstrapping a threading-capable Mac OS X SBCL instance from source)

Speak your mind!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s