Basic Concurrency and Parallelism in Common Lisp – Part 1 (Setup)


In my last post I had mentioned that I would like to start on the Functional Programming series by creating a basic functional library that would start off by implementing the (Untyped) Lambda Calculus from scratch in various languages. However, before we jump down right into that, I thought I’d post a few short tutorials on how to use concurrency and parallelism in Common Lisp. Information on this topic is quite difficult to come by on the Internet, and it would serve as a good refresher course for me as well. Please note that I am myself but a beginner in the advanced features of Common Lisp, so bear with me if I do make some silly goof-ups.

I’ll start off with a short tutorial on the required setup. After that, I will show how we can build SBCL with threading support on Mac OS X, Then I will tackle basic concurrency support in Common Lisp, and finally, I will conclude this short series with a brief tutorial on parallelism support in Common Lisp.

One fundamental problem that arises when one talks of concurrency and parallelism in Common Lisp is that these topics are not covered by the ANSI Common Lisp standard. As such, each implementation is free to choose to provide support for these vital features as well they choose.
In this series, I will cover concurrency using the wonderful (if limited) Bordeaux library and also the native SBCL threading APIs. Parallelism will be covered using the lparallel library.

I’ll be using an Mac OS X (El Capitan) system for showing these examples (as well as the demos in the next two posts in this series). However, I’ve also tested out these steps on my Windows and Ubuntu Linux VMs, so you should be well covered whichever platform you might have.

Setup

So, let’s get started. However, we need to get some basic setup details take care of first. To give you an idea of what’s coming up, you will need the following set up in order to follow this series:

  • A good Common Lisp implementation (SBCL would be my recommendation)
  • Emacs with SLIME
  • Quicklisp.
  • Bordeaux threading library (installed using Quicklisp).
  • lparallel threading library (installed using Quicklisp).

And that’s all there it to it! Let’s break it down step by step.

Which Common Lisp to install?

If you don’t have a Common Lisp implementation already installed, you can choose from a wide variety of options:

Not all of these implementations may be available on your platform, nor may all of them support threading on your platform. Take care to read the fine print!

I have worked with SBCL, LispWorks and CCL. For this series, I’ll be using my favourite Common Lisp implementation – SBCL throughout with small snippets of how things might work in other major implementations.

In case you want to use SBCL with Mac OS X, please check out my next blog post on how to bootstrap SBCL using itself (we need to build SBCL from source to enable thread support).

Emacs and SLIME

Of course, I assume that you are already familiar with and/or have configured Emacs along with SLIME. There is really no other way to actually work with Common Lisp in my opinion.
In case you are an absolute beginner and would like to try things out before committing to the arduous task of manually configuring Emacs and SLIME, I’d recommend LispBox (https://common-lisp.net/project/lispbox/). It is slightly outdated, but still works well enough (jump to the “LispBox Setup” section if you take this route).

In case you would like to setup it yourself (a great learning experience in itself), the following would be my recommended steps:

  • Install emacs for your platform. Even if your OS comes pre-installed with emacs, I’d still recommend updating to the latest one manually (some package managers are very brittle in my experience): https://www.gnu.org/software/emacs/download.html
  • Install SLIME. This is a good starting point in case you are on a non-Windows machine. If you are on a Windows machine, this is what worked for me. Of course, it usually doesn’t work straight off the box. A little tweaking will be necessary.

This video is also a well-prepared guide on how to install Emacs, SLIME, and Quicklisp. It is targeted for Windows, but it should mostly work for other platforms as well. Highly recommended in case you don’t enjoy reading reams of text!

Installing Quicklisp

Right. Now that Emacs has been installed and SLIME configured, let’s install Quicklisp. Quicklisp is a wonderful library manager created by Zach Beane. It makes installing third-party libraries almost seamless. The best part is that it handles all the libraries’ dependencies itself so you don’t have to sign up for that brand new sanatorium!

Download Quicklisp:

timmyjose@ubuntu:~/Software$ curl -O https://beta.quicklisp.org/quicklisp.lisp
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 57144  100 57144    0     0   199k      0 --:--:-- --:--:-- --:--:--  200k
timmyjose@ubuntu:~/Software$ ls
lein  #lein#  nim-0.14.2  quicklisp.lisp  racket-6.5.0.5  rust-1.9.0  
sbcl-1.3.8-x86-64-linux  sbcl-1.3.8-x86-64-linux-binary.tar  scala-2.11.8

Load up Quicklisp in SBCL (or whichever your flavour might be) and trigger the installation:

timmyjose@ubuntu:~/Software$ sbcl --load quicklisp.lisp
This is SBCL 1.3.8, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.

  ==== quicklisp quickstart 2015-01-28 loaded ====

    To continue with installation, evaluate: (quicklisp-quickstart:install)

    For installation options, evaluate: (quicklisp-quickstart:help)

* (quicklisp-quickstart:install)

; Fetching #<URL "http://beta.quicklisp.org/client/quicklisp.sexp">
; 0.82KB
==================================================
838 bytes in 0.00 seconds (818.36KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/client/2016-02-22/quicklisp.tar">
; 240.00KB
==================================================
245,760 bytes in 0.16 seconds (1518.99KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/client/2015-09-24/setup.lisp">
; 4.94KB
==================================================
5,054 bytes in 0.00 seconds (4935.55KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/asdf/2.26/asdf.lisp">
; 194.07KB
==================================================
198,729 bytes in 0.06 seconds (3289.34KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/dist/quicklisp.txt">
; 0.40KB
==================================================
408 bytes in 0.00 seconds (398.44KB/sec)
Installing dist "quicklisp" version "2016-08-25".
; Fetching #<URL "http://beta.quicklisp.org/dist/quicklisp/2016-08-25/releases.txt">
; 350.44KB
==================================================
358,854 bytes in 0.26 seconds (1347.86KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/dist/quicklisp/2016-08-25/systems.txt">
; 269.74KB
==================================================
276,212 bytes in 0.09 seconds (2900.41KB/sec)

  ==== quicklisp installed ====

    To load a system, use: (ql:quickload "system-name")

    To find systems, use: (ql:system-apropos "term")

    To load Quicklisp every time you start Lisp, use: (ql:add-to-init-file)

    For more information, see http://www.quicklisp.org/beta/

NIL

To add the initialisation code for Quicklisp to the SBCL init file (so that Quicklisp is
automatically loaded whenever SBCL is launched):

* (ql:add-to-init-file)
I will append the following lines to #P"/home/timmyjose/.sbclrc":

  ;;; The following lines added by ql:add-to-init-file:
  #-quicklisp
  (let ((quicklisp-init (merge-pathnames "quicklisp/setup.lisp"
                                         (user-homedir-pathname))))
    (when (probe-file quicklisp-init)
      (load quicklisp-init)))

Press Enter to continue.


#P"/home/timmyjose/.sbclrc"

And finally, to use Quicklisp with Emacs and SLIME:

* (ql:quickload "quicklisp-slime-helper")
To load "quicklisp-slime-helper":
  Install 3 Quicklisp releases:
    alexandria quicklisp-slime-helper slime
; Fetching #<URL "http://beta.quicklisp.org/archive/slime/2016-05-31/slime-v2.18.tgz">
; 1076.19KB

<elided messages>

slime-helper.el installed in "/home/timmyjose/quicklisp/slime-helper.el"

To use, add this to your ~/.emacs:

  (load (expand-file-name "~/quicklisp/slime-helper.el"))
  ;; Replace "sbcl" with the path to your implementation
  (setq inferior-lisp-program "sbcl")

("quicklisp-slime-helper")

And we’re done with Quicklisp!

Installing the Bordeaux Threading Library

Now that we’ve got Quicklisp installed and configured, life should be much easier. To install the Bordeaux library, simply follow the following steps:

CL-USER> (ql:quickload :bt-semaphore)
To load "bt-semaphore":
  Load 1 ASDF system:
    alexandria
  Install 2 Quicklisp releases:
    bordeaux-threads bt-semaphore
; Fetching #<URL “http://beta.quicklisp.org/archive/bordeaux-threads/2016-03-18/
bordeaux-threads-v0.8.5.tgz">
; 19.63KB
==================================================
20,105 bytes in 0.01 seconds (1308.92KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/bt-semaphore/2013-10-03/bt-semaphore-20131003-git.tgz">
; 4.09KB
==================================================
4,191 bytes in 0.00 seconds (1364.26KB/sec)
; Loading "bt-semaphore"
[package bordeaux-threads]........................
[package bt-semaphore].
(:BT-SEMAPHORE)

Installing the lparallel Parallelism library

Installing the lparallel library follows pretty much the same idiom:

CL-USER> (ql:system-apropos "lparallel")
#<SYSTEM lparallel / lparallel-20160825-git / quicklisp 2016-08-25>
#<SYSTEM lparallel-bench / lparallel-20160825-git / quicklisp 2016-08-25>
#<SYSTEM lparallel-test / lparallel-20160825-git / quicklisp 2016-08-25>
; No value
CL-USER> (ql:quickload "lparallel")
To load "lparallel":
  Load 2 ASDF systems:
    alexandria bordeaux-threads
  Install 1 Quicklisp release:
    lparallel
; Fetching #<URL “http://beta.quicklisp.org/archive/lparallel/2016-08-25/
lparallel-20160825-git.tgz">
; 76.71KB
==================================================
78,551 bytes in 0.93 seconds (82.93KB/sec)
; Loading "lparallel"
[package lparallel.util]..........................
[package lparallel.thread-util]...................
[package lparallel.raw-queue].....................
[package lparallel.cons-queue]....................
[package lparallel.vector-queue]..................
[package lparallel.queue].........................
[package lparallel.counter].......................
[package lparallel.spin-queue]....................
[package lparallel.kernel]........................
[package lparallel.kernel-util]...................
[package lparallel.promise].......................
[package lparallel.ptree].........................
[package lparallel.slet]..........................
[package lparallel.defpun]........................
[package lparallel.cognate].......................
[package lparallel]
("lparallel")

Connecting from SLIME to LispWorks or ACL

In case you’re using a commercial Common Lisp implementation like LispWorks or Allegro Common Lisp (ACL), you can still use Emacs for your development. You just need to fire up your CL implementation, start Emacs, and then connect your SLIME instance to the CL backend.

To connect to a LispWorks instance, start it up, and in the REPL, type in the following:

CL-USER 2 : 1 > (load #P"C:/Users/Timmy Jose/AppData/Roaming/.emacs.d/elpa/slime-2.18/swank-loader.lisp")
;  Loading text file C:\Users\Timmy Jose\AppData\Roaming\.emacs.d\elpa\slime-2.18\swank-loader.lisp
#P"C:/Users/Timmy Jose/AppData/Roaming/.emacs.d/elpa/slime-2.18/swank-loader.lisp”

CL-USER 4 : 2 > (swank-loader:init)
;;; Compiling file C:\Users\Timmy Jose\AppData\Roaming\.emacs.d\elpa\slime-2.18\packages.lisp ...
;;; Safety = 3, Speed = 1, Space = 1, Float = 1, Interruptible = 1
;;; Compilation speed = 1, Debug = 2, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is  on
;;; Cross referencing is on

<EXTRA MESSAGES ELIDED>

CL-USER 5 : 2 >(swank:create-server :port 9999)
;; Swank started at port: 9999.
9999

And from Emacs, invoke the following command (follow the prompts and enter the same port number as that used in the previous command):

M-x slime-connect

And now we can see that we are indeed connected to a LispWorks instance:

; SLIME 2016-04-19
CL-USER> (lisp-implementation-type)
"LispWorks Personal Edition”
CL-USER> (lisp-implementation-version)
"6.1.1"

Follow the same process for an ACL instance as well (after M-x slime-connect from Emacs):

CL-USER> (lisp-implementation-type)
"International Allegro CL Free Express Edition”
CL-USER> (lisp-implementation-version)
"10.0 [Windows] (Dec 2, 2015 16:05)"
("lisp_build 80")

(Note that SLIME can only be connected to a single backend at a time).

Next Steps

Well, I hope things went smoothly, and now you have a stonking hot Common Lisp environment to conquer the world with! On a more serious note, the hardest part should be installing and configuring Emacs itself. However, I would still recommend putting in the effort because the payoff is immense.

Next up — how to compile SBCL from source with threading support on a Mac OS X system.

Advertisements
Basic Concurrency and Parallelism in Common Lisp – Part 1 (Setup)

Speak your mind!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s