Compile time evaluation in Rust macros… or not (contrast with Common Lisp)

Rust macros are a great help in reducing boilerplate as well as creating tools to perform advanced code manipulation at compile time (the nom crate comes to mind). However, I did run into its limitations (again) when I started tinkering with a small idea.

Well, the idea that I had is quite simple – create a macro that will reverse the words of a string, but defer checking of types to runtime (so that correct entries would still produce output). However, this turned out to be not so simple, again due to the fact that Rust macros apparently do not provide a way to – perform compile-time checks, or eval code during macroexpansion (like Lisp and Scheme/Racket macros do).

Here was my first shot at creating that simple macro in Rust:

use std::any::Any;
use std::io::{self, Write};

fn is_string(s: &Any) -> bool {
    s.is::<String>()
}

macro_rules! reverse_string {
    ($string: expr) => {{
                if !is_string($string) {
                    writeln!(io::stderr(), "{:?} must be a String", $string).unwrap();
                    std::process::exit(1);
                }
                                  
                let mut rev_string = String::new();

                for word in $string.split_whitespace().rev() {
                    rev_string.push_str(word);
                    rev_string.push(' ');
                }
                rev_string
    }};
}


fn main() {
    let my_string = "Hello, world. How do you do?".to_string();
    println!("{}", reverse_string!(&my_string));
}

This works as expected, of course:

Macushla:Type_Checking_Macro_Rust z0ltan$ rustc reverse.rs && ./reverse
do? you do How world. Hello,

Now, suppose I added a call to the same macro, reverse_string on an integer instead of a string, then the results are not quite what I wanted:

fn main() {
    let my_string = "Hello, world. How do you do?".to_string();
    println!("{}", reverse_string!(&my_string));

    let my_num = 99;
    println!("{}", reverse_string!(&my_num));
}

Running which gives:

Macushla:Type_Checking_Macro_Rust z0ltan$ rustc reverse.rs && ./reverse
error[E0599]: no method named `split_whitespace` found for type `&{integer}` in the current scope
  --> reverse.rs:17:37
   |
17 |                 for word in $string.split_whitespace().rev() {
   |                                     ^^^^^^^^^^^^^^^^
...
31 |     println!("{}", reverse_string!(&my_num));
   |                    ------------------------ in this macro invocation

error: aborting due to previous error(s)

And what I really wanted was to see output for the string argument, and then the nice error message that I generate inside the macro. So what’s going on? Why can’t I generate the template, so to speak, of the macroexpansion and defer the actual error checking to runtime? Let’s expand the macro and take a peek:

Macushla:Type_Checking_Macro_Rust z0ltan$ rustc -Z unstable-options --pretty expanded reverse.rs

This produces a ton of output, but the relevant part of the expanded macro is here:

 let my_num = 99;
    ::io::_print(::std::fmt::Arguments::new_v1(
        {
            static __STATIC_FMTSTR: &'static [&'static str] = &["", "\n"];
            __STATIC_FMTSTR
        },
        &match (&{
            if !is_string(&my_num) {
                io::stderr()
                    .write_fmt(::std::fmt::Arguments::new_v1(
                        {
                            static __STATIC_FMTSTR: &'static [&'static str] =
                                &["", " must be a String\n"];
                            __STATIC_FMTSTR
                        },
                        &match (&&my_num,) {
                            (__arg0,) => {
                                [::std::fmt::ArgumentV1::new(__arg0, ::std::fmt::Debug::fmt)]
                            }
                        },
                    ))
                    .unwrap();
                std::process::exit(1);
            }
            let mut rev_string = String::new();
            for word in &my_num.split_whitespace().rev() {
                rev_string.push_str(word);
                rev_string.push(' ');
            }
            rev_string
        },) {
            (__arg0,) => {
                [
                    ::std::fmt::ArgumentV1::new(__arg0, ::std::fmt::Display::fmt),
                ]
            }
        },
    ));

These, of course, correspond to the fully expanded form of the following two lines:

    let my_num = 99;
    println!("{}", reverse_string!(&my_num));

Now we begin to see why the code doesn’t work as expected. Here’s how it works –
macroexpansion happens as part of the overall compilation phase. During this time, the Rust Type Checker is still very much active (so we cannot inject arbitrary code that doesn’t satisfy the Type Checker or the Borrow Checker). Now, Rust doesn’t really have a way to “escape” or defer the actual checking till runtime. This is as much due to the Type Checker as to the fact that the Rust macro system does not provide such means (as Lisp or Scheme/Racket macros do).

So, in this case, the Type Checker sees this snippet: for word in &my_num.split_whitespace().rev(), realises that we are trying to call split_whitespace on an i32 variable, and immediately stops with a compilation error.

The other part (though not directly relevant here) is that all the defensive error checks using if !is_string(...) wouldn’t really work even if we were to try to check that at compile time, since Rust macros do not have, as far as I know, any way of doing compile-time conditional checking.

So, at this point I just stopped with the Rust version. Now, let’s try and implement the same macro using Common Lisp:

(defmacro reverse-string (x)
  "reverse the words of the string, error checking done at runtime"
  `(if (not (stringp ,x))
       (error "~a must be a string, not a ~a~%" ,x (type-of ,x))
       ,(let ((collect (gensym))
              (lst (gensym))
              (f (gensym))
              (s (gensym)))
             `(labels ((,collect (,lst)
                         (reduce (lambda (,f ,s)
                                   (concatenate 'string ,f " " ,s)) ,lst)))
                (,collect (reverse (loop for i = 0 then (1+ j)
                                      as j = (position #\Space ,x :start i)
                                      collect (subseq ,x i j)
                                      while j)))))))
(defun main ()
  (let ((s "Hello world")
         (d 99))
    (format t "~a reversed is ~a~%" s (reverse-string s))
    (format t "~a reversed is ~a~%" d (reverse-string d))))


(defun view-macro-expansion (form)
  "helper function to display the macro-expanded form for the
   supplied form"
  (macroexpand form))

The only point of interest is the reverse-string macro. It’s pretty much the same logic as in the attempted Rust macro – create a template that checks, at runtime, whether the supplied argument is a string, and if not, generate a proper error message. If indeed the argument is a string, then reverse the words of the original string – this is the bit being done inside the loop macro.

The interesting bit is that the Lisp distro that I use – SBCL, does do rigorous compile-time analysis, and actual gives plenty of notice that it’s deleting redundant code (corresponding to the actual call in main, (format t "~a reversed is ~a~%" d (reverse-string d)) which the compiler realises will never actually be executed). However, the expanded macro has the relevant checks, and the relevant call itself is preserved so that the macro behaves exactly as desired:

CL-USER> (main)
Hello world reversed is world Hello

and, in the Lisp debugger,

99 must be a string, not a (INTEGER 0 4611686018427387903)
   [Condition of type SIMPLE-ERROR]

Restarts:
 0: [RETRY] Retry SLIME REPL evaluation request.
 1: [*ABORT] Return to SLIME's top level.
 2: [REMOVE-FD-HANDLER] Remove #<SB-IMPL::HANDLER INPUT on descriptor 14: #<CLOSURE (LABELS SWANK/SBCL::RUN :IN SWANK/BACKEND:ADD-FD-HANDLER) {1002F80ADB}>>
 3: [ABORT] Exit debugger, returning to top level.

Backtrace:
  0: (MAIN)
  1: (SB-INT:SIMPLE-EVAL-IN-LEXENV (MAIN) #<NULL-LEXENV>)
  2: (EVAL (MAIN))

Excellent! And here is how the expanded form of the macro call actually looks like:

CL-USER> (view-macro-expansion '(reverse-string 99))
(IF (NOT (STRINGP 99))
    (ERROR "~a must be a string, not a ~a~%" 99 (TYPE-OF 99))
    (LABELS ((#:G602 (#:G603)
               (REDUCE
                (LAMBDA (#:G604 #:G605)
                  (CONCATENATE 'STRING #:G604 " " #:G605))
                #:G603)))
      (#:G602
       (REVERSE
        (LOOP FOR I = 0 THEN (1+ J) AS J = (POSITION #\  99 :START I)
              COLLECT (SUBSEQ 99 I J)
              WHILE J)))))
T

Of course, I am being a bit unduly harsh on Rust here because Common Lisp, despite all vendor-specific quirks, is still pretty much a dynamic language, so we reasonably expect it to defer most type checking to runtime. In the case of Rust, it is a very strongly-typed static language, so it can ill afford to leave a lot of checking to runtime especially since it is hardly expected to have a runtime to carry out those checks (even though Rust does have a runtime, I suspect it’s quite lightweight). In any case, an interesting little experiment.

Advertisements
Compile time evaluation in Rust macros… or not (contrast with Common Lisp)

How to build OpenJDK 9 on macOS Sierra

Continuing with my research into Programming Languages (and Compilers in particular), I started looking through the (considerably abstruse) documentation on the OpenJDK site. My patience paid off in the end and I ended up getting a lot of useful information on what I was looking for in particular – the javac source code and documentation. One of my earmarked projects for this year is to modify javac to allow some for extra syntax, and since this involves adding additional semantics, I was keen to download and build OpenJDK for myself.

The steps are surprisingly very well documented and easy to follow. To give a personal perspective on the build process (including a gotcha regarding JDK9), here is the list of steps to download OpenJDK 9 and build the whole system from scratch (for experimentation/contributing to OpenJDK, what have you). Note that the latter steps are specifically for macOS. However, most of the steps are more or less the same across platforms):

  1. First, install Mercurial if you already don’t have it. This was a bit irksome for me at first (being primarily a Git user). If you have Homebrew installed, then it is as trivial as:
    $ brew install hg

    Then, ensure that you have the following settings in your ~/.hgrc file. In case you don’t have a .hgrc file, create one like so:
    $ touch ~/.hgrc

    Add the following lines to the ~/.hgrc file:

    [settings]
    fetch = 
    mq = 
    
  2. Now, clone the OpenJDK 9 repository (for some reason, hg tclone did not work for me (as specified in the documentation), but the following did steps worked just fine:
     $ hg clone http://hg.openjdk.java.net/jdk9/dev 9dev
     $ cd 9dev
     $ sh ./get_source.sh
     

    The last step will retrieve all the necessary code and build environment into your local repository, so this does take some time.

  3. Now, one caveat here – to build OpenJDK, you need an existing JDK installation (the documentation mentions that any standard variant will do). This is called the “boot JDK”. However, I found that the build does not go through with JDK 9 (from the error messages, it looks like Reflection due to the new modules system is the main culprit). In that case, you can use an older version (say, JDK 8) to perform the build instead.

    In the case of macOS, you can use the following command to find the available JDKs on your machine:

     Macushla:RacketMacros z0ltan$ /usr/libexec/java_home -V
     Matching Java Virtual Machines (2):
      9, x86_64:	"Java SE 9-ea"	/Library/Java/JavaVirtualMachines/jdk-
      9.jdk/Contents/Home
      1.8.0_112, x86_64:	"Java SE 8"	
      /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home
    
      /Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home
     

    The last line shows the current default JDK. Before proceeding with the build, you can either change the default version, or you can simply pass in a different “boot JDK” to the configuration step (preferable).

    Now, let’s generate the configuration file:

    bash configure --with-boot-jdk /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home/ --disable-warnings-as-errors
    

    Note that I pass in two flags to bash configure : --with-boot-jdk specifying the JDK that I want to use to build OpenJDK, and --disable-warnings-as-errors (this is required on macOS).

    If this command runs successfully, the build is almost certainly going to succeed.

  4. Finally, trigger the build (this should take a bit of time again):
    $ make images
    

    This will dump the generated OpenJDK images into the build directory.

  5. You can test that the generated image is valid:
    Macushla:9dev z0ltan$ ./build/macosx-x86_64-normal-server-release/jdk/bin/java -version
    openjdk version "9-internal"
    OpenJDK Runtime Environment (build 9-internal+0-adhoc.z0ltan.9dev)
    OpenJDK 64-Bit Server VM (build 9-internal+0-adhoc.z0ltan.9dev, mixed mode)
    

    And there you go. All done in under half an hour!

Note that there is a plethora of build options and flags (including cross-compilation capabilities), and you should definitely check out the two HTML pages in the following location of your repository – ./common/doc/building.html, and ./common/doc/testing.html.

Now the real work begins! 😀

How to build OpenJDK 9 on macOS Sierra

Mutual Recursion demo in Rust and Racket (inspired by Haskell)

This is a quick post on a Rust version of the Haskell evens and odds program demonstrating mutual recursion (as shown in Hutton’s book).

First off, the Haskell code:

  evens :: [a] -> [a]
  evens [] = []
  evens (x:xs) = x : odds xs

  odds :: [a] -> [a]
  odds [] = []
  odds (_:xs) = evens xs

A sample run:

*Main> let string = "abcde"
*Main> evens string
"ace"
*Main> odds string
"bd"
*Main> string
"abcde"

So the whole ideas is to have the evens functions display all the characters in even positions (starting from 0), and then odds function likewise display all the characters in odd positions.

The evens function acts as the actual accumulator whilst odds is only used as a trampoline for continuing the recursion.

Now, for a rough Rust version of it (preserving the original character array):


fn main() {
    fn evens<T: Copy>(xs: &[T]) -> Vec<T> {
        if xs.is_empty() {
            Vec::new()
        } else {
            cons(&xs[0], &odds(&xs[1..]))
        }
    }

    fn odds<T: Copy>(xs: &[T]) -> Vec<T> {
        if xs.is_empty() {
            Vec::new()
        } else {
            evens(&xs[1..])
        }
    }

    fn cons<T: Copy>(x: &T, xs: &[T]) -> Vec<T> {
        let mut vec = Vec::new();

        vec.push(*x);

        for e in xs.iter() {
            vec.push(*e);
        }
        vec
    }

    let string = String::from("abcde");

    println!("{}",
             String::from_utf8(evens(&string.clone().into_bytes())).unwrap());
    println!("{}",
             String::from_utf8(odds(&string.clone().into_bytes())).unwrap());

    println!("{}", string);
}

And a quick run:

Macushla:EvensAndOdds z0ltan$ rustc evens_and_odds.rs
Macushla:EvensAndOdds z0ltan$ ./evens_and_odds
ace
bd
abcde

So, as can be clearly seen, the original string is left unmodified. Of course this version looks quite dirty, but the nice bit is that &[T] accepts parameters of type Vec (or reference variants) and vice-versa. This enables using slicing extensively and naturally inside the functions. The vector copying code could, of course, be made to work with an API call, but I feel this is much better in its explicit form.

The Racket version looks much nicer, being more amenable to functional constructs than Rust:

#lang racket

(define (evens xs)
  (if (null? xs)
      '()
      (cons (car xs) (odds (cdr xs)))))

(define (odds xs)
  (if (null? xs)
      '()
      (evens (cdr xs))))

(define (main)
  (let ([string "abcde"])
    (displayln (list->string (evens (string->list string))))
    (displayln (list->string (odds (string->list string))))
    (displayln string)))

And a final run for the Racket version:

evens-and-odds.rkt> (main)
ace
bd
abcde
Mutual Recursion demo in Rust and Racket (inspired by Haskell)

A bit of play with Rust macros

Here are a couple of macros that I wrote up on a slow evening. The first one provides a nice literal way of creating maps, and the second one mimicks Haskell’s list comprehension syntax in a rather crude manner.

My aim originally had been to:

  • have a simple if-then-else structure created in Rust, something like so:
    fn main() {
      let res = if 2 > 3 then "Yes" else "No";
    }
    

    However, this did not appear to be possible with the current macro support in Rust since a fragment of type expr (expression) can be followed only by the following sigils –=> ; ,. So much for that.

  • and secondly, to be able to replicate Haskell’s list comprehension syntax in its entirety, allowing for any possible generator. However, I then realised that Rust’s macro system also did not support actual evaluation of code during expansion like Common Lisp or Scheme/Racket does, even though it does work on the actual AST and not merely text. So out that went through the window as well.

So here’s the first macro demo:

use std::collections::HashMap;

macro_rules! my_map {
    ($($key: expr => $val: expr)*) => {
        {
            let mut map = HashMap::new();

            $(
                map.insert($key, $val);
            )*

          map
        }
    };
}

fn main() {
    let my_map = my_map!{
        1 => "One"
        2 => "Two"
        3 => "Three"
        4 => "Four"
        5 => "Five"
    };

    println!("{:?}", my_map);
}

Nothing fancy here, but you must say that it does look nicer, almost Ruby-esque! Here’s the code run:

Macushla:List_Comprehension_Macro z0ltan$ ./map
{2: "Two", 1: "One", 3: "Three", 5: "Five", 4: "Four"}

And here’s the poor man’s list comprehension that is not only limited in scope, but also entirely inflexible in its syntax (I just couldn’t be bothered tinkering with it for little return):

macro_rules! compr {
    ($id1: ident | $id2: ident <- [$start: expr ; $end: expr] , $cond: expr) => {
        {
            let mut vec = Vec::new();

            for num in $start..$end + 1 {
                if $cond(num) {
                    vec.push(num);
                }
            }

            vec
        }  
    };
}

fn even(x: i32) -> bool {
    x%2 == 0
}

fn odd(x: i32) -> bool {
    x%2 != 0
}

fn main() {
    let evens = compr![x | x <- [1;10], even];
    println!("{:?}", evens);

    let odds = compr![y | y <- [1;10], odd];
    println!("{:?}", odds);
}

As you can see, the ident fragments are completely for show, I cannot use .. in the generator (again due to restrictions on what can follow a expr fragment), and the guards don’t even check against the supposed identifier that’s supposed to be collecting the results into the final list/vector! Anyway, here’s the code run output:

Macushla:List_Comprehension_Macro z0ltan$ rustc list_compr.rs
Macushla:List_Comprehension_Macro z0ltan$ ./list_compr
[2, 4, 6, 8, 10]
[1, 3, 5, 7, 9]

Not too shabby, eh? In all seriousness though, while Rust’s macros are a big improvement over the debacle that is C’s macro system, the surface similarities to Racket’s powerful macro system ends right there – on the surface. As it stands now, there are far too many restrictions on the macro system for it to be considered a viable way of extending the syntax of the language, or even creating new languages (as in the Racket world).

EDIT: The list comprehension macro can actually be improved by using tt for the start and end of the range – this also allows .. to be used much in the Haskell way. Moreover, now we can actually simulate the exact Haskell syntax (for this very specific case, of course), and also make use of the identS as well.

Here’s the new code, but I’m keeping the old code up to remind myself that I can be a doddering idiot at times!

Updated macro:

macro_rules! compr {
    ($id1:ident | $id2:ident <- [$start:tt..$end:tt] , $cond:tt $id3:ident) => {
        {
            let mut vec = Vec::new();
 
            for $id1 in $start..$end + 1 {
                if $cond($id3) {
                    vec.push($id2);
                }
            }
 
            vec
        }  
    };
}
 
fn even(x: i32) -> bool {
    x%2 == 0
}
 
fn odd(x: i32) -> bool {
    x%2 != 0
}
 
fn main() {
    let evens = compr![x | x <- [1..10], even x];
    println!("{:?}", evens);
 
    let odds = compr![y | y <- [1..10], odd y];
    println!("{:?}", odds);
}

And a run just to make sure it’s working:

Macushla:List_Comprehension_Macro z0ltan$ rustc list_compr.rs
Macushla:List_Comprehension_Macro z0ltan$ ./list_compr
[2, 4, 6, 8, 10]
[1, 3, 5, 7, 9]

That’s much better!

A bit of play with Rust macros

The Observer pattern in Rust

Here is a simple implementation of the Observer pattern in Rust. First, the code, and then a small bit of explanation of the implementation itself.

The project structure:

Macushla:rust-observer z0ltan$ tree
.
├── Cargo.lock
├── Cargo.toml
└── src
    ├── lib.rs
    └── main.rs

1 directory, 4 files

The client code:

extern crate rust_observer;

use rust_observer::*;

fn main() {
    let mut foo = EvenCounter::new();
    let (bar, baz, quux) = (Box::new(EvenObserver::new("bar".to_string())),
                            Box::new(EvenObserver::new("baz".to_string())),
                            Box::new(EvenObserver::new("quux".to_string())));

    foo.register(bar);
    foo.register(baz);
    foo.register(quux);

    foo.run();
}

Nothing special here – just creating an Observable object of type EvenCounter, and then three instances of EvenObserver, which implements the Observer type.

The idea is to have the observers get notified whenever the observable object’s counter is an even number (it runs the counter in an infinite loop).

And finally, the actual implementation:

/// the observable type
pub trait Observable<T> {
    fn register(&mut self, observer: Box<Observer<Item = T>>);
}

pub trait Observer {
    type Item;

    fn notify(&self, val: &Self::Item);
}


/// the actual structs which implement the Observable and Observer
/// traits

/// the specific Observable
pub struct EvenCounter {
    counter: u32,
    observers: Vec<Box<Observer<Item = u32>>>,
}

impl EvenCounter {
    pub fn new() -> Self {
        EvenCounter {
            counter: 0u32,
            observers: Vec::new(),
        }
    }

    pub fn run(&mut self) {
        loop {
            use std::thread;
            use std::time::Duration;

            thread::sleep(Duration::from_millis(self.get_rand_duration()));

            self.counter += 1;

            if self.counter % 2 == 0 {
                for observer in self.observers.iter() {
                    observer.notify(&self.counter);
                }
            }
        }
    }

    fn get_rand_duration(&self) -> u64 {
        if cfg!(target_os = "windows") {
            500u64
        } else {
            use std::process::Command;
            use std::str::FromStr;

            let rand_cmd = Command::new("sh")
                .arg("-c")
                .arg("echo $(( $RANDOM%1000 + 1 ))")
                .output()
                .expect("failed to get OS RNG");

            u64::from_str(&String::from_utf8(rand_cmd.stdout).unwrap().trim()).unwrap()
        }
    }
}

impl Observable<u32> for EvenCounter {
    fn register(&mut self, observer: Box<Observer<Item = u32>>) {
        self.observers.push(observer);
    }
}

/// the specific Observer type
pub struct EvenObserver {
    name: String,
}

impl EvenObserver {
    pub fn new(name: String) -> Self {
        EvenObserver { name: name }
    }

    fn name(&self) -> &str {
        &self.name
    }
}

impl Observer for EvenObserver {
    type Item = u32;

    fn notify(&self, val: &Self::Item) {
        println!("{} got {}", self.name(), val);
    }
}

As you can see, the types themselves are implemented as traits, and then the specific concrete types used for the example implement these traits.

The Observable type is quite simple – it is generic over the type that its observers can get notified about. Registration of observers is done through the required method, register. Also note the Box type being used here instead of a real trait object. Sometimes it’s much easier just to box things up than bothering about lifetimes and such.

The Observer type is more interesting. It has a single method, notify which the observable object uses to call the observers. The type of value that is retrieved from the observable object is made an “Associate Type” of the Observer trait. I find this to be a much cleaner approach than defining the trait itself to be parameterized over a generic type.

A simple test run:

Macushla:rust-observer z0ltan$ cargo run
   Compiling rust_observer v0.1.0 (file:///Users/z0ltan/Projects/Playground/DesignPatterns/Observer/Rust/rust-observer)
    Finished dev [unoptimized + debuginfo] target(s) in 0.61 secs
     Running `target/debug/rust_observer`
bar got 2
baz got 2
quux got 2
bar got 4
baz got 4
quux got 4
bar got 6
baz got 6
quux got 6
bar got 8
baz got 8
quux got 8
bar got 10
baz got 10
quux got 10
bar got 12
baz got 12
quux got 12
bar got 14
baz got 14
quux got 14
^C

Very satisfying indeed!

The Observer pattern in Rust

A visitor pattern implementation in Rust

I have recently been studying the ANTLR parser-generator framework, and one of its core approaches is separating the core parsing logic from implementation details. It achieves this using two major patterns – the Listener pattern and the Visitor pattern. This is easily achieved in an Object-oriented language like Java. I was curious to see how that would translate into a language like Rust, hence this post.

Rust, while not quite being an Object-oriented language, does support static and dynamic dispatch. In this case, we are interested in dynamic dispatch, and “Trait Objects” are the main way in which Rust support this feature. However, one problem with using trait objects is that the runtime really doesn’t directly provide us with any way of retrieving the specific type at runtime. This means that calling any struct specific methods on the fly isn’t really that straightforward. The other problem (specific to the Visitor pattern) is that Rust doesn’t have the notion of method overloading. This means that the only sane approach to implementing the pattern in Rust would be to pattern match against the trait object and invoke the appropriate logic, which leads us back to the first point.

A bit of Googling leads us to a practical (if not so elegant) approach of determining the concrete type that implemented the trait at runtime by using std::any::Any. This approach entails providing a method that returns a trait object of type Any for each concrete type that implements the trait, and also explicitly checking the concrete type during pattern matching – there really is no way of extracting the concrete type directly. Instead, we have to try downcasting the trait object to each concrete type and checking if we have a match or not. While this approach works fine for our custom types (the types and count of which we know), this clearly wouldn’t work in a scenario where we are working with third-party types.

In any case, here is the code implementing a simple Visitor pattern where we visit each component of a House type, and perform some custom processing independent of the actual data structures.

Here is the overall layout of the code:

Macushla:rust_visitor z0ltan$ tree
.
├── Cargo.lock
├── Cargo.toml
└── src
    ├── lib.rs
    ├── main.rs
    └── visitors.rs

1 directory, 5 files

Here is the client code, main.rs:

extern crate rust_visitor;

use rust_visitor::*;

fn main() {
    let house = House::new();

    // simply print out the house elements
    house.accept(&visitors::HouseElementListVisitor::new());
   
    println!();
     
    // do something with the elements of a house
    house.accept(&visitors::HouseElementDemolishVisitor::new());
}

So we can see that we create the House element, and then pass in two different implementations of the HouseElementVisitor trait.

Here is the code defining the components of a House:

use std::any::Any;

pub mod visitors;

use visitors::HouseElementVisitor;

/// This represents and element of a house.
/// Each type that implements this trait
/// supports being processed by a visitor.
pub trait HouseElement {
    fn accept(&self, visitor: &HouseElementVisitor);
    fn as_any(&self) -> &Any;
}


/// Define the house entity and its elements

pub struct House {
    components: Vec<Box<HouseElement>>,
}

impl House {
    pub fn new() -> Self {
        House {
            components: vec![Box::new(Livingroom::new()), Box::new(Bedroom::new()), 
                            Box::new(Bathroom::new()), Box::new(Kitchen::new())],
        }
    }
}

impl HouseElement for House {
    fn accept(&self, visitor: &HouseElementVisitor) {
        for component in self.components.iter() {
            component.accept(visitor);
        }

        visitor.visit(self);
    }

    fn as_any(&self) -> &Any { self }
}

struct Livingroom {}

impl Livingroom {
    fn new() -> Self {
        Livingroom{}
    }
}

impl HouseElement for Livingroom {
    fn accept(&self, visitor: &HouseElementVisitor) {
        visitor.visit(self);
    }

    fn as_any(&self) -> &Any { self }
}


struct Bedroom {}

impl Bedroom {
    fn new() -> Self {
        Bedroom {}
    }
}

impl HouseElement for Bedroom {
    fn accept(&self, visitor: &HouseElementVisitor) {
        visitor.visit(self);
    }

    fn as_any(&self) -> &Any { self }
}


struct Bathroom {}

impl Bathroom {
    fn new() -> Self {
        Bathroom {}
    }
}

impl HouseElement for Bathroom {
    fn accept(&self, visitor: &HouseElementVisitor) {
        visitor.visit(self);
    }

    fn as_any(&self) -> &Any { self }
}


pub struct Kitchen {}

impl Kitchen {
    fn new() -> Self {
        Kitchen {}
    }
}

impl HouseElement for Kitchen {
    fn accept(&self, visitor: &HouseElementVisitor) {
        visitor.visit(self);
    }

    fn as_any(&self) -> &Any { self }
}

As can be seen, each implementation of the HouseElement trait defines a method, as_any that returns self as the specific type of the trait object, &Any. This will help the visitor implementation match and find the actual concrete type to do the processing specific to that visitor and that particular HouseElement component.

Finally, here is the actual implementation of the visitors:

//! This module defines the `HouseElementVisitor` trait which defines the capabilities
//! of a type to process the components of a `HouseElement` entity, and also defines
//! a couple of visitor types that do completely different processing.

use ::*;

pub trait HouseElementVisitor {
    fn visit(&self, element: &HouseElement);
}

pub struct HouseElementListVisitor {}

impl HouseElementListVisitor {
    pub fn new() -> Self {
        HouseElementListVisitor {}
    }
}

impl HouseElementVisitor for  HouseElementListVisitor {
    fn visit(&self, element: &HouseElement) {
        if element.as_any()
            .downcast_ref::<House>()
            .is_some() {
            println!("Visiting the House...");
        }

        if element.as_any()
            .downcast_ref::<Livingroom>()
            .is_some() {
            println!("Visiting the Living Room...");
        }

        if element.as_any()
            .downcast_ref::<Bedroom>()
            .is_some() {
            println!("Visiting the bedroom...");
        }

        if element.as_any()
            .downcast_ref::<Kitchen>()
            .is_some() {
                println!("Visiting the kitchen...");
        }
    }
}


pub struct HouseElementDemolishVisitor {}

impl HouseElementDemolishVisitor {
    pub fn new() -> Self {
        HouseElementDemolishVisitor {}
    }
}

impl HouseElementVisitor for HouseElementDemolishVisitor {
    fn visit(&self, element: &HouseElement) {
        if element.as_any()
            .downcast_ref::<House>()
            .is_some() {
            println!("Annihilating the House...!!!");
        }

        if element.as_any()
            .downcast_ref::<Livingroom>()
            .is_some() {
            println!("Bombing the Living Room...!!!");
        }

        if element.as_any()
            .downcast_ref::<Bedroom>()
            .is_some() {
            println!("Decimating the bedroom...!!!");
        }

        if element.as_any()
            .downcast_ref::<Kitchen>()
            .is_some() {
                println!("Destroying the kitchen...!!!");
        }
    }
}

Dirty, but at least it works. The problem, as mentioned before, is that in order to perform specific processing using dynamic dispatch, we use the trait object &HouseElement. However, in order to match against the concrete type, we need to try and match the downcasted reference against each concrete type. This is what the element.as_any().downcast_ref::() code does.

Let’s try it out!

Macushla:rust_visitor z0ltan$ cargo run
   Compiling rust_visitor v0.1.0 (file:///Users/z0ltan/Projects/Blog/Visitor_in_Rust/rust_visitor)
    Finished dev [unoptimized + debuginfo] target(s) in 0.56 secs
     Running `target/debug/rust_visitor`
Visiting the Living Room...
Visiting the bedroom...
Visiting the kitchen...
Visiting the House...

Bombing the Living Room...!!!
Decimating the bedroom...!!!
Destroying the kitchen...!!!
Annihilating the House...!!!

Excellent! Clunky and inelegant, but at least it works.

UPDATE: The implementations of the HouseElementVisitor trait can be simplified into a more elegant match expression block and using the is method of the Any trait as shown below:

Updated HouseElementListVisitor:

impl HouseElementVisitor for  HouseElementListVisitor {
    fn visit(&self, element: &HouseElement) {
        match element.as_any() {
            house if house.is::<House>() => println!("Visiting the house..."),
            living if living.is::<Livingroom>() => println!("Visiting the Living room..."),
            bedroom if bedroom.is::<Bedroom>() => println!("Visiting the bedroom..."),
            kitchen if kitchen.is::<Kitchen>() => println!("Visiting the kitchen..."),
            _ => {}
        }
    }
}

And the HouseElementDemolishVisitor implementation:

impl HouseElementVisitor for HouseElementDemolishVisitor {
    fn visit(&self, element: &HouseElement) {
        match element.as_any() {
            house if house.is::<House>() => println!("Annihilating the house...!!!"),
            living if living.is::<Livingroom>() => println!("Bombing the Living room...!!!"),
            bedroom if bedroom.is::<Bedroom>() => println!("Decimating the bedroom...!!!"),
            kitchen if kitchen.is::<Kitchen>() => println!("Destroying the kitchen...!!!"),
            _ => {}
        }
    }
}

Much more readable in my opinion, and it also hides the somewhat ugly downcast_ref::() bit!

A visitor pattern implementation in Rust

A basic multimap prototype in Rust

Recently, I have been working on a couple of projects, and in one of them, the logic required the use of a multimap. Now, this project is in Rust, and the standard Rust map types, `HashMap` and `BTreeMap` do not support duplicate keys.
Of course, there are full-blown crates available that provide multimap functionality.

However, this being a dead simple project, I decided to roll one out myself. This version simply wraps a `HashMap<K, Vec>` instance inside the custom data structure, but the advantage is that the client API remains consistent and never the wiser about the innards of the actual multimap type. Since I just needed `insert`, `get`, and `iter`, I implemented those for the custom multimap. If needed, the entire API of the `HashMap` type could be implemented for `MHashMap` as well.

Here is the code. It is simple enough that no explanation is really required:

#![allow(dead_code)]

/// A simple wrapper around a HashMap to simulate the basic functionality
/// of a multimap.

use std::collections::HashMap;
use std::hash::Hash;
use std::collections::hash_map::Iter;


#[derive(Debug)]
struct MHashMap<K: Eq + PartialEq + Hash, V>{
    map: HashMap<K, Vec<V>>,
}

impl<K, V> MHashMap<K, V> 
    where K : Eq + PartialEq + Hash
{
    fn new() -> Self {
        MHashMap {
            map: HashMap::new(),
        }
    }

    fn iter(&self) -> Iter<K, Vec<V>> {
        self.map.iter()
        
    }

    fn insert(&mut self, k: K, v: V) {
        if self.map.contains_key(&k) {
                self.map.get_mut(&k).unwrap().push(v);
        } else {
            self.map.insert(k, vec![v]);
        }
    }

    fn get(&self, k: &K) -> Option<&Vec<V>> {
        self.map.get(&k)
    }

    fn get_mut(&mut self, k: &K) -> Option<&mut Vec<V>> {
        self.map.get_mut(&k)
    }
}


fn main() {
    let mut mmap = MHashMap::new();

    mmap.insert(1, "One");
    mmap.insert(1, "Uno");
    mmap.insert(1, "Ein");
    mmap.insert(2, "Two");
    mmap.insert(3, "Three");

    println!("{:?}", mmap);

    let key = 1;

    match mmap.get(&key) {
        Some(vals) => println!("Got {:?} for key {}", vals, key),
        None => println!("Key {} not found!", key),
    }

    for entry in mmap.iter() {
        println!("{} = {:?}", entry.0, entry.1);
    }
}

A small test run:

Macushla:tcp_study z0ltan$ rustc mhash.rs
Macushla:tcp_study z0ltan$ ./mhash
MHashMap { map: {3: ["Three"], 1: ["One", "Uno", "Ein"], 2: ["Two"]} }
Got ["One", "Uno", "Ein"] for key 1
3 = ["Three"]
1 = ["One", "Uno", "Ein"]
2 = ["Two"]

Short and simple!

A basic multimap prototype in Rust