Talking with a Lisp

As far as I can recall, it was the first grade and there was this question that was staring at me from the book:

“If one cat can catch one mouse in one minute, how many
minutes does it take for three cats to catch three mice?”

Simple, is it not? Of course we all know that the answer is one minute. To someone grokking the basics of mathematics and logic, not so apparently easy. The fact is that it took me quite a bit of time to realize that it was not a simple case of arithmetic correspondence to (wrongly of course) equate three cats catching three mice in three minutes! What I realized after some ponderous consideration is that all the three cats were working in “parallel” to catch the three mice!

That then was my first lesson in parallelism.

c:\> shutdown -r -m \\<IP Address or Computer Name>

As promised in my last post, here it is – the real deal on the generation and re-generation of UUIDs for VMDK files on an ESX Server platform. A bit of background on this topic might be of interest and possibly of use to the reader.

It was around six months ago that I became involved in some work requiring me to dabble with VMDK files at an intimate level. One of the steps necessitated the re-generation and assignment of UUID for a VMDK file (of course, the base VMDK file and not the ‘flat’ or ‘rdmp’ file). As much as I tried searching online, I could not find any constructive approach to actually generating these UUIDs. This made sense to the extent that VMware had vested interest in preserving the cloud of secrecy to its UUID generation algorithms. At this juncture,  I must clarify that the UUID generation scheme for a VMDK is different from that of a Virtual Machine and other ESX Server platform components. For instance,  a Virtual Machine (whose metadata is represented in the form of a .vmx file) may have the following UUID information:

uuid.location = “56 4d 3e 6b 50 fe 21 4c-e6 9d 74 44 da ac de 23″
uuid.bios = “56 4d 3e 6b 50 fe 21 4c-e6 9d 74 44 da ac de 23″

The algorithm (or at least the basic methodology) by which these are generated is more or less documented in existing VMware documents. This will not be discussed in the blog. Digressing no more, in brief,  I found absolutely no information on how to generate a VMDK UUID for the ESX Server environment.  The VI-SDK (and the vSphere SDK) documentation provides information on an API that is used to change the UUID of a VMDK. The API is:

SetVirtualDiskUUID with four parameters described in order as

_this (Managed Object Reference to the VirtualDiskManager

name (String)    // referring to the Datastore path or URL of the VirtualDisk

datacenter(Managed Object Reference to the Datacenter) // not required for ESX Server

uuid(String)  // The new UUID to update the VMDK file with.

Now the interesting part is that the ‘uuid’ component is a necessary parameter to this API call. This absolutely makes no sense for my requirements of course, as I need the system to generate me a new UUID just like the ‘vmkfstools’ command can generate on the fly. In case you are not familiar with this method, consider the following representative example:

Suppose we want to regenerate the UUID of a VirtualDisk named New_W2K3_3.vmdk then follow the following steps:

1. Locate the ‘vmkfstools’ binary (in case it has not been already set in the PATH enviroment variable) using

[root@z0ltanik]# whereis vmkfstools

You should see a response such as

vmkfstools: /usr/sbin/vmkfstools /usr/man/man1/vmkfstools.1.gz

2. Suppose that vmfkstools resides in /usr/sbin, traverse to the directory where the VMDK file exists (or use relative/absolute paths) and invoke

[root@z0ltanik New_W2k3]# /usr/sbin/vmkfstools -J getuuid New_W2k3_3.vmdk

The output will be something like the following

UUID is 60 00 C2 95 78 96 04 68-53 cb d0 5e 0f a7 d0 32

This is the current UUID of the VMDK file.

3. To regenerate (and assign) a new UUID to the aforementioned VMDK file, simply invoke the following command. The response follows suit, displaying the
new UUID

[root@z0ltanik New_W2k3]# /usr/sbin/vmkfstools -J setuuid New_W2k3_3.vmdk

UUID is 60 00 C2 97 5e 42 fd 6f-bc ee 83 c5 51 c6 40 34

Et voila! You are all done. I had been hoping for a similar behavior from the SetVirtualDiskUUID API. Perhaps the VMware SDK engineers should have attended Joshua Bloch’s API design sessions eh? Just kidding of course – for the most part the vSphere (and its older avatar, the VI-SDK) SDK is impeccably designed. Enough digression already mates! Let’s get down to the meat of a simple Java example of how to read UUIDs from a file and translate them into new valid and perfectly acceptable UUIDs subject of course to the following caveats:

1. The user is responsible for writing the business logic to ensure the unique-ness of the new UUID in the Virtual Machine namespace (yes, NOT the ESX Server namespace). A trivial task indeed.

2. This code represents a representative example and suited to my requirements of retrieving a VMDK file’s existing VMDK and modifying it. However, this can be easily modified to generate UUIDs from scratch ad infinitum. The moot point here is that the first 7 characters of the UUID must always be “60 00 C2 9“. There you go – a revelation for you lads!

3. No major error condition checks, boundary condition checks etc. Feel free to add them yourself.

The code:

     /**
      * Program to read UUIDs from 'uuid.txt', generate valid UUIDs and display them on the console.
      *
      * @author Timmy z0ltan Jose
      */

      import java.util.Random;
      import java.io.BufferedReader;
      import java.io.FileReader;
      import java.io.IOException;

      public class SetUuid {

      public static void main(String args[]) {

            String oldUuid = null;
            BufferedReader myReader = null;

           try {
                    myReader = new BufferedReader(new FileReader("uuid.txt"));

                    while( (oldUuid = myReader.readLine()) != null) {

                            System.out.println("\nOld Uuid = " + oldUuid);

                            System.out.println("New Uuid = " + getNewUuid(oldUuid));
                     }

                } catch (IOException ex) {
                        ex.printStackTrace();
                } finally {
                      if (myReader != null) {
                          try {
                                 myReader.close();
                               } catch ( IOException ioex) {
                                      ioex.printStackTrace();
                                  }
                      }
                }
      }

      private static String getNewUuid(String oldUuid) {

            String fixed = null;
            String toggle = null;
            String randomOne = null;
            String randomTwo = null;
            String timestamp = null;
            String newUuid = " ";

            StringBuffer origUuid = new StringBuffer(oldUuid);

            for (int i = 0; i < origUuid.length(); i++) {

               if (origUuid.charAt(i) == ' ' || origUuid.charAt(i) == '-') {
                      origUuid.deleteCharAt(i);
               }
             }

            fixed = origUuid.substring(0, 7);

            toggle = Integer.toHexString(new Random().nextInt(16));

            byte[] randBytes = new byte[4];

            new Random().nextBytes(randBytes);
            String result = "";

            for (int i=0; i < randBytes.length; i++) {
                 result += Integer.toString( ( randBytes[i] & 0xff ) + 0x100, 16).substring( 1 );
            }

            randomOne = result;

            byte[] b = new byte[2];
            randomTwo = "";

            new Random().nextBytes(b);

            for (int j = 0; j < b.length; j++) {

                  randomTwo += Integer.toHexString( (b[j] & 0xff) + 0x100).substring(1);
            }

             timestamp = toggle + Long.toHexString(System.nanoTime());

             String firstHalf = fixed + toggle + randomOne;

             for (int j = 1; j < firstHalf.length(); j+=2) {

                    newUuid += " " + firstHalf.charAt(j-1) + firstHalf.charAt(j);
             }

              newUuid += "-";

              String secondHalf = randomTwo + timestamp;
              String newerUuid = (" " + secondHalf.charAt(0) + secondHalf.charAt(1)).trim();

              for (int k = 3; k < secondHalf.length(); k+=2) {

                     newerUuid += " " + secondHalf.charAt(k-1) + secondHalf.charAt(k);
               }

              return (newUuid + newerUuid.trim()).trim();

      }
    }

And here, of course, is the uuid.txt file:

60 00 C2 97 50 b3 99 57-47 b9 2b fd fb 45 ed 5f
60 00 C2 9f f0 04 aa 90-0f 6e 94 0a b0 33 fa f3
60 00 C2 93 54 19 90 f8-f1 8f 7b 25 7b 72 e2 3a
60 00 C2 97 c1 57 ca e4-a2 24 98 3f c2 0d 55 c2
60 00 C2 9e b1 00 fb 5e-a2 45 99 09 7f 0d fd 00

Sample output from a single run of the above code:

Old Uuid = 60 00 C2 97 50 b3 99 57-47 b9 2b fd fb 45 ed 5f
New Uuid = 60 00 C2 90 bb dd a8 31-b3 a8 0c 67 37 cc 6a f0

Old Uuid = 60 00 C2 9f f0 04 aa 90-0f 6e 94 0a b0 33 fa f3
New Uuid = 60 00 C2 93 16 22 c0 93-76 1d 3c 67 37 e5 b2 2c

Old Uuid = 60 00 C2 93 54 19 90 f8-f1 8f 7b 25 7b 72 e2 3a
New Uuid = 60 00 C2 9e 2f 80 ad 29-53 aa ec 67 37 fd 55 43

Old Uuid = 60 00 C2 97 c1 57 ca e4-a2 24 98 3f c2 0d 55 c2
New Uuid = 60 00 C2 95 24 ba 3b b4-17 18 5c 67 38 27 7b bd

Old Uuid = 60 00 C2 9e b1 00 fb 5e-a2 45 99 09 7f 0d fd 00
New Uuid = 60 00 C2 9c 69 24 b2 49-44 f2 cc 67 38 3e 81 b0

The code is simple enough as to not necessitate the presence of comments. However, if needed, I will explain the modus operandi in a separate blog. Right now I have to go get my power meal done. So long!

P.S: Some screwy issues with the code formatter. Well, que sera sera….. for now!

P.P.S: Fixed the formatting issue – should have used the <pre> tag instead of <code>….. duh!

In my last post I had mentioned some IP issues (as in ‘Intellectual Property’ ) as to why I could not pursue with the planned detailed charter of VMware VI/vSphere SDK tutorials. As it turns out, the situation has changed considerably in the meantime and I will be posting the tutorials on a hopefully regular basis. There have been some updates in the meantime of particular interest on which there is not much literature available online. I refer specifically to the UUID generation/regeneration of VMDK files for Virtual Machines.  The rules pertaining to these I reverse-engineered when I was working on a particular feature that necessitated the usage of the ’setVirtualDiskUuid’ API. Unfortunately, management did not have the cajones to carry forth my proposal and as such I consider it blog-worthy material! Hopefully it will be of use to other engineers working on the VMware ESX Server platforms. This will be the topic of discussion in the next blog post followed by an extensive and intensive VMware SDK (both VI and vSphere) tutorial series.  Wish me luck with the litigations!

For all these examples, VMware VI SDK 2.0/2.5 is assumed to be installed. Basically, version 2.5 provides some extra features for which you may check the relevant SDK manual.

As mentioned in the first post about using VMware APIs to perform operations on the ESX Server, the ESX 3.0, 3.5 and 3i services are basically exposed as a web service defined in in one WSDL file – vim.wsdl (all names are suffixed with “25″ for the VI SDK 2.5 version). This file contains the format and definition of the services provided by the ESX Server. In order to use these services from a Java client, we need to first generate the stubs from the WSDL file, compile those generated files and then bundle those classes into a JAR. These steps are demonstrated as follows:

1. Generate the stubs from the WSDL file.

> java -Xms256M -Xmx256M -classpath “./;./lib/axis.jar;./lib/axis-ant.jar;./lib/commons-discovery-0.2.jar;./lib/commons-logging.jar;./lib/jaxrpc.jar;./lib/log4j.jar;./lib/saaj.jar;./lib/wsdl4j-1.5.1.jar;./lib/vim.jar;./lib/javamail-2002-08-28.jar;./lib/activation.jar;” org.apache.axis.wsdl.WSDL2Java -w -O-1 -Nurn:vimService=com.vmware.vim -Nurn:vim2=com.vmware.vim -o. “./vim.wsdl“

Note: Here, the current directory is assumed to be the one where the WSDL file resides. As can be seen by the notation, ‘lib’ is a sub-directory where all the requisite JAR files (for the generation of the VI SDK specific jar(s)) reside. Also note the use of the open source Apache Axis Client used to convert the WSDL definition into Java classes, specifically the WSDL2Java class. This is unbelievably convenient if one desires to generate the latest JARs instead of relying on the pre-generated JAR files in the VI SDK.

2. Compile the stubs into classes:

> javac -J-Xmx128M -classpath “./;./lib/axis.jar;./lib/axis-ant.jar;./lib/commons-discovery-0.2.jar;./lib/commons-logging.jar;./lib/jaxrpc.jar;./lib/log4j.jar;./lib/saaj.jar;./lib/wsdl4j-1.5.1.jar;./lib/vim.jar;./lib/javamail-2002-08-28.jar;./lib/activation.jar;” com\vmware\vim\*.java

Note: The package structure can be clearly seen in the command above – com.vmware.vim. This is the general package structure for the VI SDK API classes.

3. Generate the vim.jar (or vim25.jar) file:

> jar cf ./vim.jar com\vmware\vim\*.class

Note: This bundles all the required class files into a convenient vim.jar file. The most important point to note here is that even after generating the vim.jar (or vim25.jar depending on whether you are using the VI SDK version 2.0 or 2.5), all these JARs are still required to be in the classpath of the application:

axis.jar, axis-ant.jar, commons-discovery-0.2.jar, commons-logging.jar, jaxrpc.jar, log4j.jar, saaj.jar, wsdl4j-1.5.1.jar, javamail-2002-08-28.jar

So there you go – these three simple steps (which can be automated in a simple script) are all you need to get your hands on the tools to start hacking on your ESX Server.

In the next tutorial, we’ll take our first baby steps into actual usage of the APIs and a bit of theoretical explanation is also in the offing.

Peace!

I meant tomorrow.

Phew! It seems like ages now when I take a look at my last post. I had laid out a pretty comprehensive charter for the VMware ESX Server operations tutorial series and yet due to unforeseen circumstances, I had to lay low for a while. To cut to the chase, I will complete the entire series in one fell swoop starting tomorrow. I just need to get my bearings right and fulfill my promise before leaving on a trans-Himalayan trek . Have fun and thanks for the patience!

Respect.

This is the first of a series of planned blogs pertaining to the use of the VMware SDK APIs to seamlessly exploit VMware ESX Server services programmatically.
This is the result of the experience that I have gained by working on a live project to provide VMware support to an existing Java-based enterprise software product.
An early disclaimer is highly in order: All these examples are idiomatic and do not relate to the actual code used in the aformentioned product. Phew! Now that we have got that out of our chest, let’s proceed onto the meat of the matter directly.

The roadmap of this tutorials series is roughly sketched out as follows:

Part 1 – Introduction to VMware and definition of the scope of the tutorial.

Part 2 – The VMware ESX server service exposed as a web-service, brief discussion of the WSDL file, generation of the required JAR files for the Java client.

Part 3 – Introductory code samples – logging on to the VMware ESX server, connecting and verification of session.

Part 4 - In depth discussion of Raw LUNs and VMFS disks and their applications in industry. Technical differences and varied approach in code.

Part 5 - In depth discussion of the VMware ESX Server SDK.

Part 6 - Continued discussion on the VMware ESX Server SDK.

Part 7 - Concluding session on the VMware ESX Server SDK.

Part 8 – Operations on the Virtual Machine -code samples.

Part 9 - Operations on the ESX Server (Host) – code samples.

Part 10 – Operations using the VMware SDK APIs specifically for the Storage domain – code samples.

Part 11 – Threading, concurrency and gotchas while using integrating the VMware programs with existing products.

Alright, so that is pretty much the road ahead. More sessions can be charted out based on feedback. Any specific operations can be discussed in-depth in future blogs.

Introduction to VMware:

VMware is a virtualization software which basically comes in two flavors – desktop and enterprise. In this tutorial series, I will be dealing with the enterprise server solution – VMware ESX Server and how to write programs to access the services provided by the server, which can be integrated with enterprise solutions. I will be using Java as the client language throughout this tutorial. This is not a series where the internal details/ performance statistics et cetera will be discussed, as also the relative merits of VMware with respect to other virtualization software. Those topics will be, at best, touched upon and skimmed through. Now there are enough resources to implement client programs to avail of the services exposed by the VMware ESX server but not nearly enough resources which show how to actually go beyond writing mere scripts and actually create enterprise ready VMware solutions. More details can be availed on the VMware site or better still, on Wikipedia.

As I have mentioned already, I will not be delving too deeply in the specifications of the ESX Server. Such information is widely available (though the VMware site documentation, in my most humble opinion, sucks!) on the Internet.

That’s all for part 1. The next session is shortly forthcoming. Till then, peace!

Hilarious. Ludicrous. Ridiculous. Nefarious.

Four words that came to mind when I chanced to read the comments section of this blog. I had posted a comment on it around a month back. It is William Reardon’s (I have no idea who he is) allegedly technical blog and this particular one was one denigrating and denouncing Scribd. I was amazed at the bilious and vicious wording of the blog (maybe he was pissed off at something?) and more than that, the lack of any justification for his rant irked me tremendously. I promptly stated the fact that the site did indeed provide a way to store files as PDFs (which the author had vehemently denied) and also that it was quite a pointless blog. What happened next is there for all to see in the comments section. One thing is for sure – whatever respect I could have had for the author was quickly dissipated within the first two paragraphs. I guess the author had forgotten that noblesse oblige applies to all. Even programmers.

That is right. Today is officially ‘Goof-Off Day’ for me. The title may be slightly misleading in that it implies a deliberate attempt at mooching off a fine wednesday work day. In fact, if it were indeed a story right out of The Daily Wtf, it would definitely be so. However, I reiterate (at pain of sounding redundant but it’s just the goofing-off potion at work) that it is a natural progression of events.  Now onto a more serious discussion of this phenomenon.

In every workplace, but especially in an industry placing so much emphasis on “thinking”, it is inevitable that every so often, even the most assiduous mind-worker has to take a break. No compulsion, no rebellion, no coercion. Our industry is characterized by periodic variations of extreme stess and complete leisure (with a general slope of gradations in between of course but not so pronouncedly as the two end-points). For a complete and statistics-backed analysis of such matters pertinent to Software Management, what better resource to point to than Peopleware. I first came across this title when I was reading through the interesting (but not so useful – my opinion strictly!) book called Smart and Gets Things Done: Joel Spolsky’s Concise Guide to Finding the Best Technical Talent . He had highly recommended this book but the title of the book put me off initially. It reminded me of those weird and untractable academic Software Engineering books from my college days. I hated them. I positively despised, abhorred and reviled them. In hindsight, my revulsion seems to be justified as I am yet to see one positive use of anything mentioned in those fat books. The one and only Software Engineering I would recommend is The Mythical Man Month. And now, Peopleware. You had better go read that book. Now.  Afore you get mired in this goofy business! As I was saying (pray, indeed this is a day for digressions!) every software engineer inevitably finds that he needs a day now and then to just plain do anything but his main work. It can be anything - chatting, surfing, going to the gym, dozing off in the dorm, writing blogs (as I am doing right now!), catching up on news, handling personal calls (which I am definitely not doing right now), going out for a stroll, playing some football (that’s soccer for you Uncle Sam), planning for the evening ahead, meditating or just plain staying in stasis like a zombie (nothing different from the status quo for some). To demonstrate what I mean, let me recount the activities that I have engaged in today - in strictly chronological order:

1. Reached office… made my favorite cup of stomach-burning black coffee… glanced at the headlines.

13.  Seems like I’m goofing off major today…well, maybe I deserve it!…hey, I could write a blog on that!

And here I am – eating ten minutes of your time to compensate for the day that I have mooched off… and enjoyed! Tom and Timothy (of Peopleware fame) would certainly approve! The fact is that it is inevitable that one has to just do anything but that which brings in the moolah at the end of the month. In some ways it clears up the head, refreshes the spirit and also it is healthy to take a break on official time! Try it out… works wonders… and the fact is, if you tell me that you do not mooch off at all – I’ll show you a horse that can bark in German!

So folks, how do you goof off your day?