It has been my constant experience that VMware creates excellent software and deplorable documentation. Ever since my first tryst with VMware around 2008 or so when I was given the task of working with the VMware VI-SDK (for the ESX platform, and later on for the ESXi platform as well), it has been a wonderful experience seeing those APIs work beautifully, simply, and powerfully. Working on the SDK also led me to explore the internals of the ESX platform itself, and it was quite an enriching experience in and of itself. However, that was also the first time that I experienced the horrendous cesspool of detritus that VMware calls its documentation. Finding anything of use on its website is an exercise in futility, and the forums don’t fare much better. The best way I found to work my way through this mess was to read the SDK code itself, explore the ESXi console, and try out lots of small prototype programs to see if the data was indeed correct. One saving grace was the availability of the MOB (Managed Object Browser) which arguably taught me more than anything else.
Recently I had a need to set up logging/tracing on the SFCB CIMOM used by the ESXi 4.x and 5.x series, and it was déjà vu all over again. Granted that SFCB is not a VMware product, but since it comes pre-installed on every ESXi box, I would have expected some basic guides explaining the configuration and workings of the CIMOM vis-à-vis the ESXi kernel. Absolutely none whatsoever. So in order to spare people the trouble of not finding any relevant results, let me share my experience in setting up the logging mechanism for SFCB specifically in reference to ESXi.
SFCB (Small Footprint CIM Broker) is an excellent CIMOM that is lightweight, and has a very nice pluggable interface for third-party CIM Providers. An added benefit is that a Provider crash is isolated, and is not allowed to crash the CIMOM itself. No wonder then that it is already the de facto CIMOM for a wide variety of platforms – various Linux flavors, and ESXi of course (which used to depend on the unwieldy gargoyle that is OpenPegasus till Common Sense won out). Now, with ESXi in mind, here are the three simple steps needed to configure the SFCB CIMOM for logging/tracing (especially useful when an errant CIM Provider needs to be checked for anomalous behavior):
1. Log on to the ESXi shell (you need to have SSH enabled through the vSphere Client before you can open a console through a tool such as PuTTY) and check the contents of the /etc/sfcb/sfcb.cfg file (showing the default configuration here):
~ # cat etc/sfcb/sfcb.cfg.new # Do not modify this header # VMware ESXi 5.5.0 build-1106514 # # set logLevel using advanced config: CIMLogLevel httpPort: 5988 enableHttp: true httpProcs: 2 httpsPort: 5989 enableHttps: true httpsProcs: 4 provProcs: 16 httpLocalOnly: true doBasicAuth: true basicAuthLib: sfcBasicPAMAuthentication useChunking: true keepaliveTimeout: 1 keepaliveMaxRequest: 10 providerTimeoutInterval: 120 sslKeyFilePath: /etc/vmware/ssl/rui.key sslCertificateFilePath: /etc/vmware/ssl/rui.crt sslClientTrustStore: /etc/sfcb/client.pem sslClientCertificate: ignore certificateAuthLib: sfcCertificateAuthentication registrationDir: /var/lib/sfcb/registration providerDirs: /usr/lib /usr/lib/cmpi /usr/lib/cim enableInterOp: true threadStackSize: 524288 rcvSocketTimeOut: 0 requestQueueSize: 10 threadPoolSize: 5 intSockTimeout: 600 maxSemInitRetries: 5 maxFailureThreshold: 3 cimXmlFdSoftLimit: 512 cimXmlFdHardLimit: 1024
You can see above the default values for various parameters as present on my local ESXi 5.5 machine. The contents should not be too different, if at all, for other versions of ESXi. This file, /etc/sfcb/sfcb.cfg is the main configuration file for the SFCB CIMOM. You can change various parameters by reading up the general documentation of the SFCB CIMOM for other Operating Systems.
2. Now, add the following lines to the /etc/sfcb/sfcb.cfg file:
traceLevel: 1 traceMask: 0x0000103 traceFile: /vmfs/volumes/50cb7c7d-30e72dbe-a165-ac162d8be508/timmy/z0ltan.log
So your new file might look something like the following:
~ # cat /etc/sfcb/sfcb.cfg # Do not modify this header # VMware ESXi 5.5.0 build-1106514 # # set logLevel using advanced config: CIMLogLevel httpPort: 5988 enableHttp: true httpProcs: 2 httpsPort: 5989 enableHttps: true httpsProcs: 4 provProcs: 16 httpLocalOnly: true doBasicAuth: true basicAuthLib: sfcBasicPAMAuthentication useChunking: true keepaliveTimeout: 1 keepaliveMaxRequest: 10 providerTimeoutInterval: 120 sslKeyFilePath: /etc/vmware/ssl/rui.key sslCertificateFilePath: /etc/vmware/ssl/rui.crt sslClientTrustStore: /etc/sfcb/client.pem sslClientCertificate: ignore certificateAuthLib: sfcCertificateAuthentication registrationDir: /var/lib/sfcb/registration providerDirs: /usr/lib /usr/lib/cmpi /usr/lib/cim enableInterOp: true threadStackSize: 524288 rcvSocketTimeOut: 0 requestQueueSize: 10 threadPoolSize: 5 intSockTimeout: 600 maxSemInitRetries: 5 maxFailureThreshold: 3 cimXmlFdSoftLimit: 512 cimXmlFdHardLimit: 1024 traceLevel: 1 traceMask: 0x0000103 traceFile: /vmfs/volumes/50cb7c7d-30e72dbe-a165-ac162d8be508/timmy/z0ltan.log
traceLevel dictates the level of logging that you wish to generate (in my experience, a level of ‘1’ should suffice for most cases, but levels 2, 3, or even 4 can be tried out depending on your requirements. The higher the level, the finer the level of logging). However, beware that increasing the level of logging also increases the memory and CPU overheads on the ESXi box, so set the level of logging with a discriminating approach.
traceMask is a bitmask that allows SFCB to enable logging for specific components (a very useful feature that produces smaller and more relevant logs). The various components are listed below along with their bitmasks. Either the int or the hex mask can be used. Also, in order to generate logs for multiple components, their bitmasks may be ORed together to generate a single bitmask to be set as the traceMask. (For instance, I have my bitmask set to: 0x0000103 (providerMgr | providerDrv | providers).
Traceable Components: Int Hex providerMgr: 1 0x0000001 providerDrv: 2 0x0000002 cimxmlProc: 4 0x0000004 httpDaemon: 8 0x0000008 upCalls: 16 0x0000010 encCalls: 32 0x0000020 ProviderInstMgr: 64 0x0000040 providerAssocMgr: 128 0x0000080 providers: 256 0x0000100 indProvider: 512 0x0000200 internalProvider: 1024 0x0000400 objectImpl: 2048 0x0000800 xmlIn: 4096 0x0001000 xmlOut: 8192 0x0002000 sockets: 16384 0x0004000 memoryMgr: 32768 0x0008000 msgQueue: 65536 0x0010000 xmlParsing: 131072 0x0020000 responseTiming: 262144 0x0040000 dbpdaemon: 524288 0x0080000 slp: 1048576 0x0100000
traceFile, as the name suggests, refers to the location where you want the trace outputs to be logger. By default this is stderr (console), but this can be made to point to a file location (as seen in the sample config file shown previously). I would suggest setting this file in a persistent location with enough space availability (such as on an available datastore). The reason is that if you should choose a location within the root folder (say, /mylogs/test.log), it can quickly overwhelm your ESXi machine. Remember that everything under the root folder in ESXi is necessarily in volatile memory with size restrictions, and from my experience, these logs can quickly grown to hundreds of MBs in size.
3. Restart the SFCB CIMOM in order to reflect the changes to the config file:
~# /etc/init.d/sfcbd-watchdog restart
Note: If you want to go about in a cleaner way, I would recommend that you stop the SFCB CIMOM as the first step (before modifying the config file):
~# /etc/init.d/sfcbd-watchdog stop
Confirm that the SFCB CIMOM has indeed shutdown:
~# /etc/init.d/sfcbd-watchdog status
And then proceed with the steps mentioned before, and when the config file has been updated with the changes, start the SFCB CIMOM again:
~# /etc/init.d/sfcbd-watchdog start
Followed by a final confirmation that the SFCB CIMOM is up and running:
~# /etc/init.d/sfcbd-watchdog status
And that’s all there is to it! Now you should be able to see the log file being populated with log messages as the SFCB CIMOM starts running, and you can then trigger your own CIM operations (such as querying for specific CIM classes on your CIM provider), and those operations should be logged in the log file as well.