Enabling and configuring Tracing/Logging for the SFCB CIMOM on ESXi 5.x machines


It has been my constant experience that VMware creates excellent software and deplorable documentation. Ever since my first tryst with VMware around 2008 or so when I was given the task of working with the VMware VI-SDK (for the ESX platform, and later on for the ESXi platform as well), it has been a wonderful experience seeing those APIs work beautifully, simply, and powerfully. Working on the SDK also led me to explore the internals of the ESX platform itself, and it was quite an enriching experience in and of itself. However, that was also the first time that I experienced the horrendous cesspool of detritus that VMware calls its documentation.  Finding anything of use on its website is an exercise in futility, and the forums don’t fare much better. The best way I found to work my way through this mess was to read the SDK code itself, explore the ESXi console, and try out lots of small prototype programs to see if the data was indeed correct. One saving grace was the availability of the MOB (Managed Object Browser) which arguably taught me more than anything else.

Recently I had a need to set up logging/tracing on the SFCB CIMOM used by the ESXi 4.x and 5.x series, and it was déjà vu all over again. Granted that SFCB is not a VMware product, but since it comes pre-installed on every ESXi box, I would have expected some basic guides explaining the configuration and workings of the CIMOM vis-à-vis the ESXi kernel. Absolutely none whatsoever. So in order to spare people the trouble of not finding any relevant results, let me share my experience in setting up the logging mechanism for SFCB specifically in reference to ESXi.

SFCB (Small Footprint CIM Broker) is an excellent CIMOM that is lightweight, and has a very nice pluggable interface for third-party CIM Providers. An added benefit is that a Provider crash is isolated, and is not allowed to crash the CIMOM itself. No wonder then that it is already the de facto CIMOM for a wide variety of platforms – various Linux flavors, and ESXi of course (which used to depend on the unwieldy gargoyle that is OpenPegasus till Common Sense won out). Now, with ESXi in mind, here are the three simple steps needed to configure the SFCB CIMOM for logging/tracing (especially useful when an errant CIM Provider needs to be checked for anomalous behavior):

1. Log on to the ESXi shell (you need to have SSH enabled through the vSphere Client before you can open a console through a tool such as PuTTY) and check the contents of the /etc/sfcb/sfcb.cfg file (showing the default configuration here):

~ # cat etc/sfcb/sfcb.cfg.new
# Do not modify this header
# VMware ESXi 5.5.0 build-1106514
#
# set logLevel using advanced config: CIMLogLevel
httpPort:       5988
enableHttp:     true
httpProcs:      2
httpsPort:      5989
enableHttps:    true
httpsProcs:     4
provProcs:      16
httpLocalOnly:  true
doBasicAuth:    true
basicAuthLib:   sfcBasicPAMAuthentication
useChunking:    true
keepaliveTimeout: 1
keepaliveMaxRequest: 10
providerTimeoutInterval: 120
sslKeyFilePath: /etc/vmware/ssl/rui.key
sslCertificateFilePath: /etc/vmware/ssl/rui.crt
sslClientTrustStore: /etc/sfcb/client.pem
sslClientCertificate: ignore
certificateAuthLib:   sfcCertificateAuthentication
registrationDir: /var/lib/sfcb/registration
providerDirs: /usr/lib /usr/lib/cmpi /usr/lib/cim
enableInterOp:  true
threadStackSize:     524288
rcvSocketTimeOut: 0
requestQueueSize: 10
threadPoolSize: 5
intSockTimeout: 600
maxSemInitRetries: 5
maxFailureThreshold: 3
cimXmlFdSoftLimit: 512
cimXmlFdHardLimit: 1024

You can see above the default values for various parameters as present on my local ESXi 5.5 machine. The contents should not be too different, if at all, for other versions of ESXi. This file, /etc/sfcb/sfcb.cfg is the main configuration file for the SFCB CIMOM. You can change various parameters by reading up the general documentation of the SFCB CIMOM for other Operating Systems.

2. Now, add the following lines to the /etc/sfcb/sfcb.cfg file:

traceLevel: 1
traceMask: 0x0000103
traceFile: /vmfs/volumes/50cb7c7d-30e72dbe-a165-ac162d8be508/timmy/z0ltan.log

So your new file might look something like the following:

~ # cat /etc/sfcb/sfcb.cfg
# Do not modify this header
# VMware ESXi 5.5.0 build-1106514
#
# set logLevel using advanced config: CIMLogLevel
httpPort:       5988
enableHttp:     true
httpProcs:      2
httpsPort:      5989
enableHttps:    true
httpsProcs:     4
provProcs:      16
httpLocalOnly:  true
doBasicAuth:    true
basicAuthLib:   sfcBasicPAMAuthentication
useChunking:    true
keepaliveTimeout: 1
keepaliveMaxRequest: 10
providerTimeoutInterval: 120
sslKeyFilePath: /etc/vmware/ssl/rui.key
sslCertificateFilePath: /etc/vmware/ssl/rui.crt
sslClientTrustStore: /etc/sfcb/client.pem
sslClientCertificate: ignore
certificateAuthLib:   sfcCertificateAuthentication
registrationDir: /var/lib/sfcb/registration
providerDirs: /usr/lib /usr/lib/cmpi /usr/lib/cim
enableInterOp:  true
threadStackSize:     524288
rcvSocketTimeOut: 0
requestQueueSize: 10
threadPoolSize: 5
intSockTimeout: 600
maxSemInitRetries: 5
maxFailureThreshold: 3
cimXmlFdSoftLimit: 512
cimXmlFdHardLimit: 1024
traceLevel: 1
traceMask: 0x0000103
traceFile: /vmfs/volumes/50cb7c7d-30e72dbe-a165-ac162d8be508/timmy/z0ltan.log

Explanation:

traceLevel dictates the level of logging that you wish to generate (in my experience, a level of ‘1’ should suffice for most cases, but levels 2, 3, or even 4 can be tried out depending on your requirements. The higher the level, the finer the level of logging). However, beware that increasing the level of logging also increases the memory and CPU overheads on the ESXi box, so set the level of logging with a discriminating approach.

traceMask is a bitmask that allows SFCB to enable logging for specific components (a very useful feature that produces smaller and more relevant logs). The various components are listed below along with their bitmasks. Either the int or the hex mask can be used. Also, in order to generate logs for multiple components, their bitmasks may be ORed together to generate a single bitmask to be set as the traceMask. (For instance, I have my bitmask set to: 0x0000103 (providerMgr | providerDrv | providers).

      Traceable Components:     Int        Hex
 	       providerMgr:          1	0x0000001
 	       providerDrv:          2	0x0000002
 	        cimxmlProc:          4	0x0000004
 	        httpDaemon:          8	0x0000008
 	           upCalls:         16	0x0000010
 	          encCalls:         32	0x0000020
 	   ProviderInstMgr:         64	0x0000040
 	  providerAssocMgr:        128	0x0000080
 	         providers:        256	0x0000100
 	       indProvider:        512	0x0000200
 	  internalProvider:       1024	0x0000400
 	        objectImpl:       2048	0x0000800
 	             xmlIn:       4096	0x0001000
 	            xmlOut:       8192	0x0002000
 	           sockets:      16384	0x0004000
 	         memoryMgr:      32768	0x0008000
 	          msgQueue:      65536	0x0010000
 	        xmlParsing:     131072	0x0020000
 	    responseTiming:     262144	0x0040000
 	         dbpdaemon:     524288	0x0080000
 	               slp:    1048576	0x0100000

traceFile, as the name suggests, refers to the location where you want the trace outputs to be logger. By default this is stderr (console), but this can be made to point to a file location (as seen in the sample config file shown previously). I would suggest setting this file in a persistent location with enough space availability (such as on an available datastore). The reason is that if you should choose a location within the root folder (say, /mylogs/test.log), it can quickly overwhelm your ESXi machine. Remember that everything under the root folder in ESXi is necessarily in volatile memory with size restrictions, and from my experience, these logs can quickly grown to hundreds of MBs in size.

3. Restart the SFCB CIMOM in order to reflect the changes to the config file:

~# /etc/init.d/sfcbd-watchdog restart

Note: If you want to go about in a cleaner way, I would recommend that you stop the SFCB CIMOM as the first step (before modifying the config file):

~# /etc/init.d/sfcbd-watchdog stop

Confirm that the SFCB CIMOM has indeed shutdown:

~# /etc/init.d/sfcbd-watchdog status

And then proceed with the steps mentioned before, and when the config file has been updated with the changes, start the SFCB CIMOM again:

~# /etc/init.d/sfcbd-watchdog start

Followed by a final confirmation that the SFCB CIMOM is up and running:

~# /etc/init.d/sfcbd-watchdog status

And that’s all there is to it! Now you should be able to see the log file being populated with log messages as the SFCB CIMOM starts running, and you can then trigger your own CIM operations (such as querying for specific CIM classes on your CIM provider), and those operations should be logged in the log file as well.

Advertisements
Enabling and configuring Tracing/Logging for the SFCB CIMOM on ESXi 5.x machines

One thought on “Enabling and configuring Tracing/Logging for the SFCB CIMOM on ESXi 5.x machines

  1. Starting with 6.5 there is an esxcli command for managing sfcb.cfg :
    esxcli system wbem set -l [debug|info|warning|error]
    esxcli system wbem set –enabled false
    esxcli system wbem set –enabled true

    Like

Speak your mind!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s