Monday, December 12, 2011

Changing vCenter database within the DSN breaks Storage Management Service

I was doing a vCenter 4.1 -> 5.0 upgrade over the weekend and ran into a few glitches (see my other post here)

After the successful upgrade, the Storage Management Service failed to start, with error 'failed to initialize' in the vCenter Service Status plugin.

As this was a Production vCenter, and we had eaten most of our outage window, we couldn't afford the time to try to upgrade on the production DB (PRODVC), have it fail, then wait for the 30GB's to restore then try again. So we restored onto a test DB, and tested on that which meant all we needed to do was revert the snapshot on the vCenter to be back at the stable 4.1 state. anyway......

  • Stopped vCenter
  • Updated the vCenter DSN to point to the copy (TESTVC)
  • We then ran a fix on TESTVC (dropped an index or two)
  • Upgraded vCenter
  • We then made the decision to restore the upgraded DB onto the PRODVC 
    • Seemed the less complex route. This was a production vCenter with about 20 hosts and ~900 vms, rolling back to the 4.1 DB would have upset all the managed hosts. Then require the fix to be applied, then the upgrade process again....
  • Stop vCenter
  • Restored TESTVC onto PRODVC
  • Changed the DSN back to PRODVC
    • (Checked SQL jobs were OK)
  • All is good in da hood.

Once we upgraded, the Storage Management Service (SMS) would not start, and vCenter Service Status would report 'failed to initialize'

When I had a look in sms.log I noticed the following errors:

Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: inherited from com.vmware.vim.binding.vim.fault.VimFault:inherited from com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.

OK - so it is a database/authentication error…….Then these:

2011-12-11 16:20:57,140 [Thread-15] DEBUG com.vmware.vim.sms.provider.VasaProviderImpl - [setProviderMoIdCounter] Setting the providerMoIdCounter to 0
2011-12-11 16:21:11,551 [Thread-14] ERROR com.vmware.vim.sms.provider.VcProviderImpl - Failed populating service cache
org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Cannot open database "TESTVC" requested by the login. The login failed.)

Notice that it was referring to the old name of the DB? So it is not using the DSN to find the database each time, but to just to write out its own properties file once…

2011-12-11 19:40:09,729 [Thread-14] INFO com.vmware.vim.common.vdb.VdbConfig - Property file=C:\ProgramData\VMware\VMware VirtualCenter\

Opened it

# For Windows, just deduce the JDBC URL and user/password from the
# data source information in the VC registry

Simply corrected the DB name in the JDBC string and then wait (up to) 30 mins for it to start again and off you go :)

So rather than fallout from our upgrade fun, I think the cause of this one was changing the target database in the DSN after the SMS service had written out its little JDBC URL in the props file.


Anonymous said...

why wait 30 mins for the service to start?

winter said...

i didnt feel like restarting the web services, and had a few other things on the boil. SMS will attempt to start every 30 mins itself, if you need it !now! you could always restart tomcat

Jeff Judy said...

Editing fixed it for me. Thanks!!

Jon Hicks said...

Fixed a similar issue for me in 5.1. Thanks!

Anonymous said...

Fixed for me on 5.1 - thanks.

Nicky Paul said...

Sending a SMS or MMS is a standout amongst the most well-known assignments performed on the Twilio Platform. Communicating something specific is as basic as POST-ing to the Messages asset, however since it's a typical activity it merits strolling through in point of interest underneath.
bulk sms to woocommerce customers


blogs I read


My photo

Enjoys spending time in layer 6 & 7. Passionate about delivering quality education.
Based in Sydney.