Powering up there were no issues -- all VMFS datastores were mounted and there are no alarms nor alerts. So after checks and everything was all set, ESXi hosts were exited from maintenance mode and virtual machines were started.
Most of the virtual machines booted back. But some of those that were powered up did not proceed on booting but instead prompted the error: "The operation is not allowed in the current state of the host."
This was completely new to me. And VMware forums and public knowledge base had an almost similar error: "The operation is not allowed in the current connection state of the host." Which was not quite what I was facing.
I was using the vSphere Web Client, so I tried using the old graphical user interface. Things did not seem to change -- same error.
I restarted the VMware VirtualCenter Server service but I still ended up with the same error. So I started to scratch my head. Of the several virtual machines that manifested this error, one thing was common: they belonged to similar ESXi hosts.
I went back to reading the posts with the almost identical error and it suggested to restart the management agents on the ESXi hosts. So I tried that but instead of just the management agents, I executed a stop and start of the entire services.sh.
Below are the traces of the execution.
~ # services.sh stop Running hp-ams.sh stop Terminating hp-ams-watchdog process with PID 35513 50567 50568 Stopping process 35479 35477 ... hpHelper process is now stopped. Running vmware-fdm stop Stopping vmware-fdm:success Running hp-mst.init stop Unable to unload module mst: Module not found Running xorg stop Running sfcbd stop This operation is not supported. Please use /etc/init.d/sfcbd-watchdog stop Running wsman stop Stopping openwsmand Running snmpd stop Running sfcbd-watchdog stop Running hpcru.init stop Running hpilo.init stop Running hpsmx.init stop Running vpxa stop watchdog-vpxa: Terminating watchdog process with PID 34561 vpxa stopped. Running vobd stop watchdog-vobd: Terminating watchdog process with PID 33332 vobd stopped Running lacp stop watchdog-net-lacp: Terminating watchdog process with PID 33542 Running memscrubd stop memscrubd is not running Running smartd stop watchdog-smartd: Terminating watchdog process with PID 34425 smartd stopped Running dcbd stop watchdog-dcbd: Terminating watchdog process with PID 34370 Running cdp stop watchdog-cdp: Terminating watchdog process with PID 34328 Running nscd stop watchdog-nscd: Terminating watchdog process with PID 34283 Running slpd stop Stopping slpd Running storageRM stop watchdog-storageRM: Terminating watchdog process with PID 34231 storageRM stopped Running hostd stop watchdog-hostd: Terminating watchdog process with PID 34191 hostd stopped. Running vmfstraced stop watchdog-vmfstracegd: PID file /var/run/vmware/watchdog-vmfstracegd.PID does not exist watchdog-vmfstracegd: Unable to terminate watchdog: No running watchdog process for vmfstracegd vmfstracegd is not running Failed to clear vmfstracegd memory reservation Running lbtd stop watchdog-net-lbt: Terminating watchdog process with PID 34130 net-lbt stopped Running sdrsInjector stop watchdog-sdrsInjector: Terminating watchdog process with PID 34094 sdrsInjector stopped Running rhttpproxy stop watchdog-rhttpproxy: Terminating watchdog process with PID 34053 rhttpproxy stopped. Running sensord stop sensord is not running Running hp-oem.init stop Running usbarbitrator stop watchdog-usbarbitrator: Terminating watchdog process with PID 33941 usbarbitrator stopped Running DCUI stop Disabling DCUI logins VobUserLib_Init failed with -1 Running ntpd stop Stopping ntpd watchdog-ntpd: Terminating watchdog process with PID 33876 Connect to localhost failed: Connection failure Running vsantraced stop watchdog-vsantraced: Terminating watchdog process with PID 33807 vsantraced stopped watchdog-vsantracedUrgen: Terminating watchdog process with PID 33836 vsantracedUrgen stopped Persisting traces to /locker/vsantraces ~ # ~ # services.sh start Running vsantraced start Scratch partition is not backed by persistent storage Storing traces to /vmfs/volumes/4dc4866f-040de4b6-1f88-0017a477f825/vsantraces vsantraced started vsantracedUrgen started Running ntpd start Connect to localhost failed: Connection failure Starting ntpd Running DCUI start Enabling DCUI login: runlevel = VobUserLib_Init failed with -1 Running usbarbitrator start usbarbitrator started Running hp-oem.init start Running sensord start sensord started Running rhttpproxy start rhttpproxy started. Running sdrsInjector start sdrsInjector started Running lbtd start net-lbt started Running vmfstraced start VMFS Global Tracing is not enabled. Running hostd start Ramdisk 'hostd' with estimated size of 803MB already exists hostd started. Running storageRM start storageRM started Running slpd start Starting slpd Running nscd start nscd started Running cdp start cdp started Running dcbd start dcbd started Running smartd start smartd started Running memscrubd start The checkPages boot option is FALSE, hence memscrubd could not be started. Running lacp start LACP daemon started Running vobd start vobd started Running vpxa start Connect to localhost failed: Connection failure Running hpsmx.init start Running hpilo.init start Running hpcru.init start Running sfcbd-watchdog start Connect to localhost failed: Connection failure Connect to localhost failed: Connection failure Running snmpd start Running wsman start Starting openwsmand Running sfcbd start This operation is not supported. Please use /etc/init.d/sfcbd-watchdog start Running xorg start Running hp-mst.init start Running vmware-fdm start Starting vmware-fdm:success Running hp-ams.sh start Starting hpHelper service... ~ #
One thing to note here is that during the stop and start of services.sh vCenter server will complain about the host being disconnected. This is normal as the daemons handshaking and talking to each other lose connectivity.
After the stop and start was done, powering up of the virtual machines that had errors were successfully executed. Another point chalked to experience.
