I am just trying to understand the reason behind ‘Why the VM Managers went inactive’ all of a sudden, even though they are pinging well, credentials are working fine, test connection is also successful, there is enough space under Archive Size, command vmman.bat- status shows no error and states all VM Managers are Ok with no error, running VM manager tool version ( 9.2.19), ILMT version 9.2.19
Steps we have taken:
Increase the Archive_size as it was showing exceeded.
Re-Run Capacity scan and Upload results.
Force the upload of VM manager results.
Restarted Fill Db
Restarted VM Manager Data collector and Application Services.
Run manual imports and waited for scheduled imports to run.
One of the VM man logs says connection time out so we increased the time out as well.
After doing all of this, the results are still the same. Does anyone encountered such problem and knows what to do next.
Have you confirmed that new VM Manager upload files are under the UploadManagerData folder on the root server before FillDB was restarted and then import run?
Check the import log for VM Manager data processing section to see if there is more information
First of all, the VM Manager is actually a java process by its own. Check for a process running com.ibm.license.mgmt.datacollector.Run. You starts, stops and check this with the vmman.bat command.
This process collects every 30 minute data from the defined VM Managers. The every 12th hour all the resulting files are zipped together and put in the upload directory. The timing can be changed in
the file vmmainconf.properties in …\BESClient\LMT\VMMAN\config.
Are there any files in …\BESClient\LMT\VMMAN\upload named vmman_scan_.xml? ( is the number of the defined VM Manager).
Check the timestamp, should not be older than 12 hours.
There may be more than one file for a vmmgr, it depends on if anything has changed since the last scan (but the save data may be the same, thou).
After 12 hours all these vmman_scan.xml are tar-ed (maybe zipped on Win) and sent to the BigFix server.
Check …\BESServer\UploadManagerData\BufferDir\sha1<ID-server> for a file
vmman_scan*_vmman.tar.gz (or zip)
Check timestamp of the file.
If there is content in the vmman_scan*.xml, the data collection is probably working.
If the uploaded results is in the server Bufferdir, the upload is working.
Then check in ILMT the import log.
The results is not found in the ILMT reports until an import is done after the upload.
Thanks for taking out time and sharing your thoughts really appreciate that.
Before i get down to check the suggested files, one more question can it be possible that all the vmmans shows ok at the backend ( verified it by running vmman.bat -status) but dont update their status on the Web UI.?
If yes, what can be done to get that reflected on webui.
It might be so, if the userid for an VM Manager is valid (ie it can be used to successfully login in to a vCenter), but does not have the rights to see the computers you need to have the information about.
That is why you should check what is in the vmman_scan_* files.
If they does not exists or are older then 12h, probably there is a problem with to connect.
But if the do exist, but not contain any of the UUID of you computers, then the userid for the VM Manager does not have the rights to see them.
Search all of the vmman_scan_* files for UUID’s that are flagged for no vm manager data.
(See https://www.ibm.com/support/knowledgecenter/SS8JFY_9.2.0/com.ibm.lmt.doc/Inventory/admin/t_obtaining_the_list_of_virtual_machines_uuids.html how to obtain the UUID of your computers. If there are computers flagged for no vm data, there is a link in the Overview panel in th ILMT WebUI, and in one of the columns the will see a TLM_VM_xxxxxx value, where the xxxxxx is the UUID)
If not found, in any of the files, it will not be transported to ILMT, ie the WebUI.
The steps before ILMT knowns anything about the VM Manager is:
The java process collects the information to vmman_scan_* files
Every 12 hour all these files are zipped together and uploaded to the BigFix server
At the next import (usually at midnight) the data is imported to the ILMT database and seen in the WebUI.
Check in the import log (from the WebUI) for some lines not far from the bottom:
2020-09-03 22:01:33 (+0:00:00.002) INFO: ETL Validate Central VM Manager Tool Remote ID: Start
2020-09-03 22:01:33 (+0:00:00.100) INFO: ETL Validate Central VM Manager Tool Remote ID: Success
2020-09-03 22:01:33 (+0:00:00.003) INFO: ETL from ImportVmManScans: Start
2020-09-03 22:01:49 (+0:00:16.498) INFO: [ILMTCore] (ImportThread) com.ibm.ilmt.core.ILMTCoreVMMan::printStatistics::STATISTICS: 164 files from 5 VM Managers were processed. Processed clusters: 368, skipped clusters: 3510, processed hosts: 962, skipped hosts: 6770, processed guests: 1064 (includes 0 guests without VM hierarchy changes and 1037 without any changes), skipped guests: 32983, duplicated guests: 0
2020-09-03 22:01:50 (+0:00:00.115) INFO: ETL from ImportVmManScans: Success
If there are no such lines, or no imported files/VM Managers, then step 2 did not work.
You can then try to run the fixlet that forces the VM Manager data upload, make a manual import.
Thanks for such a detailed explanation, really appreciate that you taking out time and sharing your advise here.
I just wanted to update that the issue was at the root server end where in the files are not getting processed so we offload the root server by removing the client which are directly connecting to root server and then we restarted the fill db to clear the files and it worked.