Monitoring VMWare ESXi with the ELK Stack

Monitor anything; this is what fills your head after playing with Elasticsearch, Logstash and Kibana.

ELK Stack, meet VMWare Server.  Since VMWare’s ESXi runs on some Linux kernel, it shares the logging facilities we’re familiar with on Linux systems.  Going about forwarding syslog messages to a remote box is a bit different thou.  Luckily, VMWare’s Knowledge Base is very thorough if you know where to look.

1. Enable SSH

This process needs to be done from the command line so ssh needs to be enabled on the server.  This can be done from vSphere Client.

2. Check syslog configuration

~ # esxcli system syslog config get

Local Log Output: /vmfs/volumes/524f647f-31ac7e75-b467-08606ed69d08/systemlogs

Local Logging Default Rotation Size: 1024

Local Logging Default Rotations: 8

Log To Unique Subdirectory: false

Remote Host:

If Remote Host: means there is no syslog forwarding set.

3. Enable Syslog forwarding

~ # esxcli system syslog config set –loghost=’tcp://192.168.1.37:3514′

4. Re-checking syslog configuration

~ # esxcli system syslog config get

Local Log Output: /vmfs/volumes/524f647f-31ac7e75-b467-08606ed69d08/systemlogs

Local Logging Default Rotation Size: 1024

Local Logging Default Rotations: 8

Log To Unique Subdirectory: false

Remote Host: tcp://192.168.1.37:3514

5. Restarting syslog

~ # esxcli system syslog reload

At this point, syslog has been restarted, however, tcp is blocked by firewall by default.

6. Checking host firewall

~ # esxcli network firewall get

Default Action: DROP

Enabled: true

Loaded: true

7. Disabling firewall

~ # esxcli network firewall set -e=false

8. Reloading firewall

~ # esxcli network firewall refresh

It would be wise to just let TCP traffic thru firewall but I had difficulties doing so.  If you are on an internal network, you may be ok.

9. Re-checking firewall

~ # esxcli network firewall get

Default Action: DROP

Enabled: false

Loaded: true

This is all the changes that need to be done on ESXi.  You can log out now.

On Logstash conf file, just add an input of type tcp matching the port defined in host.

10. Editing logstash conf file

tcp {

port => 3514

}

Restart Logstash and let it warm up. If everything went well, we should see events trickling into Elasticsearch. Better yet, lets point a browser to Kibana and see them instead!

Not too bad.  From here, the next logical step would be to create a custom mapping for ESXi host and build a dashboard on top of this