ESXi 6.0 reports “error code: 15” during Remediate update in VUM operation


Another vCenter another ESXi with problems applying last updates.

In this case is a HP DL360 G9 with ESXi 6.0 build 3568940.

Using VMware Update Manager to scan it shows 17 updates to install, stage is 7(the rest are older versions), when remediate the host we get this:

Remediate entity esxi721.localdomain. The host returns esxupdate error code: 15. The package manager transaction is not successful.
Check the Update Manager log files and esxupdate log files for more details.

Again in issue lot of troubleshooting to check were was the problem here.

Looking at the esxupdate.log there is some information about the locker folder:

2016-04-24T15:11:44Z esxupdate: downloader: DEBUG: Downloading from http://esxi721.localdomain:9084/vum/repository/hostupdate/vmw/vib20/tools-light/VMware_locker_tools-light_6.0.0-2.34.3620759.vib…
2016-04-24T15:12:48Z esxupdate: LockerInstaller: WARNING: There was an error in cleaning up product locker:

[Errno 2] No such file or directory: ‘/locker/packages/var/db/locker’
2016-04-24T15:12:48Z esxupdate: esxupdate: ERROR: An esxupdate error exception was caught:

So need to investigate in the ESXi host. In VMware KB about this ‘error 15’ it says to double check the folder/link Locker > Store

I double check the link to see if the link exists and also the folder, and all is ok. Next check locker folder/link and if locker link is valid

Check if store location is correct

All is ok, so need to check locker/packages folder to see if Version(in this case folder 6.0.0) exists.

The folder doesn’t exist, and there is no floppies, vmtools folders that have all the files that ESXi and VUM needs for the updates. In the VMware KB recommends to delete old folder and links and recreate, in this case we don’t need to delete nothing, but to recreate and copy the necessary files(we will use another ESXi host with the same build).

Connecting to another host we will use SCP to copy all files to this ESXi host.

First if  you don’t have your SSH Client enable in the host firewall, you need to enabled to do the next task using SCP command.

To enable SSH Client in the source ESXi host:

Note: Don’t forget to disable SSH Client after do this tasks.

After you run SCP command you will be prompted for the root password of the remote host and once you have successfully authenticated the files will copy.

Only when trying to copy the files we find the real issue. Did not find anything in the logs related to this. Space issue to apply the updates.

So need to double check the root space.

Here I don’t see any issues with the space, but see big files from the Tardisk

Checking filesystems I see that the one is use for Locker is 100% full.

So next step is to find big files logs, and also inside /tmp if there is any dump files, or other big files that are contributing to this issue.

As we can see there is some big temp files in the list, so the next step is to delete some.

Note: Double check which files do you want to delete. Don’t delete any log files that you could need for any troubleshooting or audit.

After deleting the files that we will not need(and also deleted the files that we copy from the previous ESXi host), and also all folders inside Locker/Store folder, we can check the space.

We now have space around 0% and lot of free space.

Well copy the files again from the other ESXi host and finish 100%.

Now using VUM we will scan, stage and remediate the ESXi host and the problem is fix. After a final reboot(from remediate) the ESXi is fully updated.

Hope this can help.

Backup ASA config with PowerShell

During my years in the networking business one of my frustrations is that it is very hard to backup the configuration of an ASA. There are some commercial products like Solarwinds that can accomplish this goal, but it costs money. An open source alternative like Rancid is also available but is pretty hard to configure.
Determined to find a solution I started searching the internet and came across some PowerShell scripts.  I’m not a PowerShell specialist, but I do know how to put together the separate scripts. So to be clear, I did not invent the scipts I just put them together.

So let’s take a look at the script:

Read-Host  “Enter Password” -AsSecureString | ConvertFrom-SecureString | Out-File c:\<map>\cred01.txt
–I don’t want to sent the password of the ASA user plain over the network. So with the above line I make sure the password is encrypted. It is possible to convert the password back to plain text, but then you’ll need access to the server. So it is not rocksollid save, but safer then sending the password in plain text over the internet. If you make sure that the useraccount only has minimal rights on the ASA, there is minimal change of getting unwanted guests on your ASA. The line converts the plain password to an encrypted password and writes it to a .txt file.

$ASApw = Get-Content “c:\<map>\cred01.txt” | ConvertTo-SecureString #-AsPlainText #-Force
$BSTR = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($ASApw)
$ASApw = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR)
–The above three lines are needed to convert the encrypted password from the credentials file. This is needed because the ASA is unable to read an encrypted password.

$ASAIP = “<ip address>”
$ASAUser = “<username>”
$ASAEnablepw = $ASApw

#Modifies the ASA firewall
#Starts by writing a “commands” file#
echo en >>unicode.txt
echo $ASAEnablepw >>unicode.txt
echo “conf t” >>unicode.txt
echo “no pager” >>unicode.txt
echo “show run” >>unicode.txt
echo “pager 24” >>unicode.txt
echo “copy running-config startup-config” >>unicode.txt
echo “running-config” >>unicode.txt
echo exit >>unicode.txt
echo exit >>unicode.txt

#Converts the file to ASCII format (separate file)#
$lines = gc “unicode.txt
$lines | out-file -encoding Ascii -filepath commands.txt
–The above lines writes the actual ASA commands to the commands.txt file.

#Using the command file and plink.exe connects and runs the commands#
c:/Windows/System32/plink.exe -ssh -l $ASAUser -pw $ASApw $ASAIP -m commands.txt > “c:\<map>\ASA.txt
–To make things work you need to download the Plink tool. It is the command line version of Putty. It can be downloaded for free. I put the tool in de c:\windows\system32 folder, but you can place it everywhere you want. This line writes the configuration of the ASA to an .txt file.

#removes the files it created earlier#
del unicode.txt
del commands.txt

As you can see it’s actually a pretty easy script an above all it’s free.
To make a daily backup, create a task through “Task scheduler”.