Tuesday, February 9, 2016

ESX Syslog - Loud and Proud

We recently started pumping our ESX syslog data to our centralized syslog server and quickly found out that ESX likes to talk. We noticed our ESX servers were clocking hundreds of millions of logs, a lot of info which we probably don't need. What we found was by default ESX is set to send verbose logs... which is essentially Debug and above.

See the below table for all severity levels and a short description.
This table is borrowed from the Syslog Wikipedia page...

Severity level

0EmergencyemergSystem is unusableThis level should not be used by applications.
1AlertalertShould be corrected immediatelyLoss of the primary ISP connection.
2CriticalcritCritical conditionsA failure in the system's primary application.
3ErrorerrError conditionsAn application has exceeded its file storage limit and attempts to write are failing.
4WarningwarningMay indicate that an error will occur if action is not taken.A non-root file system has only 2GB remaining.
5NoticenoticeEvents that are unusual, but not error conditions.
6InformationalinfoNormal operational messages that require no action.An application has started, paused or ended successfully.
7DebugdebugInformation useful to developers for debugging the application.

To tone this down a bit you can change the level at which ESX sends logs.
For ESX 3.5/4.1 see the following link...

For ESX 5.1 +

Using vSphere click on a host to manage then choose the configuration tab of the host

Under Software | Advanced Settings | Config | HostAgent | log

Decrease the config.HostAgent.log.level to 'panic', 'error', 'warning or 'info' rather than 'verbose.

Exchange 2013 Cumulative Update - This Account is not...

Exchange 2013 CU10

When trying to install Exchange 2013 Cumulative Update 10(CU10) we received an error reporting that the installing users account is not a member of schema admins, other elevated groups, that AD doesn't exist, or that the forest level is not 2003 or later. 

We verified that we had proper permissions on the account, AD exists ;-), and that the forest level was at 2012. So none of these applied... :-(

Exact errors from the ExchangeSetup.txt log file

[1] Failed [Rule:SchemaUpdateRequired] [Message:The Active Directory schema isn't up-to-date, and this user account isn't a member of the 'Schema Admins' and/or 'Enterprise Admins' groups.]

[1] Failed [Rule:AdInitErrorRule] [Message:Setup encountered a problem while validating the state of Active Directory: Exchange organization-level objects have not been created, and setup cannot create them because the local computer is not in the same domain and site as the schema master.  Run setup with the /prepareAD parameter on a computer in the domain YourDomain and site YourDomainsSiteName, and wait for replication to complete.  See the Exchange setup log for more information on this error.]

[1] Failed [Rule:ForestLevelNotWin2003Native] [Message:The forest functional level of the current Active Directory forest is not Windows Server 2003 native or later. To install Exchange Server 2013, the forest functional level must be at least Windows Server 2003 native.]

[1] Failed [Rule:CannotAccessAD] [Message:Either Active Directory doesn't exist, or it can't be contacted.]

Our Remedy

Depending on the current state of your Schema and what updates you might have applied Active Directory may need to be prepped before the cumulative update may be installed. Which was the case for us but it wouldn't let us update the schema from any server except the schema master.

  1. Find your schema master. To find the schema master open an elevated command prompt and run: netdom query fsmo
  2. Login to the schema master server, and copy and extract the CU to a local drive
  3. Open an elevated command prompt to the location of the CU (this will not work from Powershell) and run: .\setup.exe /Preparead /iacceptexchangeserverlicenseterms
  4. Give it a couple minutes to complete preparing AD and you'll be good to go to retry the update on your Exchange server/s
  5. Run the CU setup.exe from the Exchange 2013 server/s and await completion.

Mobile Device - Access Denied

Having trouble allowing access to ActiveSync for a mobile device?  You set Allowed through the GUI but it comes back Access Denied?

Try Powershell…

Start by discovering the DeviceID of the stubborn device with a Get-MobileDeviceStatistics –Mailbox user@domain.com command. 

You’ll be looking for DeviceID and the current DeviceAccessState.

Get-MobileDeviceStatistics -Mailbox jdoe@domain.com

Once you know the DeviceID you can enter the following command to allow that device to utilize ActiveSync. Note the DeviceID at the end of the command.

Set-CASMailbox -Identity jdoe@domain.com -ActiveSyncAllowedDeviceIDs G5CV17DITH25F0P91LHFG46FMG

To verify your work rerun the Get-MobileDeviceStatistics command to see if DeviceAccessState now shows Allowed.

Get-MobileDeviceStatistics -Mailbox jdoe@domain.com

Monday, February 8, 2016

Windows 10 is coming...

Just a warning to the masses.

Microsoft has started sneaking Windows 10 upgrades to user PCs by switching the Windows 10 upgrade from an "optional update." to a "Recommended Update" in Windows Updates.

Microsoft launched Windows 10 earlier last year and offered the free upgrade for Windows 7, 8 and 8.1 PCs. If you have Automatic Windows Updates enabled on your Window 7, 8 or 8.1 PCs and set to install critical updates and security patches, you should watch out because Windows Update might start upgrading your PC to the latest version of Windows 10. If you don't keep an eye out and deselect the now recommended update the Windows 10 upgrade process might just be downloaded and installed without your notice.

Microsoft states that you won't be forced to upgrade to Windows 10 as there will still be a EULA prompt that will require you to click through and confirm the Windows 10 upgrade after the files have been quietly downloaded to your computer. But we know no one reads the fine print we all just click accept.

If the Windows 10 upgrade is accidentally completed, there is still a way to revert back to your previous OS. Your newly upgraded OS offers a 31-day grace period in which you will be able to revert to your old installation after trying Windows 10. 

Anyways, just an FYI...

Windows 10 - Handle is Invalid at Logon

Windows 10, VMWare ESXi 5.5 Virtual Machine, Citrix VDA 7.6.300 or 7.7  – Handle is Invalid

Windows 10 updates installed on a VM with Citrix VDA 7.6.300 or 7.7 and when a domain log in is attempted a "Handle is Invalid" error message is received. If logging in as a local admin, a welcome circle spins and the lock screen appears.

The newest 7.8 VDA fixes this issue.

If you can't upgrade to 7.8 VDA - 

Hold the SHIFT key and click power, restart. This will take you to the Win 10 Troubleshooting screen.

Click the Troubleshoot icon.

Click Advanced options.
Click Startup Settings

Click on Restart.

Hit the F4 key to go into Safe Mode.

Log in as the local administrator.

Even safe mode is broken at this point – the start menu and search options won’t work, and some icons from the task bar will be missing. By default, there is a Windows Explorer icon in the task bar, and it works – click on it.

From Windows Explorer, right click on This PC, and select Properties.

In the properties window click on Control Panel in the address bar to get to the control panel.

The search function works from the control panel – search for Installed Updates or Windows Updates and click on View Installed Updates.

There are several KBs that cause the issue: Find the following KBs and uninstall them if present:

After the updates are uninstalled, reboot the virtual machine, and the Handle is Invalid error will be gone. During the reboot it will look like updates are being installed instead of uninstalled, but once logged double check installed updates to make sure these are gone.

Citrix has a hotfix for the Windows 10 VDA - CTX205784. This might prevent the error from returning or further break things. I've had it do both. Use at your own risk.

View Known Issues from CTX205398.