vROps – Virtual Machine CPU Stress

Posted on February 7, 2019 by Calvin Scoltock

Now that I have vRealize Operations Manager (vROps) working again I have discovered that out of the 572 active alerts 359 of them are for “Virtual machine has continuous high CPU usage causing stress”.

I have taken a closer look at these and noticed that one of them was a virtual machine that had been provisioned with SQL Server installed on it but wasn’t being used and there were no user databases on it.

By default, vROps considers a virtual machine to have CPU stress if the CPU demand goes over 70% of the configured CPU. This virtual machine has 2 vCPUs and is running on a host with 2.6Ghz processors in it, so vROps calculates the virtual machine has 5.2Ghz CPU capacity. Therefore, if the CPU demand goes over 3.64Ghz then it will be considered to be stressed.

With the standard vROps policy this alert looks at the worst 1 hour period over the last 30 days. Looking at the CPU demand on this virtual machine over the previous week I could see that between 6pm and 8pm it was using 100% of the CPU most of the time. In fact every Sunday for a period of about 2 hours this virtual machine uses very high CPU, the previous Sunday it was 10am until 12pm and the week before 1pm to 3pm. Further investigation showed that all of the virtual machines had high CPU at some point on a Sunday due to virus scanning. I don’t really care that the virtual machines had high CPU on a Sunday, so I changed the vROps policy to only analyse data for virtual machines Monday to Sunday.

This has reduced the amount of alerts for CPU stress by a considerable amount but I am still getting some due to some of the virtual machines using more than 70% of CPU for a period of 1 hour at some point over the previous 30 days. To reduce the amount of alerts I am receiving for CPU stress further I have configured the policy to consider the entire 30 day range instead of a sliding 60 minute window.

Posted in VMware, vROps | Leave a comment

vRealize Operations Manager

Posted on January 18, 2019 by Calvin Scoltock

I thought it was time to have another look at vRealize Operations Manager (vROps). The last time I used it was probably version 5.7 and called vCenter Operations Manager (vCOps). I have access to a couple of vSphere 6 environments with vROps 6.5 installed:

  • A production environment spread across two sites with a vCenter server at both sites, Site Recovery Manager, some VMware vSphere Metro Storage Clusters, approximately 60 ESXi hosts and 800 virtual machines
  • A Test and Development environment on a single site with a single vCenter server, approximately 35 ESXi hosts and 600 virtual machines

The 1st thing I notice is that the initial screen when you login is completely different to the older version I was used to using.

In vCOps 5.7 it looked something like this

With vROps 6.5 it looks more like this

It took me a while to get used to this new layout, but I soon preferred it. I like to go into vROps, from the home dashboard along the top I can select vCenter Server, Cluster Compute Resource, Host System, Virtual Machine and Datastore one at a time and quickly look at any Health, Risk and Efficiency alerts.

When I first got access to these environments they had not been looked at for a while and vROps was not working in either of them. In the Production environment the issue was with disk space full on the vROps appliance and in the Test and Development environment the DNS server had been changed but vROps had not been updated and therefore it could not resolve any of the host names. Both were fairly easy fixes.

I will dig deeper into vROps and post additional details along my journey.

Posted in Uncategorized | Leave a comment

Copy-VMGuestFile

Posted on October 18, 2018 by Calvin Scoltock

I had a requirement to copy files to some web server VMs running in a DMZ that I could not access across the network. The only connectivity I had to them was via the Virtual Machine Console and logging on as a local user. The files to be copied were from an internal application server, I was adding a vDisk to this application server, disconnecting it and then adding the vDisk to the web servers (as there was a datastore shared between the hosts the application serer and web servers were running on).

I then discovered the PowerCLI command Copy-VMGuestFile. Using the following command I could copy the file I required from the application server to my local workstation.

Copy-VMGuestFile -Source E:\MyApp\WarFile\MyAPP.war -Destination D:\Temp\MyApp.wap  -GuestToLocal -VM MYAPPServer01

I could then copy the file up to the Web Servers with the following command

Copy-VMGuestFile -Source D:\Temp\MyApp.war -Destination D:\Tomcat\webapps\MyApp.war -LocalToGuest -VM MYAPPWeb01 -GuestUser Administrator -GuestPassword xxxxxxxxxx

Note on the command to copy the file FROM the application server I use the parameter -GuestToLocal and on the command to copy the file TO the web server I use -LocalToGuest. Also because the user I am logged on to my local workstation as does not have any permissions on the web server I have to specify a user account and password that does have permissions on the web server, I used the local administrator account, with the parameters -GuestUser Administrator -GuestPassword xxxxxxxxxx, I didn‘t need to specify an account on the command to copy from the application server because the account I was running the PowerCLI session under had permissions to this server.

If the file already exists in the destination you need to include the -Force parameter, e.g.

Copy-VMGuestFile -Source E:\MyApp\WarFile\MyAPP.war -Destination D:\Temp\MyApp.war -GuestToLocal -VM MYAPPServer01 -Force
Copy-VMGuestFile -Source D:\Temp\MyApp.war -Destination D:\Tomcat\webapps\MyApp.war -LocalToGuest -VM MYAPPWeb01 -GuestUser Administrator -GuestPassword xxxxxxxxxx -Force

Instead of listing the Guest Username and Password on the command line you could prompt for the credentials instead, e.g.

Copy-VMGuestFile -Source D:\Temp\MyApp.war -Destination D:\Tomcat\webapps\MyApp.war -LocalToGuest -VM MYAPPWeb01 -GuestCredential (Get-Credential)

As I had a number of web servers to copy the file to and they were all named MYAPPWeb with two digits at the end, e.g. MYAPPWeb01, MYAPPWeb02, e.t.c. I used the following to copy the file to all of them

$webcred = Get-Credential -Message “Please provide credentials for the Web Servers”
Copy-VMGuestFile -Source E:\MyApp\WarFile\MyAPP.war -Destination D:\Temp\MyApp.war -GuestToLocal -VM MYAPPServer01 -Force
ForEach($WebServer in Get-VM MYAPPWeb??) {
Copy-VMGuestFile -Source D:\Temp\MyApp.war -Destination D:\Tomcat\webapps\MyApp.war -LocalToGuest -VM $WebServer -GuestCredential $webcred -Force
}

Posted in Uncategorized | Leave a comment

VMware Cloud on AWS Now Available in EU West (London) Region

Posted on March 20, 2018 by Calvin Scoltock

On the 7th March 2018 VMware released SDDC Version 1.3 of their VMware Cloud on AWS service which added the AWS EU West (London) Region. This expands on the original US West (Oregon) and US East (North Virginia). US West (Oregon) was the 1st region VMware Cloud on AWS was available in from September 2017 with US East (North Virginia) added on 28th Nov 2017 as part of SDDC Version 1.2. Addition European and Asia Pacific Regions will be added in the future. These will probably be Frankfurt soon and then Asia Pacific in the second half of 2018.

This may be a benefit for companies that have restrictions on their data residing with the UK or Europe and also allows European companies to locate their services on VMware Cloud on AWS closer to them.

Posted in AWS, VMware, VMware Cloud on AWS | Leave a comment

VMware Tools Upgrade

Posted on December 11, 2017 by Calvin Scoltock

Recently I had to upgrade VMware Tools on a number of VMs due to ESXi being upgraded on the hosts. I remember a few years ago that there was talk of being able to upgrade VMware Tools without having to restart the VM. Great I thought but I couldn’t remember which version this came in with but as my VMs were on a version of VMware Tools that came with ESXi 6.something and I was going to a later version of 6.something then I was sure that I was going to be able to upgrade VMware Tools on my production VMs without having to restart them.

After a bit of research I discovered that yes you could upgrade VMware Tools without rebooting the VM but any driver updates required a reboot on Windows servers and running the VM for a prolonged time without rebooting following the VMware Tools upgrade was not recommended.

If you just right click on a VM and select “Upgrade VMware Tools” it WILL reboot the VM without warning you, although I think you can put an advanced option in to suppress the reboot, it is something like /reallynoreboot or you can perform a VMware Tools Upgrade from PowerCLI without a reboot e.g.

 Get-VM <vmname> | Update-Tools -NoReboot

You can use what you like to select multiple VMs at once, e.g.

 Get-VM WINSRV* | Update-Tools -NoReboot

(to get all of the VMs starting WINSRV)

 or

 Get-VM –Location Production | Update-Tools -NoReboot

(to get all of the VMs within the Production folder – think this includes subfolder)

or

Get-VM –Location ESXCluster01 | Update-Tools -NoReboot

(to get all the VMs on the vSphere ESXCluster01 cluster)

If you are selecting multiple VMs then it is probably best to also add the -RunAsync option on to the Update-Tools command so that they are updated at the same time and not sequentially, e.g.

Get-VM –Location ESXCluster01 | Update-Tools –NoReboot -RunAsync

You can also update the Tools using VMware Update Manager, there is a Predefined baseline for VMware Tools Upgrade to Match Host. You attached this baseline to a VM folder, cluster or even the top level, i.e. the vCenter Server. When you remediate this baseline you have the option of running it immediately, scheduling it at a specific time, e.g. 7am, or at the next Power Cycle. Of course, even if you have attached the baseline to the top level you can select to remediate at a lower level (even down to a single VM). It would be worth attaching this baseline to the top level if only to use it to scan for VMs that are non-compliant. There is also a baseline for upgrading the VM Hardware and can be used to confirm all the VMs were on the latest VM Hardware.

On some of my VMs I used VMware Update Manager to update VMware tools at next power cycle. When you do this it enables the option in the VM to “Check and upgrade VMware Tools before each power on”, see below

I rebooted one of the VMs to confirm that it would work. It does not update tools before the reboot but after the VM powers back on, so once booted up VMware Tools are updated then the following message is displayed on the console (as you are unlikely to be on the console you will not see the message) and the server reboots a 2nd time.

It appears to leave the “Check and upgrade VMware Tools before each power on” option set on the VM, so next time there is a VM Tools update it will update the VMs again at the next power on of the VM. Not sure if I like the fact this is left on or not. Also, you need to manually do a scan of the baseline against the VMs again for the updated VMs to change from non-compliant to compliant.

When you schedule the VMware Tools upgrade with VUM you can select an option to create a snapshot before upgrading. The good thing about this is that you can set how long the snapshot stays for before it is deleted, the default is 36 hours (maybe this can be changed in the VUM settings) but I reduced it to 12 hours and in one case only 2 hours. All of the snapshots were removed automatically.

I had a problem with using VUM to update some of the servers that I scheduled to run and set them to create a snapshot. Included in the list were a couple of the clustered SQL servers. I could see that the task failed in recent tasks but could not find an error message to explain why but assuming it was down to not being able to snapshot the clustered servers so I ran it again on these two servers WITHOUT a snapshot and it worked.

Using the option to update at next reboot is a risk as it might catch someone out. Also doing a VMware Tools upgrade and suppressing the reboot causes a risk as if there is a problem with one of the new drivers it will not get detected until the server is rebooted by which time you probably don’t remember that VMware Tools was updated so makes troubleshooting the issue harder.

Posted in VMware, VMware Tools, vSphere | Leave a comment

VMware on AWS Update

Posted on March 20, 2017 by Calvin Scoltock

I had a session with Adam Osterhost, @osterholta, from the VMware Cloud SET Team last week to update vExperts on the soon to be released VMware on AWS Service.

This service looks awesome, here are some notes I made from the session.

Clouds are becoming the new Silos, e.g. we are starting to see different clouds such as VMware on-premise private clouds, Microsoft Azure, Amazon Web Services, and Google Cloud Platform all being treated as Silos as they have their own management tools and skills required to look after them.

VMware are working on a Cross Cloud Architecture that will sit above these different clouds to provide a single architecture providing a common set of tools for: –

  • Management and Operations
  • Network and Security
  • Data Management and Governance

They already have VMware Cloud Foundation that sits above the following: –

  • On-Premise vSphere
  • VMware vCloud Air
  • vSphere running on the IBM Cloud

and soon vSphere running on AWS

This will be expanded to allow management of cloud specific non-VMware based infrastructure such as: –

  • AWS
  • Google Cloud Platform
  • Microsoft Azure
  • IBM Cloud

Use cases for VMware on AWS include: –

  • Maintain and Expand
    • Regional Capacity
    • DR and Backup
  • Consolidation and Migrate
    • Data Centre Consolidation
    • Application Migration
  • Workload Flexibility
    • Dev, Test, Lab and Training
    • Cyclic Demand

VMware on AWS will bring operational consistency while leveraging existing customer skill-set and tools across their IT environment.

It will initially be running: –

  • vSphere 6.5 on bare metal AWS hardware and not nested ESXi
  • All-flash VSAN and you will have full control of it to use or switch off features such as
    • Compression
    • Deduplication
  • NSX
  • vCenter

You will be able to use vCenter Linked Mode to your on-premise vCenter.

As VMware on AWS is using standard VMware technology then anything that works with vCenter will work with VMware on AWS such as

  • 3rd party products
  • other VMware products such as vRealize Operations
  • your own developed scripts

You will be able to use all of the AWS Services with VMware on AWS such as: –

  • Amazon EC2
  • Amazon S3
  • Amazon RDS
  • Amazon IoT
  • Amazon Direct Connect
  • Amazon IAM

As VMware on AWS will be a fully VMware managed service and they will control when vSphere is upgraded then they will introduce new functionality to allow vCenter Linked Mode to work with the same version and the previous version, i.e. on day 1 you will be able to use linked mode with VMware on AWS running 6.5 with your on-premise 6.0 vSphere infrastructure. When vSphere vNext comes out and the VMware on AWS is upgraded to it then as long as you have already upgraded your on-premise infrastructure to 6.5 then the Linked Mode will still work. This gives you more control over when you upgrade your on-premise infrastructure instead of forcing you to upgrade your on-premise when VMware on AWS is upgraded.

For best functionality, it is recommended that you have your on-premise infrastructure running the same version as VMware on AWS so that you can utilse the new functionality in the later version.

For VMware on AWS, as new VMware features only need to be tested on one type of hardware then this will allow VMware to introduce new functionality into the VMware on AWS Service before it is generally available for on-premise SDDC.

VMware are currently working with a few customers to develop the VMware on AWS Service and there will be a beta program opening up in the Spring with General Availability planned for Mid 2017

The service will go live in Virginia first and then EMEA, probably UK or Ireland, but will expand to all AWS regions/zones.

With the initial offering all hosts will have to be in the same Amazon Zone but at a later date you may be able to split across AZs for availability.

The VMware on AWS service will be sold on the number of hosts you require, not on the number of VMs you want to run. This will allow you to oversubscribe the hosts if you wish to run more VMs. The minimum number of hosts that can be purchased is likely to be 4 and will scale up. There was mention of up to 16 hosts on the session with Adam but the final details have not been worked out yet so it may be even more than that, he did show screenshots displaying options for 64 hosts.

The cost of the service will work out less the longer you commit, e.g. if you purchase a 2 year service then the cost per month will be less than if you only commit to the service for a single month.

This will be a VMware Service, delivered, operated, sold and supported by VMware. If you need support you contact the same VMware Global Support that you contact when you need support with you on-premise VMware infrastructure. You never need to contact Amazon, if there is an issue with the AWS side of things then VMware will deal directly with Amazon and therefore you are never passed backwards and forwards between the two with them pointing the finger at each other.

There will be a new web portal for requesting the service which will have a full RESTful API from Day 1 allowing features such as provisioning, scaling and billing. This web portal will be based on HTML5 for the best performance and good response times. You will be able to use the Flex Client but 90% of the functionality will be available from the HTML5 client and this should be enough for most people.

You will be able to right click on your AWS vSphere Cloud Cluster and select resize, select how many hosts and within minutes, yes minutes, you get the new capacity, i.e. additional hosts. Imagine doing this with your on-premise cloud. How long does it take to expand a vSphere Cluster on premise from the day you decide it needs to be expanded? You need to get a specification for the new hosts, qet a quote from your hardware provided, order the hardware, install the hardware, configure the hardware, install and configure ESXi, e.t.c. it can take weeks to do this. Remember the days when you had to do this every time you needed a new server and now you just provision a new VM in minutes. With VMware on AWS adding additional capacity will be like adding new VMs.

You will be able to utilise Elastic DRS which can resize your VMware on AWS Cloud by adding or removing hosts depending on workload resource requirements.

There will NOT be a dedicated Management Cluster so vCenter, NSX Manager, e.t.c. will be running on the same hosts as the VMs you want to run in this cloud but in different resource pools.

If you do not currently have NSX then you will be able to deploy a no-charge NSX Edge appliance within your on-premise infrastructure to allow functionality with VMware on AWS.

When VMware upgrade the VMware on AWS service they will add an extra host into your cluster so that one host can be taken out of service to be upgraded so you do not lose any capacity.

The specification of each host will be very similar to the Amazon I3 specification.

You will be able to vMotion between your on-premise infrastructure and VMware on AWS.

For more updates on VMware on AWS you can follow @vmwarecloud on twitter or search for the hashtag #VMWonAWS

Posted in AWS, VMware, VMware Cloud on AWS | Leave a comment

VMware on AWS

Posted on February 24, 2017 by Calvin Scoltock

Coming soon VMware on AWS.

Soon you will be able to run VMware’s Software Defined Data Center (SDDC) including vSphere, vSAN and NSX on the Amazon Web Services (AWS) Cloud. It is currently in Technology Preview but expect more details regarding general availability soon.

This will mean that you will be able to run any application across vSphere based on premise private cloud, public and hybrid clouds environments.

It will be sold and supported by VMware as an on-demand service utilising Amazon’s global enterprise-grade secure highly-scalable cloud resources.

Some of the customer scenarios this is targeted at are: –

  • Application Development
  • Testing
  • Disaster Recovery
  • Geographical Expansion
  • Burst Capacity
  • Data Centre Migration

You will be able to manage it from an existing vCenter Management Interface.

By utilising VMware on AWS should allow companies to free up time spent managing infrastructure to spend more time innovating.

Dr. Matt Wood, AWS GM – Product Strategy, and Mark Lohmeyer, VMware VP Products, demonstrate how a VMware SDDC can be spun up in a few clicks and a matter of minutes in the video below. It is simply a case of selection a geographic location for the Data Center you want to spin up, the size of the Data Centre and how you want to pay for it. They also demonstrate how you can vMotion a Virtual Machine from an on-premise private VMware cloud to VMware on AWS just as you would vMotion on VM between hosts within your own private vSphere environment with no downtime.

//players.brightcove.net/1534342432001/Byh3doRJx_default/index.html?videoId=5179048452001

It looks like this will also include Elastic DRC, where a new host will be spun up on AWS when there is a shortage of resources to satisfy all of the active workloads and DRS will rebalance the workloads across the infrastructure including the new host(s).

For more information see https://www.vmware.com/cloud-services/vmware-cloud-aws.html

Posted in AWS, VMware, VMware Cloud on AWS | Leave a comment

VMware Snapshot Performance Impact

Posted on February 16, 2017 by Calvin Scoltock

A number of people have asked me recently about the performance impact of having VMware snaphots on a VM. So here is my understanding of the performance impact of having snapshot(s) on a VM.

The issue with having a snapshot on a VM is that when a block of data is changed instead of changing this in the main vmdk file it is written to a delta file.

The delta files are not pre-allocated as the main vmdk file is (assuming it is thick provisioned); so each time you have to write to the delta file first off the delta file has to be extended and then the data written. Well not exactly every time data is written, I think it extends the delta files in 16MB chunks, so only has to extend for every 16MB of new blocks that have changed. If you change the same block of data multiple times after a snapshot is created I believe it overwrites the block in the delta file so does not need to extend the file for blocks that are changing multiple times. When the files are extended it locks the VMFS volume (or it used to, not sure if this is still the case with vSphere 6.x) so it is not just the VM with the snapshot on it being impacted but also any other VM on the same datastore.

This is the reason a thick provisioned vmdk provides slightly better performance than a thin provisioned vmdk; as the space is pre-allocated in a thick provisioned vmdk therefore the file does not need to be extended to write additional data to it. If performance of the vmdk is really important then you should create eager zeroed thick provisioned disks as all blocks are zeroed in the vmdk at the time of creation, with the default of lazy zeroed thick provisioned disks each time a block is written to for the first time it needs to be zeroed before the data is written to it; don’t ask me why it needs to be zeroed and then overwritten as I don’t have a clue! Just know that eager zeroed is supposed to be better but takes longer to create the vmdk in the first place.

When you want to read a block of data from a VM with snapshot(s) on it the vmkernel needs to work out if the block you want to read is in the delta file or in the base vmdk.

I have heard that the more snapshots you have on a VM the bigger the impact on performance and have seen a VMware KB (https://kb.vmware.com/kb/1025279) suggesting that you should not have more than 32 and for better performance only 2 or 3.

This extra processing has an impact on the VM disk access times, I suspect not a great lot of impact but it is not as quick as going directly to the base vmdk to read and write blocks of data. I think you will only ever notice it, if at all, with very high disk I/O intensive workloads.

When I get change I will build a test environment and run some performance tests to see how much of an impact snapshots have, although it will probably depend on your disk subsystem.

Posted in VMware, vSphere | Leave a comment

VMware vSphere 6.5

Posted on November 17, 2016 by Calvin Scoltock

VMware vSphere 6.5 was released earlier this week, it was announced on 18th Oct 2016 at VMworld Barcelona but was officially available for download from Tuesday this week (15th Nov 2016).

Some of the new features I got to look at as part of the beta program and thought were good additions to the product were: –

  • Platform Services Controller (PSC) HA – in vCenter 6.0 we could implement HA for the PSC but this required complex configuration and the use of a load balancer, with 6.5 no load balancer is required and the configuration is much simpler.
  • vCenter HA – in the past we have had things such as VMware vCenter Heatbeat to provide HA of the vCenter but that was withdrawn some time again. Now we have an easy to setup vCenter HA solution.
  • VMware Update Manager (VUM) now included in the vCenter Server Appliance so you no longer require a Windows Server for VUM.
  • vSphere HA Orchestrated Restarts – you can now configure dependencies between VMs, e.g. an App server will not restart until a SQL server it is dependent on has been restarted.
  • Additional HA Restart Priorities – up to vSphere 6.0 you could configure one of 3 restart priorities for a VM, other than disabled, (high, medium, low). Now you have 2 more (highest and lowest) making a total of 5 – similar to Site Recovery Manager (SRM).

There are loads more new features but these are a few that I wanted to pick out and will dig unto them in greater detail in later articles.

Posted in VMware, vSphere | Leave a comment

vCenter 5.5 to 6.0 Migration

Posted on September 26, 2016 by Calvin Scoltock

When vSphere 6 was released the vCenter Server Appliance (VCSA) was brought in line with the traditional Windows based vCenter Server. This meant that the appliance version of vCenter was the recommended version to implement. But how do you get your Windows vCenter 5.5 server transitioned to a vCenter 6 appliance? VMware have now released a migration tool from a Windows vCenter 5.5 server to a vCenter 6.0 appliance, known as vSphere 6.0 Update 2m.

It should be noted that it ONLY migrates from 5.5 to 6.0. Therefore if you are still running 5.0 or 5.1 then you will need to upgrade to 5.5 BEFORE being able to use this migration tool. It does, however, work with ANY update of 5.5.

The following items are migrated with this tool from 5.5 to 6.0

  • Configuration
  • Inventory
  • Alarm data
  • Historical and Performance data (optional)

The migration tool installs a VCSA preserving the identity of the previous windows vCenter, e.g. IP address, name, certificates and UUID.

One thing to consider is VMware Update Manager (VUM) as you still require a Windows Server for this. If you already have this installed on a separate server to your vCenter 5.5 Server then you are good to go. However, if VUM is installed on the SAME server as your vCenter 5.5 then you will want to move it to a separate server BEFORE you start the migration process. The same goes for other VMware solutions such as SRM, NSX, vRops, e.t.c. If they are installed on the same Windows server as the vCenter 5.5 server then they need to be moved off to a separate server to continue to work following the migration.

The data will be migrated to an embedded vProstgres database on the VCSA even if the Windows vCenter 5.5 server was using an external database.

If you vCenter 5.5 SSO was also running on the same server as your vCenter Server then the migration tool will implement an integrated Platform Service Controller (PCS) within the new VCSA. If you had an external SSO to your vCenter Server then the migration tool will install an external PCS for the VCSA.

Plan for some down time while you migrate your vCenter Server to a 6.0 appliance. For guidance on how much time you require for this migration see https://kb.vmware.com/kb/2146420.

More information on using the vCenter Server 5.5 to vCenter Server Appliance 6.0 U2m can be found in this FAQ https://kb.vmware.com/kb/2146439.

If you are already using a Windows vCenter 6.0 Server then look out for a migration tool to allow you to migrate from this to a vCenter Server appliance to be released in the future.

Posted in VMware, vSphere, vSphere 6 | Leave a comment