A Multi-Site vSphere/SDDC Upgrade – A Step by Step Guide

I wanted to share with you the steps required to upgrade a typical multi-site vSphere/ SDDC implementation. This particular implementation had the following products.

 

Product Current version Planned Version
vCenter Server 6.0 U3 6.5 U1
ESXi 6.0 U3 6.5 U1
vSAN 6.2 6.6.1
NSX 6.2.4 6.3.5
vROPS 6.3.0 6.6.1
vRLI 3.3.1 4.5
vDP 6.1.0.173 6.1.5
vSphere Replication 6.1.1 6.5.1
VMware Tools 6.0 U3 6.5 U1

 

In order to understand the upgrade order here is some background information on this fictitious environment.

The environment has a Production site and a DR site. NSX is deployed across both sites using universal objects, SRM and vSphere Replication are in use with vSAN being used locally at each site. It also has a deployment of vROPS and vRLI at both sites.

Note. The following should be taken as a rough guide only please make sure you check the appropriate VMware Guides/KB’s for your product versions before commencing any upgrade.

 

Action

Site

Impact to vSphere

Required

VM downtime

1.      

Carry out Health Check of VC & PSC before starting (Go or No Go)

Both Sites

N/A

Yes

No

2.      

Backup All Components

Both Sites

N/A

Yes

No

3.      

Backup PSC’s with vDP & Data Protect

Both Sites

N/A

Yes

No

4.      

Deploy second PSC at each site – Configure replication see KB 2131191 for justification

Both Sites

vCenter management of ESXi hosts unavailable during upgrade.

Yes

No

5.      

Disable vSphere Replication

Both Sites

No protection of VMs (Backup is the only method of restoring)

Yes

No

6.      

Upgrade the external Platform Services Controller server 6.0.x to vCenter 6.5 for both sites

Both Sites

vCenter management of ESXi hosts unavailable during upgrade.

Yes (if using an external Platform Services Controller)

No

7.      

Upgrade vDP at both sites

Both Sites

Backups unavailable during upgrade

Yes in order to restore in to 6.5 during the upgrade

No

8.      

Upgrade NSX Manager at the DR Site

DR Site

No Changes to DR NSX during Upgrade

Yes

No

9.      

Upgrade NSX Controller Cluster at the DR Site

DR Site

NSX reverts to read only mode Change Window Required

Yes

Yes – No Disruption as long as VM’s don’t move or any changes made

10.    

Upgrade NSX Host preperation at the DR Site

DR Site

Hosts require Reboot

Yes

No

11.    

Upgrade NSX DLR’s at the DR Site

DR Site

Disruption to Service

Yes

Yes

12.    

NSX Edges at the DR Site

DR Site

Outage required while edge is redeployed and upgraded

Yes

Yes

13.    

Upgrade vCenter from vCenter 6.0.x to vCenter 6.5. at the DR Site

DR Site

vCenter management of ESXi hosts unavailable during upgrade.

Yes

No

14.    

Upgrade vROPS

Both Sites

N/A

Yes

No

15.    

Upgrade Log Insight

Both Sites

N/A

Yes

No

16.    

Use vSphere Update Manger to scan and remediate an ESXi host.

DR Site

1 host not available, reduced capacity.

Yes

No

17.    

Repeat steps for remaining hosts in the cluster.

DR Site

One host will always be unavailable while it is being upgraded with vSphere Update Manager.

Yes

No

18.    

Upgrade vSAN at the DR Site

DR Site

Possible Performance and increased risk of failure during upgrade – upgrade could take several days.

Yes

No

19.    

Upgrade NSX Manager at the Prod Site

Prod Site

No Changes to DR NSX during Upgrade

Yes

No

20.    

Upgrade NSX Controller Cluster at the Prod Site

Prod Site

NSX reverts to read only mode Change Window Required

Yes

Yes – No Disruption as long as VM’s don’t move or any changes made

21.    

Upgrade NSX Host preperation at the Prod Site

Prod Site

Hosts require Reboot

Yes

No

22.    

Upgrade NSX DLR’s at the Prod Site

Prod Site

Disruption to Service

Yes

Yes

23.    

NSX Edges at the Prod Site

Prod Site

Outage required while edge is redeployed and upgraded

Yes

Yes

24.    

Upgrade vCenter from vCenter 6.0.x to vCenter 6.5. at the Prod Site

Prod Site

vCenter management of ESXi hosts unavailable during upgrade.

Yes

No

25.    

Upgrade vSphere Replication at Both Sites

Both Sites

Replication unavailable until this points it has to be disabled before starting the upgrade process

Yes

No vSphere Replication protection during upgrade

26.    

Use vSphere Update Manger to scan and remediate an ESXi host.

Prod Site

1 host not available, reduced capacity.

Yes

No

27.    

Repeat steps for remaining hosts in the cluster.

Prod Site

One host will always be unavailable while it is being upgraded with vSphere Update Manager.

Yes

No

28.    

Upgrade vSAN at the Prod Site

Prod Site

Possible Performance and increased risk of failure during upgrade – upgrade could take several days.

Yes

No

29.    

Optional: Update VMware Tools on each VM with a vSphere Update Manager baseline.

Prod Site

Virtual machine reboot.

Recommended

Reboot

30.    

Optional: Update virtual hardware on each VM with a vSphere Update Manager baseline.

Prod Site

Virtual machine shutdown, 1+ reboots.

No

Yes; upgrade during shutdown

 

Further Upgrade Information

vSphere

 

  1. Upgrade all external Platform Services Controller servers running 6.0.x to 6.5.
  2. Upgrade vCenter Server 6.0.x to vCenter 6.5.

This step includes the upgrade of all components and installation of new components that were not previously addressed.

During the upgrade, the vCenter Server will be unavailable to perform any provisioning operations or functions such as vSphere vMotion and vSphere DRS. Once started, the vCenter database is upgraded first, followed by the vCenter binary upgrade. After the upgrade, the hosts are auto-reconnected.

  1. Upgrade vSphere Update Manager(Windows Only, on vSphere 6.5 Update Manager is included with the vCenter Appliance). In the same way as vCenter, the database is upgraded first, followed by binaries. With vSphere 6.5, vSphere Update Manager is configured and administered through the vSphere Web Client.
  2. Use vSphere Update Manager to create a baseline, scan the hosts, and then upgrade the ESXi hosts to version 6.5. A copy of the ESXi 6.5 installation media must be available and added to vSphere Update Manager.
  3. Use vSphere Update Manager to create a baseline for VMware Tools, scan virtual machines, and remediate to upgrade VMware Tools.
  4. Upgrade VMware Virtual Machine Hardware to version 13 (if not already completed previously) to take advantage of new features in vSphere 6.5.
  • This step should not be performed until all hosts have been upgraded to ESXi 6.5, because VMware Virtual Machine Hardware version 13 cannot be run on older hosts.
  1. If applicable, upgrade VMFS volumes on any ESXi host that has already been upgraded, but is not yet running VMFS6.
  • Only VMFS3 and above can be upgraded to VMFS5 on an ESXi 6.x host. Verify that older volumes are upgraded to at least VMFS3 before upgrading to ESXi 6.x.
  • Once a VMFS file system update has been completed, it is not possible to roll back this change. Therefore, hosts not running ESXi 5.x or 6.x will be unable to read these volumes.

 

 

vSAN

 

Before starting the vSAN upgrade process, ensure that the following requirements are met:

  1. The vSphere environment is up to date:
    • The vCenter Server managing the hosts must be at an equal or higher patch level than the ESXi hosts it manages.
    • All hosts should be running the same build of ESXi before vSAN cluster upgrade is started.
      • If the ESXi host versions are not matched, the hosts should be patched to the same build before upgrading.
  1. All vSAN disks should be healthy
    • No disk should be failed or absent
    • This can be determined via the vSAN Disk Management view in the vSphere Web Client
  2. There should be no inaccessible vSAN objects
    • This can be verified with the vSAN Health Service in vSAN 6.0 and above, or with the Ruby vSphere Console(RVC) in all releases.
  3. There should not be any active resync at the start of the upgrade process.
    • Some resync activity is expected during the upgrade process, as data needs to be synchronized following host reboots.
  4. Ensure that there are no known compatibility issues between your current vSAN version and the desired target vSAN version. For information on upgrade requirements, see vSAN upgrade requirements (2145248).
    • If required, update the vSAN cluster to the required build before undertaking the upgrade process to avoid compatibility concerns.

Host Preparation

Ensure you choose the right maintenance mode option. When you move a host into maintenance mode in vSAN, you have three options to choose:

  • Ensure availability
    If you select Ensure availability, vSAN allows you to move the host into maintenance mode faster than Full data migration and allows access to the virtual machines in the environment.
  • Full data migration
  • No data migration
    If you select No data migration, vSAN does not evacuate any data from this host. If you power off or remove the host from the cluster, some virtual machines might become inaccessible.

Exit maintenance mode and resync

  • When the ESXi host is upgraded and moved out of maintenance mode, a resync will occur. You can see this through the web client.
  • Ensure this is complete before moving onto the next host. A resync is occurring as the host that has been updated can now contribute to the vSAN Datastore again. It is vital to wait till this resync is complete to ensure there is no data loss.

 

 

NSX

 

For detailed information please refer to the NSX Upgrade guide.https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.3/nsx_63_upgrade.pdf

 

vROPS

 

For detailed information please refer to the vROPS Upgrade guide.

https://docs.vmware.com/en/vRealize-Operations-Manager/6.6/vrealize-operations-manager-66-vapp-deploy-guide.pdf

 

vRLI

 

For detailed information please refer to the vRLI Upgrade guide.

https://docs.vmware.com/en/vRealize-Log-Insight/4.5/com.vmware.log-insight.administration.doc/GUID-4F6ACCE8-73F4-4380-80DE-19885F96FECB.html

 

vDP

 

For detailed information please refer to the vDP Upgrade guide.

https://docs.vmware.com/en/VMware-vSphere/6.5/vmware-data-protection-administration-guide-61.pdf

 

 

SRM 6.1.1 Released

What’s New in Site Recovery Manager 6.1.1

So SRM 6.1.1 got released yesterday so I thought I would share with you what’s new!

VMware Site Recovery Manager 6.1.1 delivers new bug fixes described in the Resolved Issues section.

VMware Site Recovery Manager 6.1.1 provides the following new features:

  • Support for VMware vSphere 6.0 update 2.
  • Support for Two-factor authentication with RSA SecurID for vCenter Server 6.0U2.
  • Support for Smart Card (Common Access Card) authentication for vCenter Server 6.0U2.
  • Site Recovery Manager 6.1.1 now supports the following external databases:
    • Microsoft SQL Server 2012 Service Pack 3
    • Microsoft SQL Server 2014 Service Pack 1
  • Adding localization support for Spanish language.

See the release notes for further information.

 

 

To TPS, or not to TPS, that is the question?

micron-LRDIMM-module

Something that crops up again and again when discussing vSphere designs with customers is whether on not they should enable (Inter-VM) TPS (Transparent Page Sharing) as since the end of 2014 VMware decided to disable TPS by default.

To understand if you should enable TPS you need to firstly understand what it does.  TPS is quite a misunderstood beast as many people contribute a 20-30% memory overcommitment too TPS where in reality it’s not even close to that. That is because TPS only comes in to effect when the ESXi host is close to memory exhaustion. TPS would be the first trick in the box that ESXi uses to try to reduce the memory pressure on the host, if it can not do this then the host would then start ballooning. This would be closely followed by compression and then finally swapping, none of which are cool guys!

So should I enable (Inter-VM) TPS?… well as that great IT saying goes… it depends!

The reason VMware disabled (Inter-VM) TPS in the first place was because of their stronger stance on security (Their Secure by Default Stance), their concern was a man in the middle type attack could be launched and shared pages compromised. So in a nutshell, you need to consider the risk of enabling (Inter-VM) TPS. If you are running and offering a public cloud solution from your vSphere environment then it may be best to leave TPS disabled as you could argue you are under a greater risk of attack; and have a greater responsibility for your customers data.

If however you are running a private vSphere compute environment then the chances are only IT admins have access to the VM’s so the risk is much less. Therefore to reduce the risk of running in to any performance issues caused by ballooning and swapping you may want to consider enabling TPS, which would help mitigate against both of these.

 

 

 

Intro to Converged Blueprints in vRA 7

Intro to Converged Blueprints in vRA 7

Intro to Converged Blueprints in vRA 7

vRealize Automation 7 delivers a new unified graphical canvas for designing machines, software components and application stacks with an underlying single unified model for both machine and application blueprints for Private and Public Cloud. vRealize Automation 7 also gives you the ability to extend or define external integrations in the canvas through XaaS. In this video we are going to focus on building several types of blueprints to demonstrate this functionality.


VMware Advocacy

Get ready for NSX phase 0: Migrate from a…

A nice article from @Lenzker on how to migrate from a Standard vSwitch to a vDS in preparation for deploying NSX.

Get ready for NSX phase 0: Migrate from a vSphere Standard to a distributed switch

Get ready for NSX phase 0: Migrate from a…

The distributed switch (vDS) is a really nice piece of software within the vSphere environment. It offers a variety of very useful features (NIOC, Health Check, Central management, LACP, LBT, Netflow, and so many (features with fancy abbreviations) more). Especially if you want to move to NSX the distributed switch is mandatory. Since analytics showed me that views about my vDS blogposts are increasing it seems that there is a demand for the vDS within our virtualized environments (it’s still an enterprise+ only feature)


VMware Advocacy

Licensing VMware vSphere for VMware NSX

If you are using vSphere 5.5 Update 3 or vSphere 6 then NSX does not require a vSphere Enterprise Plus licence however if you are using vSphere 5.5 update 2 or earlier then a vSphere Enterprise Plus license is required.

This is because NSX is based on the vSphere Distributed Switch which is part of the Enterprise Plus licence feature set.

Reference: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2135310

Another VMware Cloud in Action — The Hut Group…

Another VMware Cloud in Action — The Hut Group Maximizes Online Retail and Minimizes Revenue Loss with Cloud-Based Disaster Recovery

Another VMware Cloud in Action — The Hut Group…

With more than 60 websites ranging from health and beauty to fashion, The Hut Group needed to reduce downtime to zero. As one of the fastest growing companies in the United Kingdom, The Hut Group has a lot to lose — up to £500,000 a day — if its sites were to go offline. The […] The post Another VMware Cloud in Action — The Hut Group Maximizes Online Retail and Minimizes Revenue Loss with Cloud-Based Disaster Recovery appeared first on VMware vCloud Blog .


VMware Advocacy