Fix: PVS Target VMs not booting after upgrade Citrix XenServer 6.5 to Citrix XenServer 7.x

Last week @R_Kossen and I (@pvdnborn) had a project for upgrading a Citrix XenDesktop 7.8 with Citrix XenServer 6.5 to Citrix XenDesktop 7.13 with Citrix XenServer 7.1. During the Citrix XenServer 6.5 to Citrix XenServer 7.1 upgrade we had an issue with the PVS Target devices which I want to share with you.

Before we started the upgrade of XenDesktop and XenServer, we upgraded all the PVS images with: Citrix XenDesktop 7.13 VDA, Citrix Provisioning Services 7.13 Target Device software and XenTools/Management agent for XenServer 7.1. When we successfully finished the upgrade of XenServer 6.5 to XenServer 7.1, we had problems starting all of our VDI and Session Host PVS Target Devices VMs. We had the following issues: 

• Some PVS Targets didn’t boot at all: “BNIStack failed: network stack could not be initialized”
• Some PVS Targets did boot very slow and had Realtek network adapters in Windows Device management.
• XenTools/Management agent didn’t initialize.

During the troubleshoot I realized that this behavior looks very similar to virtual hardware differences between the image and target VM. As preparation we had upgraded the PVS-images in the test environment, which has one XenServer 7.1 host. We decided to export a target device VM from this test XenServer 7.1 pool and import it to our upgraded XS6.5 to XS7.1 production pool. The image with the “BNIStack failed” error was now booting perfectly on the upgraded production host. Even the XenTools/Management agent initialized and we have a XenServer PV Network device instead of the Realtek NIC. Due to this tests we agreed that there is a difference in the virtual hardware configuration between the VMs which is causing boot issues and the imported VM from the XenServer 7.1 test pool.

We noticed a difference of the “Virtualization State” in the XenCenter GUI of the VM which didn’t boot and the VM which booted successfully.

The Virtualization state of the VM with boot issues is “Not able to receive updates from Windows Update”
 
The Virtualization state of the VM starting correctly is “Able to receive updates from Windows Update”

When connecting with SSH to the pool master, we’ve also noticed a difference between the “hardware-platform-version” and “has-vendor-device” per VM. To get all the properties of the VM use the following command: “xe vm-param-list uuid=<uuid of VM>”

The VM-Parameter list of the VM with boot issues:

The VM-Parameter list of the VM starting correctly:

To fix the boot issue, the “has-vendor-device = true” value is needed for all the VMs with boot issues after the XenServer 6.5 to XenServer 7.x upgrade. The “hardware-platform-version” and the “Virtualization state” is also modified by updating the “has-vendor-device” property.

Set the “has-vendor-device” property with the following command: “xe vm-param-set has-vendor-device=true uuid=<UUID of VM>”

To get a comma separated list UUID’s of all the VMs and Templates in the pool. Execute the following command on the pool master: “xe vm-list –minimal”

Use the comma separated UUID list to set the “has-vendor-device = true” property to all the VMs in the pool.

Be patient: Citrix XenServer 6.5 upgrade to Citrix XenServer 7.1 stuck at 72% for an hour

Last week @R_Kossen and I (@pvdnborn) had a project for upgrading a Citrix XenDesktop 7.8 with Citrix XenServer 6.5 to Citrix XenDesktop 7.13 with Citrix XenServer 7.1. During the Citrix XenServer 6.5 to Citrix XenServer 7.1 upgrade we had an issue which I want to share with you.

For the upgrade to Citrix XenServer 7.1 we used a “Rolling Pool Upgrade” with the XenServer media on a NFS Export. During the Rolling Pool Upgrade the “Completing installation…” stuck at 72%. A “Rolling Pool Upgrade” is always starting with the pool master, so I was worried about this and expected the pool master upgrade will fail.
 

When switching to another TTY (by pressing ALT+F3), I’ve noticed progress in changing permissions (chmod) of messages files on the filesystem. I think these messages are files of the XenServer performance data which are migrated from the old XenServer 6.5 partition to the new XenServer 7.1 partition. The XenServer hosts of this customer are in production for years now, so there are a lot of messages to “upgrade”. After waiting for about one hour per host the “Rolling Pool Upgrade” completed successful.

If you are experiencing this issue, be patient and press ALT+F3 to see the progress at this 72% stage. Depending on the amount of messages and the production time of your host the duration of this process is between one second and ages. We waited for 1 hour per host and 9 hours for all the hosts in the pool.

At the moment, I didn’t figure out yet how to clean-up these messages in XenServer 6.5 before upgrading to XenServer 7.x.