Friday, March 25, 2016

Avoid 1603 errors when upgrading Citrix StoreFront 2.x to Citrix StoreFront 3.5

In the last months I’ve upgraded some customers from StoreFront 2.x installations toward StoreFront 3.0, 3.0.1. Today I was facing a StoreFront 2.1 to 3.5 upgrade. My experience with upgrading StoreFront is that it will fail most times (1603 error). So I’ve learned some fixes that I want to share with you to make the upgrade of StoreFront easier.

Log files of StoreFront upgrade are located in %windir%\temp\StoreFront

Customer load balanced StoreFront set-up

  • KEMP Network load balancer for http, https round robin load balancing between StoreFront servers
  • StoreFront Server Group
    • pbo-sf01: Windows 2012, StoreFront 2.1
    • pbo-sf02: Windows 2012, StoreFront 2.1

The upgrade plan

The upgrade plan looks straight forward, VMware snapshots are used as fallback scenario:
1. Disable pbo-sf02 from load balancing
2. Shutdown pbo-sf02
3. Take snapshots of both StoreFront servers (online quiesce snapshot of pbo-sf01)
4. Start pbo-sf02 and upgrade pbo-sf02 to StoreFront 3.5


5. Test StoreFront upgrade by connecting (thin)clients directly to upgraded pbo-sf02 StoreFront server
6. Enable pbo-sf02 in load balancing and disable load balancing for pbo-sf01
7. Shutdown pbo-sf01
8. Take snapshots of both StoreFront servers (online quiesce snapshot of pbo-sf02)
9. Start pbo-sf01 and upgrade pbo-sf01 to StoreFront 3.5


10. Test StoreFront upgrade by connecting (thin)clients directly to upgraded pbo-sf01 StoreFront server
11. Test StoreFront synchronization by clicking "Propagate changes"
12. Restore load balancing

Locked StoreFront files are causing StoreFront upgrade to fail

During upgrade to StoreFront 3.5 you must ensure that the existing StoreFront files are not in use or the upgrade will fail:
  • Make sure there are no client connections to the StoreFront server before and during upgrade
  • Disable all monitoring services on the StoreFront server, periodically monitoring checks can lock files during upgrade
    • Microsoft Monitoring Agent (HealthService)
    • Comtrade service
    • Nagios NSClient++
  • Temporary disable all agentless remote monitoring checks to the StoreFront server:
    • SCOM: Put server in maintenance mode
    • Nagios: Disable checks for during upgrade period
  • Temporary disable virus scan on the StoreFront server
    • On access scanner
    • Web reputation
  • Reboot server before upgrade to ensure all active sessions are closed

Virtual path /Citrix/StoreWeb is already in use

During upgrades I also faced the following error:

An error occurred configuring the installation: System.InvalidOperationException: Physical path: C:\inetpub\wwwroot\Citrix\StoreWeb associated with Virtual path: /Citrix/StoreWeb is already in use.

 
When troubleshooting this error I did not have the path C:\inetpub\wwwroot\Citrix\StoreWeb on StoreFront server pbo-sf02, the upgrade process moved it successfully to C:\ProgramData\Citrix. So why I’m facing this error? The answer is quite simple. At the end of the upgrade process, the Citrix StoreFront website folders/applications (i.e. WebReceiver) are copied back to Internet Information Services website. I figured out that the upgrade process is using the StoreFront Server Group base URL for this. The base URL is configured to desktop.pbo.local which is the DNS A-record of the KEMP load balancer. Because the load balancing is disabled to the server pbo-sf02 I‘m upgrading, the upgrade process is contacting the other pbo-sf01 StoreFront server through the load balancer desktop.pbo.local. On the pbo-sf01 the virtual path /Citrix/StoreWeb exist so the error is legitimate. I can also imagine that if you get load balanced (i.e. by round robin) to the wrong server during upgrade, the same error occurs.


To bypass this error you can add a record for the load balanced address to the local hosts file of the StoreFront server to force traffic to the local IPv4 address. This registration should be removed after the upgrade completes. Windows host file can be found: C:\Windows\System32\drivers\etc\hosts, add line (example upgrade pbo-sf02):
 

10.175.201.118    desktop.pbo.local

Thumbs up

If you have had other issues upgrading StoreFront, please share the fixes with me.

5 comments:

  1. Hi,
    A simple question. Is it correct that if the upgrade fails you just roll back to the snapshot created before you started the upgrade?

    ReplyDelete
    Replies
    1. Yes this is correct. Reverting from a failed upgrade to "before upgrade" snapshot is no problem at all.

      Delete
  2. what about if you do not have the files required to uninstall the previous version? the install fails if you are unable to uninstall.

    ReplyDelete
  3. Thank you pat, your article helped my upgrade smoothly.

    Sathish Kumar

    ReplyDelete
  4. I know this is a little old now, but I've just come across another 1603 issue.

    I’ve managed to fix it, and will share it in the hope that is helps someone else.

    I was trying to upgrade my test 3.8 Storefront to 3.12 LTSR.
    Originally it was a 3.7, which at the time I upgraded to 3.8…all good.
    Upon following this Blog as a good practice guide, I was still getting 1603 Errors in the logs.

    The Error in Question was :

    System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Configuration.ConfigurationErrorsException: An error occurred creating the configuration section handler for citrix.deliveryservices/protocolTransitionSettings: Could not load file or assembly 'Citrix.DeliveryServices.ProtocolTransition.Configuration, Version=3.7.0.0, Culture=neutral, PublicKeyToken=e8b77d454fa2a856' or one of its dependencies. The system cannot find the file specified. (C:\Program Files\Citrix\Receiver StoreFront\Services\ProtocolTransitionService\Citrix.DeliveryServices.ProtocolTransition.ServiceHost.exe.config line 9) ---> System.IO.FileNotFoundException: Could not load file or assembly 'Citrix.DeliveryServices.ProtocolTransition.Configuration, Version=3.7.0.0, Culture=neutral, PublicKeyToken=e8b77d454fa2a856' or one of its dependencies. The system cannot find the file specified.

    So I opened up the “Citrix.DeliveryServices.ProtocolTransition.ServiceHost.exe.config” file, listed above, and it lists version 3.7.0.0, but what was installed was V3.8.
    I performed a Find/Replace for 3.7.0.0 to 3.8.0.0, and saved the file. A little maverick I know, but it’s a test server so it’s no drama if its lost.
    Tried the upgrade again and all went through successfully.
    I can only assume upgrading from 3.7 to 3.8 hasn’t updated this file correctly.

    Anyway, I hope this may be of use to someone else stuck in the mud!

    M

    ReplyDelete