The topic of Terminal Server/Citrix XenApp workloads has been at the forefront of our attention since Meltdown and Spectre patches were released. The potential for slowdowns after applying Meltdown and Spectre patches prompted us to investigate the matter for ourselves, and we released some initial findings for VDI that showed a noticeable CPU impact (with the caveat of impact being workload-dependent). Now that we’ve had a chance to finalize some similar analysis for XenApp, we’re ready to share what we’ve discovered. 

Why Test XenApp in Addition to VDI (and in lieu of pure Terminal Services)? 

Simply stated: While a tiny percentage of our customers use pure Terminal Services for RDS and app virtualization, the majority use XenApp, and that's what we've got set up in our labs. We don't mean to "pick on" XenApp per se. Our assumption is that whatever impacts we observed on XenApp performance are likely underpinned by Terminal Server characteristics. Further, we're interested in illustrating our thought process on how VDI workloads are distinct from XenApp or Terminal Service workloads since they have unique characteristics and we anticipate will show different impacts. 

Also, because this is a blog post and not a white paper, some of the technical descriptors herein are over-simplifications, so we will follow up with a technical white paper and best practices for measuring performance impacts of OS patching.  

But for now, I'd like to offer a few words on some of the fundamental differences between RDS host machines and VDI to help illustrate why Meltdown and Spectre patches impact each differently. To simplify, the concept of RDS (or as it has been historically known, Terminal Services) is that you’ve got one system (generally a server OS of some kind) acting as a host that services numerous sessions with one parent OS context containing the workload. This introduces some complexities into how that system then services a lot of the activity that happens when you move back and forth between the user and kernel (basically shuffling back and forth between where the construction of the stuff gets processed and where the actual execution itself happens). So, you’ve got a lot of distinct workloads that are taking place on a system simultaneously; these processes share CPU resources and the multiple user contexts that exist on the system itself. This means that on a Terminal Server you can generally expect more of the activities that are impacted by the speculative execution exploitation mitigating updates. 

Compare that to VDI where you’ve got a slight but significant difference. Essentially, you’ve got a similar idea where ultimately there is one shared resource (in this case a host system) that’s running multiple workloads. In this case, each VM is running the workload of an individual user context (most of the time, anyway). The net impact of this is that on a per VM basis you’ll see less of a direct impact from the mitigation patches, but in the aggregate you’re going to see a notable amount. What does that all mean? 

The takeaway: Terminal Servers would be expected to see a heavier hit from the mitigation techniques used for Meltdown and Spectre. So, that’s what we wanted to try and witness with some real workload. 

Test Methodology 

As in our initial testing, we were looking to find something that would have a much more notable impact with even a lighter workload, so we decided to test a variety of platforms. One thing we wanted to remain consistent, though, was the workload. This was done with a synthetic transaction tool and focused only on the published browser use case since we’ve found a very significant number of organizations supporting this with XenApp. 

The bulk of our testing used two configurations that are roughly reflective of some common setups we see in the wild, and we tried both the physical and virtual case for both: 

 

Old - Physical 

Patch Tested 

Meltdown and Spectre 

OS 

Windows Server 2008 R2 SP1 

CPU 

Intel Xeon E5650 @ 2.67GHz 

Memory 

32 GB 

Disk 

SSD 

User Density 

20 Sessions 

XenApp 

6.5 

Workload 

Browser only 

 

Old - Virtual 

Patch Tested 

Meltdown and Spectre 

Hypervisor 

VMware ESXi, 6.0.0, 6921384  

Guest OS 

Windows Server 2008 R2 SP1 

CPU 

Intel Xeon E5650 @ 2.67GHz 

vCPUs 

Memory 

16 GB 

Disk 

SSD local storage 

User Density 

20 Sessions 

XenApp 

6.5 

Workload 

Browser only 

 

New(er) - Physical 

Patch Tested 

Meltdown and Spectre 

OS 

Windows Server 2012 R2 

CPU 

Intel Xeon X7350@ 2.93GHz 

Memory 

144 GB 

Disk 

SSD 

User Density 

20 Sessions 

XenApp 

7.7 

Workload 

Browser only 

 

New(er) - Virtual 

Patch Tested 

Meltdown and Spectre 

Hypervisor 

VMware ESXi, 6.0.0, 6921384  

OS 

Windows Server 2012 R2 

CPU 

Intel Xeon X7350@ 2.93GHz 

vCPUs 

Memory 

72 GB 

Disk 

SSD local storage 

User Density 

20 Sessions 

XenApp 

7.7 

Workload 

Browser only 

 

Finding: Meltdown and Spectre Patches Increase CPU Usage on XenApp Machines 

Case 

Platform 

Impact 

Old 

 (Server 2008 R2 and older proc) 

Physical 

20% 

Virtual 

22% 

Newer  

(Server 2012 R2 and newer proc) 

Physical 

16% 

Virtual 

19% 

 

The physical workload with the largest impact was, as you’d expect, based around older hardware and an older OS (Server 2008 R2). We’ll also touch on another factor, which is that the XenApp machine itself was a VM. We found that on average the total impact on each XenApp server was roughly a 20% increase in CPU overhead. The newer server architecture with Windows Server 2012 R2 had a significantly lower delta, coming in with only a 16% increase. 

XenAppCPU.png

An example run of a pre- and post-update workload for the older server architecture and OS 
An example run of a pre- and post-update workload for the older server architecture and OS 
 

The news for virtual workloads is significantly worse: We found that for the older architecture there was a 22% increase in the total CPU load, and even for the newer architecture there was a 19% increase. 

What conclusions can we draw from this? Well, first things first, even with a relatively light workload, we’ve got a tendency to see a much more dramatic increase in the CPU load evidenced on a XenApp server than on a desktop OS VM. This is most likely due to the processing characteristics of the workload that you see in Terminal Server environments and has some interesting ramifications. The second, and maybe most interesting finding, is that virtualization has a direct impact on the total overhead that you’re going to see on a per XenApp VM basis, and that in turn is going to have some big impacts for density. Per before, your mileage will vary depending on the workload, but if this is an indication expect to see heavier workloads (especially things with more disk throughput) drive some potentially serious increases in load on older processor architectures. 

Microsoft's guidance on the performance impacts of patching is as follows: 

Windows Server on any silicon, especially in any IO-intensive application, shows a more significant performance impact when you enable the mitigations to isolate untrusted code within a Windows Server instance.  

TLDR: These Microsoft patches appears to have much heavier impact on Terminal Server/XenApp server workloads than XenDesktop and Horizon desktop workloads. More modern physical systems experience lower deterioration (our slightly older system clocked 16% CPU increase) than older (our older platform clocked 20% impact). Unsurprisingly, virtual servers have an even higher overhead (16% - 22% depending on the age of the architecture and OS). All workloads are different, so be sure to establish your own baselines.