Introduction to nested virtualization一文介绍了嵌套虚拟化的基本概念。本文介绍的Intel VMCS Shadowing technology这一硬件技术,正是为了提高嵌套虚拟化系统的性能而应运而生的。

1. Overview

The motivation of VMCS Shadowing: Eliminate VM Exits on guest VMCS accesses

2. implementation

  • Shadow VMCS is processor-dependent and must be accessed by L0 or L1 using VMREAD and VMWRITE instructions only
  • To avoid hardware dependencies:
    • Software defined VMCS1→2 format is part of L1 address space
    • Processor-specific shadow VMCS format is part of L0 address space
  • L0 synchronize the shadow VMCS content with the software-controlled VMCS1→2 format
  • Design simplifies live migration of L1, which does not depended on the shadow VMCS layout

3. sync process

  • Before running L2 after switching from L1 we need to update all the changes L1 did, from the shadow VMCS to VMCS1→2
  • Before switching back to L1 after running L2 we need to sync from VMCS1→2 to the shadow VMCS

4. reducing syncing cost

When Intel VMCS shadowing is used, the L0 VMM has no idea which of the more than 130 VMCS fields were accessed, since it was not involved in those accessed. The L0 VMM must therefore synchronize every filed that could have possibly been accessed, even though most of the fields are never touched.

Results from Intel Labs profiling across a wide variety of VMMs, show that approximately 90% of VMCS fields are never read and more than 95% percent are never written. As a result, for most VMMs, a full VMCS synchronization can take approximately 15 times longer than necessary.

Idea: Shadow only the necessary fields

To reduce this synchronization overhead, Intel incorporated an addtional feature into Intel VMCS Shadowing called VMREAD and VMWRITE bitmaps. These bitmaps allow for selective access to the shadow VMCS. The L0 VMM can tune the bitmaps so that the 5-10 percent of VMCS fields that are commonly accessed are written directly to the shadow VMCS, while the very rarely accessed fields are synchronized through the slower path that is managed by the L0 VMM.

By using the VMREAD/VMWRITE bitmaps, the L0 VMM gets the best of both worlds. Nearly all of the accesses go directly to the fast shadow VMCS and very few extraneous fields need to be synchronized.


  1. Making Nested Virtualization Real by Using Hardware Virtualization Features
  2. intel-vmcs-shadowing-paper
  3. Improving-KVM-x86-Nested-Virtualization-Liran-Alon-Oracle
  4. KVM forum 2013 Nested virtualization:shadow turtles