13/02/2017
Blog technique
Virtualization Based Security – Part 2: kernel communications
Adrien Chevalier
This blog post is a second article covering Virtualization Based Security and Device Guard features. In the [first part]({filename}/content/article11.md), we covered the system boot process, from the Windows bootloader to the VTL0 startup. In this part, we explain how kernel communications between VTL0 and VTL1 actually work. As they use hypercalls to communicate, we will first describe the Hyper-V hypercalls implementation, then how the kernels use them to communicate. To finish with, we list all the different hypercalls and secure service calls we have identified during this work.
Hyper-V hypercalls
Kernel communications between VTL0 and VTL1 use Hyper-V hypercalls. These hypercalls are performed using the VMCALL instruction, with the hypercall number in the RCX register, and RDX pointing to a Guest Physical Page (GPA) which contains the parameters. If the 0x10000 RCX bit is set, the hypercall is a “FAST” hypercall, and the parameters (and the return values) are stored in R8, RDX and XMM registers (size lays on the R9 register). In order to perform the call, Windows uses a “hypercall trampoline”, which is a small *fastcall *routine which basically executes a VMCALL and RET.
This routine is stored in the “hypercall page”. This page contains at this day five trampolines, and is given by Hyper-V at its startup to winload.efi, which will copy it in the VTL0 and VTL1 address spaces. The main difference between these five trampolines is that the first one just does a VMCALL/RET, but the four next ones (they are defined contiguously) store RCX into RAX, and enforce a fixed value into RCX. The second and the third enforce RCX to 0x11, and the next ones to 0x12.
These four trampolines are actually used by the different VTLs. Each kernel may “ask” Hyper-V for the 0xD0002 Virtual Processor Register value (internal Hyper-V values which can be queried or set using their identifier) using a dedicated hypercall, which will return two offsets. These offsets are hypercall page related, and will be used by the kernel to call the correct trampolines. Actually, VTL1 and VTL0 use the 0x11 trampoline to communicate with each other, and the VTL1 uses the 0x12trampoline to finalize its initialization.
As x86 and x64 hypercalls does not behave the same way (x86 hypercalls numbers are set in the EAX register), the page also embeds x86 trampolines and HyperV caches two 0xD0002 virtual registers and serves them regarding the caller’s processor mode.
The hypercall page’s content can be represented as:
VMCALL <-- first trampoline
RET
MOV ECX, EAX <-- second one (enforces 0x11, x86)
MOV EAX, 0x11
VMCALL
RET
MOV RAX, RCX <-- third (0x11, x64)
MOV RCX, 0x11
VMCALL
RET
MOV ECX, EAX <-- fourth (0x12, x86)
MOV EAX, 0x12
VMCALL
RET
MOV RAX, RCX <-- fifth (0x12, x64)
MOV RCX, 0x12
VMCALL
RET
db 0x00
db 0x00
db 0x00
db 0x00
db 0x00
db 0x00
...
...
db 0x00
We therefore have five trampolines, at offsets 0x00, 0x04, 0x0F, 0x1D and 0x28. Note that their contents can be easily retrieved in Windows crashsdumps using WinDbg or within the Hyper-V binary (hvix64.exe/hvax64.exe for intel/amd) inner code.
Remark: several hypercalls may specify a data size in the 12 least significant bits of the RCX most significant DWORD. These sizes are not the data length in bytes, but are related to the current call, and may indicate entries count, etc.
For a hypercall example, the VTL1’s ShvlProtectContiguousPages hypercall (12) parameter is a structure which follows this scheme:
typedef struct _param {
ULONGLONG infinite; // always 0xFFFFFFFFFFFFFFFF
ULONG protection_asked; // 0xD EXECUTE, 0xF WRITE
ULONG zero_value; // always 0
ULONGLONG pfns[]; // entries count is set in the hypercall number
} param;
In order to tell Hyper-V the pfns parameter size, RCX high DWORD must contain its elements count. For only one entry and a *FAST *hypercall, RCX value will therefore have to be 0x10010000C.
Secure kernel hypercalls
The two VTL are able to perform multiple hypercalls in order to communicate with Hyper-V. They may perform the same hypercalls, but Hyper-V will refuse several hypercalls if they are called from VTL0. The two VTL also use one dedicated hypercall to communicate with each other. This is summarized in Figure 1 below.
Figure 1: Hypercalls categories
Let’s first describe the “VTL1 to Hyper-V” hypercalls (in green). We will then cover the 0x12 hypercall later.
The VTL1 can use three hypercall trampolines:
ShvlpHypercallCodePage, which is equivalent of the NTOSHvlpHypercallCodePageone (offset 0), and points over the first trampoline;ShvlpVtlReturn, which enforcesRCXto 0x12 and allows for VTL0 and VTL1 communications;ShvlpVtlCall, which enforcesRCXto 0x11 and is only used during the VTL1 initialization.
The last two are retrieved using the 0xD0002 virtual register (through the 24 least significant bits of the ShvlpGetVpRegister return value, each offset is 12 bits length). These two offsets point towards the 0x11 and the 0x12 trampolines.
By the way, the VTL0 NTOS kernel gets its HvlpVsmVtlCallCodeVa value (the hypercall trampoline used for VTL0 to VTL1 communications) using the same process, as any VM gets the same hypercall page from Hyper-V.
The following tables give the possible VTL1 hypercalls, for each trampoline:
ShvlpVtlCall
ShvlpVtlReturn (VTL0 returns/calls)
ShvlpHypercallCodePage (HyperV)
Remark: We wrote an (H) when realizing an hypothesis.
VTL0 to VTL1 transition
Almost all of NTOS “Vsl” prefixed functions end up in VslpEnterIumSecureMode, with a Secure Service Call Number (SSCN). This functions calls HvlSwitchToVsmVtl1, which uses the HvlpVsmVtlCallVa hypercall trampoline (regular hyper-V hypercalls use HvcallCodeVa trampoline). The SSCN is then copied into RAX, and RCX value is set to 0x11.
Hyper-V dispatches the 0x11 hypercall into the securekernel.exe function SkpReturnFromNormalMode, which then calls IumInvokeSecureService (actually we are not sure IumInvokeSecureService is called directly or not, we think that SkpReturnFromNormalMode must be called first in order to make IumInvokeSecureService returning to VTL0 after secure service call completion). IumInvokeSecureService is mostly a big switch/case block, which handles all possible SSCNs.
Finally, the SkCallNormalMode is called, which ends up on SkpPrepareForReturnToNormalMode. Actually, the secure kernel’s NTOS calls can be considered as “fake returns” to VTL0, since they consist in 0x11 hypercalls too.
We have identified all the possible SSCNs from VTL0 in the array below. For each one, we only specified the called functions as their name is generally self-explanatory. The corresponding parameters must be however retrieved manually by reverse engineering the VTL0 callers or the VTL1 callees.
Conclusion
Almost all of NTOS “Vsl” prefixed functions end up in VslpEnterIumSecureMode, with a Secure Service Call Number (SSCN). This functions calls HvlSwitchToVsmVtl1, which uses the HvlpVsmVtlCallVa hypercall trampoline (regular hyper-V hypercalls use HvcallCodeVa trampoline). The SSCN is then copied into RAX, and RCX value is set to 0x11.
Hyper-V dispatches the 0x11 hypercall into the securekernel.exe function SkpReturnFromNormalMode, which then calls IumInvokeSecureService (actually we are not sure IumInvokeSecureService is called directly or not, we think that SkpReturnFromNormalMode must be called first in order to make IumInvokeSecureService returning to VTL0 after secure service call completion). IumInvokeSecureService is mostly a big switch/case block, which handles all possible SSCNs.
Finally, the SkCallNormalMode is called, which ends up on SkpPrepareForReturnToNormalMode. Actually, the secure kernel’s NTOS calls can be considered as “fake returns” to VTL0, since they consist in 0x11 hypercalls too.
We have identified all the possible SSCNs from VTL0 in the array below. For each one, we only specified the called functions as their name is generally self-explanatory. The corresponding parameters must be however retrieved manually by reverse engineering the VTL0 callers or the VTL1 callees.
Update
Thanks to a reader, we identified that we misread the HyperCallPage contents, and made a few modifications. We thought that there were all x64 ones, and that HyperV served one or the other depending on the VTL level, and inverted the values.
Disclaimer
These findings have almost all been retrieved by static analysis using IDA Pro. We apologize if they contain mistakes (actually they probably do), and ask the readers to take this “as it is”. Any helpful remark or criticism is welcome, just email us at etudes{at}amossys.fr!
