[vbox-dev] Question regarding channel callbacks between guest and host

[Jaeyong Yoo]

Hello VirtualBox!

While I'm reading the source, I found the following sequence of function call:

[In xpdm directory] VBoxMPHGSMIDpc () -->
VBoxHGSMIProcessHostQueue () -->
hgsmiHostCommandQueryProcess() -->
hgsmiHostCmdProcess () -->
HGSMIBufferProcess () -->
hgsmiBufferProcess () -->
HGSMIChannelHandlerCall () -->
pfnHandler() ==
[Host side] vbvaChannelHandler

It looks like a guest-side function (in VGA driver) VBoxMPHGSMIDpc calls the host-side function (in VGA emulator) vbvaChannelHandler.
How is this possible since host and guest are in different address space (if we are using nested paging table)?
Or, I'm feeling like I miss something.

Thanks,
Jaeyong

[Michael Thayer]

Hello Jaeyong,

Post by Jaeyong Yoo
[In xpdm directory] VBoxMPHGSMIDpc () -->
VBoxHGSMIProcessHostQueue () -->
hgsmiHostCommandQueryProcess() -->
hgsmiHostCmdProcess () -->
HGSMIBufferProcess () -->
hgsmiBufferProcess () -->
HGSMIChannelHandlerCall () -->
pfnHandler() ==
[Host side] vbvaChannelHandler
It looks like a guest-side function (in VGA driver) VBoxMPHGSMIDpc
calls the host-side function (in VGA emulator) vbvaChannelHandler.
How is this possible since host and guest are in different address
space (if we are using nested paging table)? Or, I'm feeling like I
miss something.

I'm not too familiar with that code, but what makes you think that
pfnHandler points to vbvaChannelHandler() inside the guest driver? See
line 626 of
src/VBox/Additions/WINNT/Graphics/Video/mp/xpdm/VBoxMPInternal.cpp:

rc = HGSMIChannelRegister(&pCommon->hostCtx.channels, u8Channel,
"VGA Miniport HGSMI channel",
VBoxVbvaChannelGenericHandlerCB,
pContexts, &s_OldHandler);

So I would guess that VBoxVbvaChannelGenericHandlerCB() is what is being
called. In any case there should be no direct calls from the guest to
the host address space.

Regards,

Michael
--
ORACLE Deutschland B.V. & Co. KG Michael Thayer
Werkstrasse 24 VirtualBox engineering
71384 Weinstadt, Germany mailto:michael.thayer at oracle.com

Hauptverwaltung: Riesstr. 25, D-80992 M?nchen
Registergericht: Amtsgericht M?nchen, HRA 95603
Gesch?ftsf?hrer: J?rgen Kunz

Komplement?rin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Gesch?ftsf?hrer: Alexander van der Ven, Astrid Kepper, Val Maher

[Jaeyong Yoo]

Sender : Michael Thayer<michael.thayer at oracle.com>
Date : 2014-04-24 23:00 (GMT+09:00)
Title : Re: [vbox-dev] Question regarding channel callbacks between guest and host
Hello Jaeyong,

Post by Jaeyong Yoo
While I'm reading the source, I found the following sequence of
[In xpdm directory] VBoxMPHGSMIDpc () -->
VBoxHGSMIProcessHostQueue () -->
hgsmiHostCommandQueryProcess() -->
hgsmiHostCmdProcess () -->
HGSMIBufferProcess () -->
hgsmiBufferProcess () -->
HGSMIChannelHandlerCall () -->
pfnHandler() ==
[Host side] vbvaChannelHandler
It looks like a guest-side function (in VGA driver) VBoxMPHGSMIDpc
calls the host-side function (in VGA emulator) vbvaChannelHandler.
How is this possible since host and guest are in different address
space (if we are using nested paging table)? Or, I'm feeling like I
miss something.

I'm not too familiar with that code, but what makes you think that
pfnHandler points to vbvaChannelHandler() inside the guest driver? See
line 626 of
rc = HGSMIChannelRegister(&pCommon->hostCtx.channels, u8Channel,
"VGA Miniport HGSMI channel",
VBoxVbvaChannelGenericHandlerCB,
pContexts, &s_OldHandler);
So I would guess that VBoxVbvaChannelGenericHandlerCB() is what is being
called. In any case there should be no direct calls from the guest to
the host address space.

Oh, I must got confused with the line 2411
src/VBox/Devices/Graphics/DevVGA_VBVA.cpp:

rc = HGSMIHostChannelRegister (pVGAState->pHGSMI,
HGSMI_CH_VBVA,
vbvaChannelHandler,
pVGAState,
&sOldChannelHandler);

where HGSMIHostChannelRegister is finally calling
rc = HGSMIChannelRegister (&pIns->channelInfo, u8Channel, NULL, pfnChannelHandler, pvChannelHandler, pOldHandler);

They must have the same function name but reside in different address space (guest and host).

Currently, I'm trying to figure out how guest and host interact with events (such as guest notifying host or host injecting interrupt to guest).
I'm suspecting that VBox is also using VT-x techniques (such as IO port/MMIO emulation or virtual APIC interrupt injection).
But, it is quite hard to point the source that actually do that. Could you give me a little heads up?

Thanks,
Jaeyong

[Michael Thayer]

Hello Jaeyong,

On 25/04/14 02:30, Jaeyong Yoo wrote:
[...]

Post by Jaeyong Yoo
Currently, I'm trying to figure out how guest and host interact with
events (such as guest notifying host or host injecting interrupt to
guest). I'm suspecting that VBox is also using VT-x techniques (such
as IO port/MMIO emulation or virtual APIC interrupt injection). But,
it is quite hard to point the source that actually do that. Could you
give me a little heads up?

[...]
I'm not too familiar with that code either, though I have a basic
understanding of what it is doing of course. I wonder though whether
you could give us a bit of a clue as to where you are trying to go.
From the messages you have send I presume you are most interested in
the Windows guest code, and you don't really need a detailed
understanding of the core virtualisation code for that - understanding
the concepts around it would be more important in that case - but it
would still take you a lot of time to work through it which you might
prefer to spend on the code which really interests you. I can still ask
one of our core virtualisation team members to give you a pointer of
course in case I am wrong there.

Regards,

Michael
--
ORACLE Deutschland B.V. & Co. KG Michael Thayer
Werkstrasse 24 VirtualBox engineering
71384 Weinstadt, Germany mailto:michael.thayer at oracle.com

Hauptverwaltung: Riesstr. 25, D-80992 M?nchen
Registergericht: Amtsgericht M?nchen, HRA 95603
Gesch?ftsf?hrer: J?rgen Kunz

Komplement?rin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Gesch?ftsf?hrer: Alexander van der Ven, Astrid Kepper, Val Maher

[Jaeyong Yoo]

Hello Michael,

[...]

From the messages you have send I presume you are most interested in the
Windows guest code, and you don't really need a detailed understanding of
the core virtualisation code for that - understanding the concepts around
it would be more important in that case - but it would still take you a
lot of time to work through it which you might prefer to spend on the code
which really interests you. I can still ask one of our core
virtualisation team members to give you a pointer of course in case I am
wrong there.

I'm trying to port the VirtualBox graphics stack to QEMU environment.
I guess there would be lot of difficulties, but I'm expecting that the only
parts need to be modified are hypervisor-facing parts. The parts that are facing
Windows would not require any changes (I hope) such as WineD3D, Display drivers
Direct3D facing part, etc

I will appreciate if one of your team members could verify my understanding :-)

Thanks,
Jaeyong

[Jaeyong Yoo]

Post by Jaeyong Yoo
Hello Michael,
[...]

Post by Michael Thayer
From the messages you have send I presume you are most interested in
the Windows guest code, and you don't really need a detailed
understanding of the core virtualisation code for that - understanding
the concepts around it would be more important in that case - but it
would still take you a lot of time to work through it which you might
prefer to spend on the code which really interests you. I can still
ask one of our core virtualisation team members to give you a pointer
of course in case I am wrong there.

I'm trying to port the VirtualBox graphics stack to QEMU environment.
I guess there would be lot of difficulties, but I'm expecting that the
only parts need to be modified are hypervisor-facing parts. The parts that
are facing Windows would not require any changes (I hope) such as WineD3D,
Display drivers Direct3D facing part, etc
I will appreciate if one of your team members could verify my
understanding :-)

One more possibility is to port VirtualBox to android. But, since android depends
on bionic c, it requires porting of many libraries (including glib). Have you
ever considered similar things in VirtualBox team?

Thanks,
Jaeyong

[Ramshankar]

Post by Jaeyong Yoo
They must have the same function name but reside in different address
space (guest and host). Currently, I'm trying to figure out how guest
and host interact with events (such as guest notifying host or host
injecting interrupt to guest). I'm suspecting that VBox is also using
VT-x techniques (such as IO port/MMIO emulation or virtual APIC
interrupt injection). But, it is quite hard to point the source that
actually do that. Could you give me a little heads up? Thanks, Jaeyong

The way we do IO and MMIO access is a bit complicated to trace because
of a lot of indirection but the structure of the overall design is quite
logical.

When executing guest-code under hardware virtualization, we'll get
VM-Exits/#VMEXIT (for VT-x, see VMMR0/HMVMXR0.cpp hmR0VmxExitIoInstr(),
hmR0VmxExitXcptPF() etc., for AMD-V see VMMR0/HMSVMR0.cpp:
hmR0SvmExitIOInstr(), hmR0SvmExitXcptPF() etc.) that end up calling into
the PDM/PGM/IOM subsystem. That then based on the port and access will
choose the access handler which was registered for the port/MMIO region.

Different devices register access handlers for IO ports using
PDMDevHlpIOPortRegister() and MMIO using PDMDevHlpMMIORegister().
Finding which access handler gets called can be a bit tricky. One of the
easier ways is, to see what has been registered using the VM debugger
("info phys" and "info handlers").

Depending on whether the device can handle the access request in ring-0
or not, we may reschedule the work to be done in ring-3.

Hope that helps.

Regards,
Ram.

[Jaeyong Yoo]

Hello Ramshankar,

The way we do IO and MMIO access is a bit complicated to trace because of
a lot of indirection but the structure of the overall design is quite
logical.
When executing guest-code under hardware virtualization, we'll get VM-
Exits/#VMEXIT (for VT-x, see VMMR0/HMVMXR0.cpp hmR0VmxExitIoInstr(),
hmR0SvmExitIOInstr(), hmR0SvmExitXcptPF() etc.) that end up calling into
the PDM/PGM/IOM subsystem. That then based on the port and access will
choose the access handler which was registered for the port/MMIO region.
Different devices register access handlers for IO ports using
PDMDevHlpIOPortRegister() and MMIO using PDMDevHlpMMIORegister().
Finding which access handler gets called can be a bit tricky. One of the
easier ways is, to see what has been registered using the VM debugger
("info phys" and "info handlers").
Depending on whether the device can handle the access request in ring-0 or
not, we may reschedule the work to be done in ring-3.

After reading your explanation and I read the code, it quite understandable.

I've got one more question: I expect that IO/MMIO is the only way of
interaction
from the guest to the host. In case of the host to the guest interaction,
does
VirtualBox rely on interrupt injection (supported by VT-x) or does it use
some
kind of software-emulated method (like Xen's event channel)?

Hope that helps.

Helped a lot. Thanks!

Regards,
Jaeyong

[Ramshankar]

Post by Jaeyong Yoo
Hello Ramshankar,

The way we do IO and MMIO access is a bit complicated to trace because of
a lot of indirection but the structure of the overall design is quite
logical.
When executing guest-code under hardware virtualization, we'll get VM-
Exits/#VMEXIT (for VT-x, see VMMR0/HMVMXR0.cpp hmR0VmxExitIoInstr(),
hmR0SvmExitIOInstr(), hmR0SvmExitXcptPF() etc.) that end up calling into
the PDM/PGM/IOM subsystem. That then based on the port and access will
choose the access handler which was registered for the port/MMIO region.
Different devices register access handlers for IO ports using
PDMDevHlpIOPortRegister() and MMIO using PDMDevHlpMMIORegister().
Finding which access handler gets called can be a bit tricky. One of the
easier ways is, to see what has been registered using the VM debugger
("info phys" and "info handlers").
Depending on whether the device can handle the access request in ring-0 or
not, we may reschedule the work to be done in ring-3.

After reading your explanation and I read the code, it quite understandable.
I've got one more question: I expect that IO/MMIO is the only way of
interaction
from the guest to the host. In case of the host to the guest interaction,
does
VirtualBox rely on interrupt injection (supported by VT-x) or does it use
some
kind of software-emulated method (like Xen's event channel)?

Hope that helps.

Helped a lot. Thanks!
Regards,
Jaeyong

I'm not sure what kind of interaction you're referring to, but I'm sure,
if one needed, it would be possible to create a some kind of shared
memory interface and use make use of special interrupts to poke things.
The guest may provide a page in its address space to the hypervisor and
this could be used to communicate using hypercalls. This obviously
requires co-operation from the guest.

Both Intel and AMD-V have instructions that allow unconditional traps
into the hypervisor (see, VMCALL for Intel and VMMCALL for AMD-V). There
are also other methods that can be used to communicate between guest and
host without using architecture-specific instructions, but require the
guest to co-operate in a defined manner.

I don't know how interrupt delivery is all that relevant to the topic at
hand, but yes, VirtualBox uses VT-x and AMD-V's interrupt injection
mechanics (except for one specific case where we inject interrupts
manually when the CPU isn't capable of real-mode execution with VT-x,
see hmR0VmxInjectEventVmcs()).

Regards,
Ram.