[vbox-dev] Crash on Win 8 guest when kernel debugging is enabled via network

Discussion:

Ray Yang

2014-08-01 05:05:58 UTC

Hi guys,

I have been fighting with this issue for quite a while but still do not
have any clue about how to solve it.

Basically, I have a Win 8 64bit guest OS running on VBox 4.3.12 (both
official release and my private build versions) on Win 7 64bit host. If I
do not enable kernel debugging via network on the guest Win 8 with
multi-core, it boots up without problem. But if the kernel debugging and
multi-core are enabled, guest OS just bug-checks at very early stage
during the boot, with the following bug-check codes:

BugCheck 124, {3, 0, 0, 0}
BugCheck 80, {4f4454, 0, 0, 0}

Every time when it bug-checks, it always does not stop on core 0. Since
they are NMI related, I dump the apic info in the guest with WinDbg, as
the following:

1: kd> !apic
Apic @ ffdab000 ID:1 (50014) LogDesc:02000000 DestFmt:ffffffff TPR F0
TimeCnt: 00000000clk SpurVec:3f FaultVec:e2 error:0
Ipi Cmd: 01000000`00000c00 Vec:00 NMI Lg:01000000 edg high
Timer..: 00000000`000300fd Vec:FD FixedDel Dest=Self edg high
m
Linti0.: 00000000`000100c0 Vec:C0 FixedDel Dest=Self edg high
m
Linti1.: 00000000`000184c0 Vec:C0 NMI Dest=Self lvl high
m
TMR:
IRR:
ISR: D1
1: kd> ~0s
0: kd> !apic
Apic @ ffdab000 ID:0 (50014) LogDesc:01000000 DestFmt:ffffffff TPR F0
TimeCnt: 00000000clk SpurVec:3f FaultVec:e2 error:0
Ipi Cmd: 02000000`000c00d1 Vec:D1 FixedDel Dest=Othrs edg high
Timer..: 00000000`000300ff Vec:FF FixedDel Dest=Self edg high
m
Linti0.: 00000000`000100c0 Vec:C0 FixedDel Dest=Self edg high
m
Linti1.: 00000000`000184c0 Vec:C0 NMI Dest=Self lvl high
m
TMR: B1
IRR: D1
ISR:

Then notice IPI has an interrupt to vector zero. This is confirmed after
apic log is enabled in VBox as well:

04:27:44.429801 0000000000001e80 EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=1 trigger_mode=1
04:27:44.429801 0000000000001e80 EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=0 trigger_mode=1
04:33:41.362210 00000000000018ec EMT-0 apic_deliver dest=2 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:49.438736 00000000000018ec EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=1 trigger_mode=1
04:33:49.439236 00000000000018ec EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=0 trigger_mode=1
04:33:49.743774 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:50.489369 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:50.856916 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.685521 00000000000018ec EMT-0 apic_deliver dest=2 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.878045 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.894047 00000000000018ec EMT-0 apic_deliver dest=2 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0

Note that the initial two interrupts to vector 0 are also seen with
successful guest boot. But the problematic one has more interrupts to
vector 0. After check more logs, I found the following two:

04:33:51.878045 0000000000001114 EMT-1 CPU1: apicMMIOWrite at
00000000fee00300
04:33:51.894047 00000000000018ec EMT-0 CPU0: apicMMIOWrite at
00000000fee00300

It seems that both core have the same local APIC address. Is this correct?

As I'm totally stuck at the moment, really appreciate if someone can give
me some direction on this issue.

Btw, VBox 4.3.14 does not work for me as its GUI refuses to start.

Thanks.

-Ray

Michal Necasek

2014-08-01 06:14:20 UTC

Permalink

See https://www.virtualbox.org/ticket/10097 . This problem was fixed in 4.3.14 for Windows 7 guests and I have currently no reason to think Windows 8 is different.

And yes, it's entirely normal that all local APICs in a system are mapped at the same address. And those IPIs with vector zero are in fact NMIs (the delivery mode says it's NMI, hence the vector is meaningless and it will be delivered through IDT vector 2 as an NMI, not a maskable interrupt).

Regards,

Michal

----- Original Message -----
From: rayyang at ybwork.com
To: vbox-dev at virtualbox.org
Sent: Friday, August 1, 2014 7:06:38 AM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna
Subject: [vbox-dev] Crash on Win 8 guest when kernel debugging is enabled via network

Hi guys,

I have been fighting with this issue for quite a while but still do not
have any clue about how to solve it.

Basically, I have a Win 8 64bit guest OS running on VBox 4.3.12 (both
official release and my private build versions) on Win 7 64bit host. If I
do not enable kernel debugging via network on the guest Win 8 with
multi-core, it boots up without problem. But if the kernel debugging and
multi-core are enabled, guest OS just bug-checks at very early stage
during the boot, with the following bug-check codes:

BugCheck 124, {3, 0, 0, 0}
BugCheck 80, {4f4454, 0, 0, 0}

Every time when it bug-checks, it always does not stop on core 0. Since
they are NMI related, I dump the apic info in the guest with WinDbg, as
the following:

1: kd> !apic
Apic @ ffdab000 ID:1 (50014) LogDesc:02000000 DestFmt:ffffffff TPR F0
TimeCnt: 00000000clk SpurVec:3f FaultVec:e2 error:0
Ipi Cmd: 01000000`00000c00 Vec:00 NMI Lg:01000000 edg high
Timer..: 00000000`000300fd Vec:FD FixedDel Dest=Self edg high
m
Linti0.: 00000000`000100c0 Vec:C0 FixedDel Dest=Self edg high
m
Linti1.: 00000000`000184c0 Vec:C0 NMI Dest=Self lvl high
m
TMR:
IRR:
ISR: D1
1: kd> ~0s
0: kd> !apic
Apic @ ffdab000 ID:0 (50014) LogDesc:01000000 DestFmt:ffffffff TPR F0
TimeCnt: 00000000clk SpurVec:3f FaultVec:e2 error:0
Ipi Cmd: 02000000`000c00d1 Vec:D1 FixedDel Dest=Othrs edg high
Timer..: 00000000`000300ff Vec:FF FixedDel Dest=Self edg high
m
Linti0.: 00000000`000100c0 Vec:C0 FixedDel Dest=Self edg high
m
Linti1.: 00000000`000184c0 Vec:C0 NMI Dest=Self lvl high
m
TMR: B1
IRR: D1
ISR:

Then notice IPI has an interrupt to vector zero. This is confirmed after
apic log is enabled in VBox as well:

04:27:44.429801 0000000000001e80 EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=1 trigger_mode=1
04:27:44.429801 0000000000001e80 EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=0 trigger_mode=1
04:33:41.362210 00000000000018ec EMT-0 apic_deliver dest=2 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:49.438736 00000000000018ec EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=1 trigger_mode=1
04:33:49.439236 00000000000018ec EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=0 trigger_mode=1
04:33:49.743774 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:50.489369 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:50.856916 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.685521 00000000000018ec EMT-0 apic_deliver dest=2 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.878045 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.894047 00000000000018ec EMT-0 apic_deliver dest=2 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0

Note that the initial two interrupts to vector 0 are also seen with
successful guest boot. But the problematic one has more interrupts to
vector 0. After check more logs, I found the following two:

04:33:51.878045 0000000000001114 EMT-1 CPU1: apicMMIOWrite at
00000000fee00300
04:33:51.894047 00000000000018ec EMT-0 CPU0: apicMMIOWrite at
00000000fee00300

It seems that both core have the same local APIC address. Is this correct?

As I'm totally stuck at the moment, really appreciate if someone can give
me some direction on this issue.

Btw, VBox 4.3.14 does not work for me as its GUI refuses to start.

Thanks.

-Ray

Ray Yang

2014-08-02 07:25:17 UTC

Permalink

Thanks Michal.

I tried the test build VirtualBox-4.3.15-95286-Win.exe and it did not
produce the issue any more.

However, my custom virtual PCI device dll cannot be loaded by 4.3.15 with
error: Unable to load R3 module (VERR_LDRVI_NOT_SIGNED).

Does this mean from 4.3.15, VBox needs the device module to be signed? Is
there any way to disable it?

Thanks.

-Ray

Post by Michal Necasek
See https://www.virtualbox.org/ticket/10097 . This problem was fixed in
4.3.14 for Windows 7 guests and I have currently no reason to think
Windows 8 is different.
And yes, it's entirely normal that all local APICs in a system are mapped
at the same address. And those IPIs with vector zero are in fact NMIs
(the delivery mode says it's NMI, hence the vector is meaningless and it
will be delivered through IDT vector 2 as an NMI, not a maskable
interrupt).
Regards,
Michal
----- Original Message -----
From: rayyang at ybwork.com
To: vbox-dev at virtualbox.org
Sent: Friday, August 1, 2014 7:06:38 AM GMT +01:00 Amsterdam / Berlin /
Bern / Rome / Stockholm / Vienna
Subject: [vbox-dev] Crash on Win 8 guest when kernel debugging is enabled via network
Hi guys,
I have been fighting with this issue for quite a while but still do not
have any clue about how to solve it.
Basically, I have a Win 8 64bit guest OS running on VBox 4.3.12 (both
official release and my private build versions) on Win 7 64bit host. If I
do not enable kernel debugging via network on the guest Win 8 with
multi-core, it boots up without problem. But if the kernel debugging and
multi-core are enabled, guest OS just bug-checks at very early stage
BugCheck 124, {3, 0, 0, 0}
BugCheck 80, {4f4454, 0, 0, 0}
Every time when it bug-checks, it always does not stop on core 0. Since
they are NMI related, I dump the apic info in the guest with WinDbg, as
1: kd> !apic
TimeCnt: 00000000clk SpurVec:3f FaultVec:e2 error:0
Ipi Cmd: 01000000`00000c00 Vec:00 NMI Lg:01000000 edg high
Timer..: 00000000`000300fd Vec:FD FixedDel Dest=Self edg high
m
Linti0.: 00000000`000100c0 Vec:C0 FixedDel Dest=Self edg high
m
Linti1.: 00000000`000184c0 Vec:C0 NMI Dest=Self lvl high
m
ISR: D1
1: kd> ~0s
0: kd> !apic
TimeCnt: 00000000clk SpurVec:3f FaultVec:e2 error:0
Ipi Cmd: 02000000`000c00d1 Vec:D1 FixedDel Dest=Othrs edg high
Timer..: 00000000`000300ff Vec:FF FixedDel Dest=Self edg high
m
Linti0.: 00000000`000100c0 Vec:C0 FixedDel Dest=Self edg high
m
Linti1.: 00000000`000184c0 Vec:C0 NMI Dest=Self lvl high
m
TMR: B1
IRR: D1
Then notice IPI has an interrupt to vector zero. This is confirmed after
04:27:44.429801 0000000000001e80 EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=1 trigger_mode=1
04:27:44.429801 0000000000001e80 EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=0 trigger_mode=1
04:33:41.362210 00000000000018ec EMT-0 apic_deliver dest=2 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:49.438736 00000000000018ec EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=1 trigger_mode=1
04:33:49.439236 00000000000018ec EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=0 trigger_mode=1
04:33:49.743774 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:50.489369 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:50.856916 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.685521 00000000000018ec EMT-0 apic_deliver dest=2 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.878045 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.894047 00000000000018ec EMT-0 apic_deliver dest=2 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
Note that the initial two interrupts to vector 0 are also seen with
successful guest boot. But the problematic one has more interrupts to
04:33:51.878045 0000000000001114 EMT-1 CPU1: apicMMIOWrite at
00000000fee00300
04:33:51.894047 00000000000018ec EMT-0 CPU0: apicMMIOWrite at
00000000fee00300
It seems that both core have the same local APIC address. Is this correct?
As I'm totally stuck at the moment, really appreciate if someone can give
me some direction on this issue.
Btw, VBox 4.3.14 does not work for me as its GUI refuses to start.
Thanks.
-Ray
_______________________________________________
vbox-dev mailing list
vbox-dev at virtualbox.org
https://www.virtualbox.org/mailman/listinfo/vbox-dev

Klaus Espenlaub

2014-08-04 08:06:17 UTC

Permalink

Ray,

Post by Ray Yang
Thanks Michal.
I tried the test build VirtualBox-4.3.15-95286-Win.exe and it did not
produce the issue any more.
However, my custom virtual PCI device dll cannot be loaded by 4.3.15 with
error: Unable to load R3 module (VERR_LDRVI_NOT_SIGNED).

That's expected. All such DLLs must be signed now, using a valid kernel
driver signature.

Post by Ray Yang
Does this mean from 4.3.15, VBox needs the device module to be signed? Is
there any way to disable it?

From 4.3.14, yes, they all need to be signed (not by us, but by anyone
having the private key with a Windows kernel driver cert - the usual
Windows kernel signing stuff). There is no way to disable this, unless
you build your own package from sources with hardening disabled (not
recommended for anything but development use).

In principle test signing should work, but there's already a report that
it's at least not obvious how to get this working:
http://thread.gmane.org/gmane.comp.emulators.virtualbox.devel/6955

Klaus

Post by Ray Yang
Thanks.
-Ray

Ray Yang

2014-08-04 17:06:19 UTC

Permalink

From the security point of view, this signature checking is reasonable.

However, there are still some people who do not have certificates for
various reasons. So if VBox can have an option to disable this by showing
some GUI warnings, it would be much more convenient.

And yes, the test sign does not work - VBox complains that there is no
trusted root in the path, which is true since the test certificate is not
issued by any trusted CA.

Ray,

That's expected. All such DLLs must be signed now, using a valid kernel
driver signature.

Post by Ray Yang
Does this mean from 4.3.15, VBox needs the device module to be signed? Is
there any way to disable it?

From 4.3.14, yes, they all need to be signed (not by us, but by anyone
having the private key with a Windows kernel driver cert - the usual
Windows kernel signing stuff). There is no way to disable this, unless
you build your own package from sources with hardening disabled (not
recommended for anything but development use).
In principle test signing should work, but there's already a report that
http://thread.gmane.org/gmane.comp.emulators.virtualbox.devel/6955
Klaus

Post by Ray Yang
Thanks.
-Ray

Post by Michal Necasek
See https://www.virtualbox.org/ticket/10097 . This problem was fixed in
4.3.14 for Windows 7 guests and I have currently no reason to think
Windows 8 is different.
And yes, it's entirely normal that all local APICs in a system are mapped
at the same address. And those IPIs with vector zero are in fact NMIs
(the delivery mode says it's NMI, hence the vector is meaningless and it
will be delivered through IDT vector 2 as an NMI, not a maskable
interrupt).
Regards,
Michal
----- Original Message -----
From: rayyang at ybwork.com
To: vbox-dev at virtualbox.org
Sent: Friday, August 1, 2014 7:06:38 AM GMT +01:00 Amsterdam / Berlin /
Bern / Rome / Stockholm / Vienna
Subject: [vbox-dev] Crash on Win 8 guest when kernel debugging is
enabled
via network
Hi guys,
I have been fighting with this issue for quite a while but still do not
have any clue about how to solve it.
Basically, I have a Win 8 64bit guest OS running on VBox 4.3.12 (both
official release and my private build versions) on Win 7 64bit host. If I
do not enable kernel debugging via network on the guest Win 8 with
multi-core, it boots up without problem. But if the kernel debugging and
multi-core are enabled, guest OS just bug-checks at very early stage
BugCheck 124, {3, 0, 0, 0}
BugCheck 80, {4f4454, 0, 0, 0}
Every time when it bug-checks, it always does not stop on core 0. Since
they are NMI related, I dump the apic info in the guest with WinDbg, as
1: kd> !apic
TimeCnt: 00000000clk SpurVec:3f FaultVec:e2 error:0
Ipi Cmd: 01000000`00000c00 Vec:00 NMI Lg:01000000 edg high
Timer..: 00000000`000300fd Vec:FD FixedDel Dest=Self edg high
m
Linti0.: 00000000`000100c0 Vec:C0 FixedDel Dest=Self edg high
m
Linti1.: 00000000`000184c0 Vec:C0 NMI Dest=Self lvl high
m
ISR: D1
1: kd> ~0s
0: kd> !apic
TimeCnt: 00000000clk SpurVec:3f FaultVec:e2 error:0
Ipi Cmd: 02000000`000c00d1 Vec:D1 FixedDel Dest=Othrs edg high
Timer..: 00000000`000300ff Vec:FF FixedDel Dest=Self edg high
m
Linti0.: 00000000`000100c0 Vec:C0 FixedDel Dest=Self edg high
m
Linti1.: 00000000`000184c0 Vec:C0 NMI Dest=Self lvl high
m
TMR: B1
IRR: D1
Then notice IPI has an interrupt to vector zero. This is confirmed after
04:27:44.429801 0000000000001e80 EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=1 trigger_mode=1
04:27:44.429801 0000000000001e80 EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=0 trigger_mode=1
04:33:41.362210 00000000000018ec EMT-0 apic_deliver dest=2 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:49.438736 00000000000018ec EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=1 trigger_mode=1
04:33:49.439236 00000000000018ec EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=0 trigger_mode=1
04:33:49.743774 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:50.489369 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:50.856916 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.685521 00000000000018ec EMT-0 apic_deliver dest=2 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.878045 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.894047 00000000000018ec EMT-0 apic_deliver dest=2 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
Note that the initial two interrupts to vector 0 are also seen with
successful guest boot. But the problematic one has more interrupts to
04:33:51.878045 0000000000001114 EMT-1 CPU1: apicMMIOWrite at
00000000fee00300
04:33:51.894047 00000000000018ec EMT-0 CPU0: apicMMIOWrite at
00000000fee00300
It seems that both core have the same local APIC address. Is this correct?
As I'm totally stuck at the moment, really appreciate if someone can give
me some direction on this issue.
Btw, VBox 4.3.14 does not work for me as its GUI refuses to start.
Thanks.
-Ray

_______________________________________________
vbox-dev mailing list
vbox-dev at virtualbox.org
https://www.virtualbox.org/mailman/listinfo/vbox-dev

Klaus Espenlaub

2014-08-04 18:08:21 UTC

Permalink

Ray,

From the security point of view, this signature checking is reasonable.
However, there are still some people who do not have certificates for
various reasons. So if VBox can have an option to disable this by showing
some GUI warnings, it would be much more convenient.

This won't happen, it'd open security holes. We didn't do this exercise
in 4.3.14 just for fun.

And yes, the test sign does not work - VBox complains that there is no
trusted root in the path, which is true since the test certificate is not
issued by any trusted CA.

Once the far more urgent issues with 4.3.14 are sorted (many users on
Windows can't start VirtualBox at all any more) we'll get to this topic.

Klaus

Post by Klaus Espenlaub
Ray,

That's expected. All such DLLs must be signed now, using a valid kernel
driver signature.

Post by Ray Yang
Does this mean from 4.3.15, VBox needs the device module to be signed? Is
there any way to disable it?

From 4.3.14, yes, they all need to be signed (not by us, but by anyone
having the private key with a Windows kernel driver cert - the usual
Windows kernel signing stuff). There is no way to disable this, unless
you build your own package from sources with hardening disabled (not
recommended for anything but development use).
In principle test signing should work, but there's already a report that
http://thread.gmane.org/gmane.comp.emulators.virtualbox.devel/6955
Klaus

Post by Ray Yang
Thanks.
-Ray

Post by Michal Necasek
See https://www.virtualbox.org/ticket/10097 . This problem was fixed in
4.3.14 for Windows 7 guests and I have currently no reason to think
Windows 8 is different.
And yes, it's entirely normal that all local APICs in a system are mapped
at the same address. And those IPIs with vector zero are in fact NMIs
(the delivery mode says it's NMI, hence the vector is meaningless and it
will be delivered through IDT vector 2 as an NMI, not a maskable
interrupt).
Regards,
Michal
----- Original Message -----
From: rayyang at ybwork.com
To: vbox-dev at virtualbox.org
Sent: Friday, August 1, 2014 7:06:38 AM GMT +01:00 Amsterdam / Berlin /
Bern / Rome / Stockholm / Vienna
Subject: [vbox-dev] Crash on Win 8 guest when kernel debugging is
enabled
via network
Hi guys,
I have been fighting with this issue for quite a while but still do not
have any clue about how to solve it.
Basically, I have a Win 8 64bit guest OS running on VBox 4.3.12 (both
official release and my private build versions) on Win 7 64bit host. If I
do not enable kernel debugging via network on the guest Win 8 with
multi-core, it boots up without problem. But if the kernel debugging and
multi-core are enabled, guest OS just bug-checks at very early stage
BugCheck 124, {3, 0, 0, 0}
BugCheck 80, {4f4454, 0, 0, 0}
Every time when it bug-checks, it always does not stop on core 0. Since
they are NMI related, I dump the apic info in the guest with WinDbg, as
1: kd> !apic
TimeCnt: 00000000clk SpurVec:3f FaultVec:e2 error:0
Ipi Cmd: 01000000`00000c00 Vec:00 NMI Lg:01000000 edg high
Timer..: 00000000`000300fd Vec:FD FixedDel Dest=Self edg high
m
Linti0.: 00000000`000100c0 Vec:C0 FixedDel Dest=Self edg high
m
Linti1.: 00000000`000184c0 Vec:C0 NMI Dest=Self lvl high
m
ISR: D1
1: kd> ~0s
0: kd> !apic
TimeCnt: 00000000clk SpurVec:3f FaultVec:e2 error:0
Ipi Cmd: 02000000`000c00d1 Vec:D1 FixedDel Dest=Othrs edg high
Timer..: 00000000`000300ff Vec:FF FixedDel Dest=Self edg high
m
Linti0.: 00000000`000100c0 Vec:C0 FixedDel Dest=Self edg high
m
Linti1.: 00000000`000184c0 Vec:C0 NMI Dest=Self lvl high
m
TMR: B1
IRR: D1
Then notice IPI has an interrupt to vector zero. This is confirmed after
04:27:44.429801 0000000000001e80 EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=1 trigger_mode=1
04:27:44.429801 0000000000001e80 EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=0 trigger_mode=1
04:33:41.362210 00000000000018ec EMT-0 apic_deliver dest=2 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:49.438736 00000000000018ec EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=1 trigger_mode=1
04:33:49.439236 00000000000018ec EMT-0 apic_deliver dest=1 dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=0 trigger_mode=1
04:33:49.743774 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:50.489369 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:50.856916 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.685521 00000000000018ec EMT-0 apic_deliver dest=2 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.878045 0000000000001114 EMT-1 apic_deliver dest=1 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.894047 00000000000018ec EMT-0 apic_deliver dest=2 dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
Note that the initial two interrupts to vector 0 are also seen with
successful guest boot. But the problematic one has more interrupts to
04:33:51.878045 0000000000001114 EMT-1 CPU1: apicMMIOWrite at
00000000fee00300
04:33:51.894047 00000000000018ec EMT-0 CPU0: apicMMIOWrite at
00000000fee00300
It seems that both core have the same local APIC address. Is this correct?
As I'm totally stuck at the moment, really appreciate if someone can give
me some direction on this issue.
Btw, VBox 4.3.14 does not work for me as its GUI refuses to start.
Thanks.
-Ray

Mihai Hanor

2014-08-04 19:50:23 UTC

Permalink

Add the test certificate to the Trusted Root Certification Authorities and to the Trusted Publishers (certmgr.msc). You may also need to boot your OS with testsigning on (bcdedit), but I'm not sure. If VirtualBox successfully loaded unsigned PDM modules (in the past), you may find that the testsigning boot option is not required, since this new signing requirements are not enforced by the OS. For testing kernel drivers, you do need testsigning to be on. Using these methods, I play with custom builds of VirtualBox, on Windows 7 and 8.1 x64 hosts.

More on test-signing drivers:

http://msdn.microsoft.com/en-us/library/windows/hardware/ff546236%28v=vs.85%29.aspx

Post by Michal Necasek
________________________________
From: Ray Yang <rayyang at ybwork.com>
To: Klaus Espenlaub <klaus.espenlaub at oracle.com>
Cc: vbox-dev at virtualbox.org
Sent: Monday, August 4, 2014 8:06 PM
Subject: Re: [vbox-dev] Crash on Win 8 guest when kernel debugging is enabled via network

From the security point of view, this signature checking is reasonable.

However, there are still some people who do not have certificates for
various reasons. So if VBox can have an option to disable this by showing
some GUI warnings, it would be much more convenient.
And yes, the test sign does not work - VBox complains that there is no
trusted root in the path, which is true since the test certificate is not
issued by any trusted CA.

Ray,

That's expected. All such DLLs must be signed now, using a valid kernel
driver signature.

Post by Ray Yang
Does this mean from 4.3.15, VBox needs the device module to be signed? Is
there any way to disable it?

? From 4.3.14, yes, they all need to be signed (not by us, but by anyone
having the private key with a Windows kernel driver cert - the usual
Windows kernel signing stuff). There is no way to disable this, unless
you build your own package from sources with hardening disabled (not
recommended for anything but development use).
In principle test signing should work, but there's already a report that
http://thread.gmane.org/gmane.comp.emulators.virtualbox.devel/6955
Klaus

Post by Ray Yang
Thanks.
-Ray

? See https://www.virtualbox.org/ticket/10097 . This problem was fixed
in
4.3.14 for Windows 7 guests and I have currently no reason to think
Windows 8 is different.
? And yes, it's entirely normal that all local APICs in a system are
mapped
at the same address. And those IPIs with vector zero are in fact NMIs
(the delivery mode says it's NMI, hence the vector is meaningless and it
will be delivered through IDT vector 2 as an NMI, not a maskable
interrupt).
? ? ? ? Regards,
? ? ? ? ? Michal
----- Original Message -----
From: rayyang at ybwork.com
To: vbox-dev at virtualbox.org
Sent: Friday, August 1, 2014 7:06:38 AM GMT +01:00 Amsterdam / Berlin /
Bern / Rome / Stockholm / Vienna
Subject: [vbox-dev] Crash on Win 8 guest when kernel debugging is
enabled
via network
Hi guys,
I have been fighting with this issue for quite a while but still do not
have any clue about how to solve it.
Basically, I have a Win 8 64bit guest OS running on VBox 4.3.12 (both
official release and my private build versions) on Win 7 64bit host. If I
do not enable kernel debugging via network on the guest Win 8 with
multi-core, it boots up without problem. But if the kernel debugging and
multi-core are enabled, guest OS just bug-checks at very early stage
BugCheck 124, {3, 0, 0, 0}
BugCheck 80, {4f4454, 0, 0, 0}
Every time when it bug-checks, it always does not stop on core 0. Since
they are NMI related, I dump the apic info in the guest with WinDbg, as
1: kd> !apic
F0
TimeCnt: 00000000clk? SpurVec:3f? FaultVec:e2? error:0
Ipi Cmd: 01000000`00000c00? Vec:00? NMI? ? ? Lg:01000000? ? ? edg high
Timer..: 00000000`000300fd? Vec:FD? FixedDel? ? Dest=Self? ? ? edg high
? ? m
Linti0.: 00000000`000100c0? Vec:C0? FixedDel? ? Dest=Self? ? ? edg high
? ? m
Linti1.: 00000000`000184c0? Vec:C0? NMI? ? ? ? Dest=Self? ? ? lvl high
? ? m
ISR: D1
1: kd> ~0s
0: kd> !apic
F0
TimeCnt: 00000000clk? SpurVec:3f? FaultVec:e2? error:0
Ipi Cmd: 02000000`000c00d1? Vec:D1? FixedDel? Dest=Othrs? ? ? edg high
Timer..: 00000000`000300ff? Vec:FF? FixedDel? ? Dest=Self? ? ? edg high
? ? m
Linti0.: 00000000`000100c0? Vec:C0? FixedDel? ? Dest=Self? ? ? edg high
? ? m
Linti1.: 00000000`000184c0? Vec:C0? NMI? ? ? ? Dest=Self? ? ? lvl high
? ? m
TMR: B1
IRR: D1
Then notice IPI has an interrupt to vector zero. This is confirmed after
04:27:44.429801 0000000000001e80 EMT-0? ? apic_deliver dest=1
dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=1 trigger_mode=1
04:27:44.429801 0000000000001e80 EMT-0? ? apic_deliver dest=1
dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=0 trigger_mode=1
04:33:41.362210 00000000000018ec EMT-0? ? apic_deliver dest=2
dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:49.438736 00000000000018ec EMT-0? ? apic_deliver dest=1
dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=1 trigger_mode=1
04:33:49.439236 00000000000018ec EMT-0? ? apic_deliver dest=1
dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=0 trigger_mode=1
04:33:49.743774 0000000000001114 EMT-1? ? apic_deliver dest=1
dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:50.489369 0000000000001114 EMT-1? ? apic_deliver dest=1
dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:50.856916 0000000000001114 EMT-1? ? apic_deliver dest=1
dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.685521 00000000000018ec EMT-0? ? apic_deliver dest=2
dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.878045 0000000000001114 EMT-1? ? apic_deliver dest=1
dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
04:33:51.894047 00000000000018ec EMT-0? ? apic_deliver dest=2
dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0 trigger_mode=0
Note that the initial two interrupts to vector 0 are also seen with
successful guest boot. But the problematic one has more interrupts to
04:33:51.878045 0000000000001114 EMT-1? ? CPU1: apicMMIOWrite at
00000000fee00300
04:33:51.894047 00000000000018ec EMT-0? ? CPU0: apicMMIOWrite at
00000000fee00300
It seems that both core have the same local APIC address. Is this correct?
As I'm totally stuck at the moment, really appreciate if someone can give
me some direction on this issue.
Btw, VBox 4.3.14 does not work for me as its GUI refuses to start.
Thanks.
-Ray

_______________________________________________
vbox-dev mailing list
vbox-dev at virtualbox.org
https://www.virtualbox.org/mailman/listinfo/vbox-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.virtualbox.org/pipermail/vbox-dev/attachments/20140804/8133df14/attachment-0001.html

Ray Yang

2014-08-04 20:59:09 UTC

Permalink

Thanks for your input but I think your reply is misleading here.

The signature checking in new VBox is only for PDM DLLs. AFAIK testsiging
set by bcdedit has nothing to do with DLLs. Although my unsigned DLLs are
loaded successfully by the old VBox, the root cause of this issue in the
new VBox is its new code to check the whole path of the certificate to
make sure everyone on this path is trusted. Agree that this is good for a
release/production environment. But not all developers can access an
official signing certificate at all times and this new feature actually
gives devs some noise here.

Post by Mihai Hanor
Add the test certificate to the Trusted Root Certification Authorities and
to the Trusted Publishers (certmgr.msc). You may also need to boot your OS
with testsigning on (bcdedit), but I'm not sure. If VirtualBox
successfully loaded unsigned PDM modules (in the past), you may find that
the testsigning boot option is not required, since this new signing
requirements are not enforced by the OS. For testing kernel drivers, you
do need testsigning to be on. Using these methods, I play with custom
builds of VirtualBox, on Windows 7 and 8.1 x64 hosts.
http://msdn.microsoft.com/en-us/library/windows/hardware/ff546236%28v=vs.85%29.aspx

From the security point of view, this signature checking is reasonable.

However, there are still some people who do not have certificates for
various reasons. So if VBox can have an option to disable this by showing
some GUI warnings, it would be much more convenient.
And yes, the test sign does not work - VBox complains that there is no
trusted root in the path, which is true since the test certificate is not
issued by any trusted CA.

Ray,

That's expected. All such DLLs must be signed now, using a valid kernel
driver signature.

Post by Ray Yang
Does this mean from 4.3.15, VBox needs the device module to be signed? Is
there any way to disable it?

Post by Ray Yang
Thanks.
-Ray

? See https://www.virtualbox.org/ticket/10097 . This problem was fixed
in
4.3.14 for Windows 7 guests and I have currently no reason to think
Windows 8 is different.
? And yes, it's entirely normal that all local APICs in a system are
mapped
at the same address. And those IPIs with vector zero are in fact NMIs
(the delivery mode says it's NMI, hence the vector is meaningless and it
will be delivered through IDT vector 2 as an NMI, not a maskable
interrupt).
? ? ? ? Regards,
? ? ? ? ? Michal
----- Original Message -----
From: rayyang at ybwork.com
To: vbox-dev at virtualbox.org
Sent: Friday, August 1, 2014 7:06:38 AM GMT +01:00 Amsterdam / Berlin /
Bern / Rome / Stockholm / Vienna
Subject: [vbox-dev] Crash on Win 8 guest when kernel debugging is
enabled
via network
Hi guys,
I have been fighting with this issue for quite a while but still do not
have any clue about how to solve it.
Basically, I have a Win 8 64bit guest OS running on VBox 4.3.12 (both
official release and my private build versions) on Win 7 64bit host.
If
I
do not enable kernel debugging via network on the guest Win 8 with
multi-core, it boots up without problem. But if the kernel debugging and
multi-core are enabled, guest OS just bug-checks at very early stage
BugCheck 124, {3, 0, 0, 0}
BugCheck 80, {4f4454, 0, 0, 0}
Every time when it bug-checks, it always does not stop on core 0. Since
they are NMI related, I dump the apic info in the guest with WinDbg, as
1: kd> !apic
F0
TimeCnt: 00000000clk? SpurVec:3f? FaultVec:e2? error:0
Ipi Cmd: 01000000`00000c00? Vec:00? NMI? ? ? Lg:01000000? ? ? edg high
Timer..: 00000000`000300fd? Vec:FD? FixedDel? ? Dest=Self? ? ? edg high
? ? m
Linti0.: 00000000`000100c0? Vec:C0? FixedDel? ? Dest=Self? ? ? edg high
? ? m
Linti1.: 00000000`000184c0? Vec:C0? NMI? ? ? ? Dest=Self? ? ? lvl high
? ? m
ISR: D1
1: kd> ~0s
0: kd> !apic
F0
TimeCnt: 00000000clk? SpurVec:3f? FaultVec:e2? error:0
Ipi Cmd: 02000000`000c00d1? Vec:D1? FixedDel? Dest=Othrs? ? ? edg high
Timer..: 00000000`000300ff? Vec:FF? FixedDel? ? Dest=Self? ? ? edg high
? ? m
Linti0.: 00000000`000100c0? Vec:C0? FixedDel? ? Dest=Self? ? ? edg high
? ? m
Linti1.: 00000000`000184c0? Vec:C0? NMI? ? ? ? Dest=Self? ? ? lvl high
? ? m
TMR: B1
IRR: D1
Then notice IPI has an interrupt to vector zero. This is confirmed after
04:27:44.429801 0000000000001e80 EMT-0? ? apic_deliver dest=1
dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=1
trigger_mode=1
04:27:44.429801 0000000000001e80 EMT-0? ? apic_deliver dest=1
dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=0
trigger_mode=1
04:33:41.362210 00000000000018ec EMT-0? ? apic_deliver dest=2
dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0
trigger_mode=0
04:33:49.438736 00000000000018ec EMT-0? ? apic_deliver dest=1
dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=1
trigger_mode=1
04:33:49.439236 00000000000018ec EMT-0? ? apic_deliver dest=1
dest_mode=0
dest_shorthand=0 delivery_mode=5 vector_num=0 polarity=0
trigger_mode=1
04:33:49.743774 0000000000001114 EMT-1? ? apic_deliver dest=1
dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0
trigger_mode=0
04:33:50.489369 0000000000001114 EMT-1? ? apic_deliver dest=1
dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0
trigger_mode=0
04:33:50.856916 0000000000001114 EMT-1? ? apic_deliver dest=1
dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0
trigger_mode=0
04:33:51.685521 00000000000018ec EMT-0? ? apic_deliver dest=2
dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0
trigger_mode=0
04:33:51.878045 0000000000001114 EMT-1? ? apic_deliver dest=1
dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0
trigger_mode=0
04:33:51.894047 00000000000018ec EMT-0? ? apic_deliver dest=2
dest_mode=1
dest_shorthand=0 delivery_mode=4 vector_num=0 polarity=0
trigger_mode=0
Note that the initial two interrupts to vector 0 are also seen with
successful guest boot. But the problematic one has more interrupts to
04:33:51.878045 0000000000001114 EMT-1? ? CPU1: apicMMIOWrite at
00000000fee00300
04:33:51.894047 00000000000018ec EMT-0? ? CPU0: apicMMIOWrite at
00000000fee00300
It seems that both core have the same local APIC address. Is this correct?
As I'm totally stuck at the moment, really appreciate if someone can give
me some direction on this issue.
Btw, VBox 4.3.14 does not work for me as its GUI refuses to start.
Thanks.
-Ray

_______________________________________________
vbox-dev mailing list
vbox-dev at virtualbox.org
https://www.virtualbox.org/mailman/listinfo/vbox-dev