[vbox-dev] Too Many Snapshots: XML_PARSE_HUGE and NS_ERROR_CALL

Discussion:

[vbox-dev] Too Many Snapshots: XML_PARSE_HUGE and NS_ERROR_CALL_FAILED

2014-06-30 16:28:55 UTC

So we are trying to get a problem with VirtualBox working. We are
heavily using the snapshot feature and hit a wall, as we probably have
exceeded the maximum possible amount of active snapshots. As each
Snapshot is written into the VBOX-file as a descendent of previous
Snapshot, at one point, the nesting exceeds 256 entries - which is an
arbitrary limit set by the libxml2. When trying to register a VM which
has too many Snapshots, vboxmanage registervm fails with an error.

We tried to work around this issue simply by adding the corresponding
XML_PARSE_HUGE flag in the function that calls to the xml-library. We
also created a patch:

--- virtualbox-4.1.12-dfsg.orig/src/VBox/Runtime/r3/xml.cpp
+++ virtualbox-4.1.12-dfsg/src/VBox/Runtime/r3/xml.cpp
@@ -1500,7 +1500,7 @@ void XmlMemParser::read(const void* pvBu
(int)cbSize,
pcszFilename,
NULL, //
encoding = auto
- XML_PARSE_NOBLANKS | XML_PARSE_NONET)))
+ XML_PARSE_NOBLANKS | XML_PARSE_NONET | XML_PARSE_HUGE)))
throw XmlError(xmlCtxtGetLastError(m_ctxt));

doc.refreshInternals();
@@ -1630,7 +1630,7 @@ void XmlFileParser::read(const RTCString
&context,
pcszFilename,
NULL, // encoding
= auto
- XML_PARSE_NOBLANKS |
XML_PARSE_NONET)))
+ XML_PARSE_NOBLANKS |
XML_PARSE_NONET | XML_PARSE_HUGE)))
throw XmlError(xmlCtxtGetLastError(m_ctxt));

doc.refreshInternals();

We successfully rebuild the source with that flag usign
dpkg-buildpackage. That successfully resolved the previous error,
however we now have the following problem by VBoxManage when trying to
register the VM in question:

vboxmanage registervm /mnt/storage/vm-vbox
VBoxManage: error: Code NS_ERROR_CALL_FAILED (0x800706BE) - Call to
remote object failed (extended info not available)
Context: "OpenMachine(Bstr(a->argv[0]).raw(), machine.asOutParam())" at
line 90 of file VBoxManageMisc.cpp

Unfortunately, we have no idea how to work with that error.

This is the VBoxSVC.log:
VirtualBox (XP)COM Server 4.1.12_Ubuntu r77245 linux.amd64 (Jun 30 2014
16:40:09) release log
00:00:00.001 main Log opened 2014-06-30T15:23:31.882224000Z
00:00:00.001 main OS Product: Linux
00:00:00.001 main OS Release: 3.11.0-24-generic
00:00:00.001 main OS Version: #41~precise1-Ubuntu SMP Wed Jun 11
13:15:06 UTC 2014
00:00:00.001 main OS Service Pack: #41~precise1-Ubuntu SMP Wed Jun
11 13:15:06 UTC 2014
00:00:00.001 main Executable: /usr/lib/virtualbox/VBoxSVC
00:00:00.001 main Process ID: 6411
00:00:00.001 main Package type: LINUX_64BITS_GENERIC (OSE)
00:00:00.107 nspr-2 Loading settings file
"/home/user/.VirtualBox/VirtualBox.xml" with version "1.12-linux"
00:00:00.120 nspr-2 Successfully initialised host USB using sysfs
00:00:00.135 nspr-2 VDInit finished
00:00:00.200 nspr-2 Loading settings file "/mnt/storage/vm.vbox" with
version "1.12-linux"

Any help would be immensely appreciated.

Klaus Espenlaub

2014-07-01 12:23:02 UTC

Permalink

not sure if this is something worth fixing, as a huge number of
snapshots causes trouble elsewhere (stack overflows). That's the reason
why the snapshot nesting depth has been limited to 250 with VirtualBox
4.3.0.

Post by a.
So we are trying to get a problem with VirtualBox working. We are
heavily using the snapshot feature and hit a wall, as we probably have
exceeded the maximum possible amount of active snapshots. As each
Snapshot is written into the VBOX-file as a descendent of previous
Snapshot, at one point, the nesting exceeds 256 entries - which is an
arbitrary limit set by the libxml2. When trying to register a VM which
has too many Snapshots, vboxmanage registervm fails with an error.

I've checked the code of libxml 2.6.31, and the only occurrence of
XML_PARSE_HUGE is in some documentation. This means your patch will
break our builds by referring to an undefined symbol.

Actually I never saw trouble parsing big XML files with the libxml2
version we're using in all our builds, 2.6.31. It sets the variable
xmlParserMaxDepth to 1024, which is plenty.

Post by a.
We tried to work around this issue simply by adding the corresponding
XML_PARSE_HUGE flag in the function that calls to the xml-library. We
--- virtualbox-4.1.12-dfsg.orig/src/VBox/Runtime/r3/xml.cpp
+++ virtualbox-4.1.12-dfsg/src/VBox/Runtime/r3/xml.cpp
@@ -1500,7 +1500,7 @@ void XmlMemParser::read(const void* pvBu
(int)cbSize,
pcszFilename,
NULL, //
encoding = auto
- XML_PARSE_NOBLANKS | XML_PARSE_NONET)))
+ XML_PARSE_NOBLANKS | XML_PARSE_NONET | XML_PARSE_HUGE)))

XML_PARSE_HUGE was apparently introduced with version 2.7.3, and thus
this needs proper version checking to make it an acceptable patch.

Post by a.
throw XmlError(xmlCtxtGetLastError(m_ctxt));
doc.refreshInternals();
@@ -1630,7 +1630,7 @@ void XmlFileParser::read(const RTCString
&context,
pcszFilename,
NULL, // encoding
= auto
- XML_PARSE_NOBLANKS |
XML_PARSE_NONET)))
+ XML_PARSE_NOBLANKS |
XML_PARSE_NONET | XML_PARSE_HUGE)))
throw XmlError(xmlCtxtGetLastError(m_ctxt));
doc.refreshInternals();
We successfully rebuild the source with that flag usign
dpkg-buildpackage.

Would only compile with a new enough libxml2...

Post by a.
That successfully resolved the previous error,
however we now have the following problem by VBoxManage when trying to
vboxmanage registervm /mnt/storage/vm-vbox
VBoxManage: error: Code NS_ERROR_CALL_FAILED (0x800706BE) - Call to
remote object failed (extended info not available)
Context: "OpenMachine(Bstr(a->argv[0]).raw(), machine.asOutParam())" at
line 90 of file VBoxManageMisc.cpp
Unfortunately, we have no idea how to work with that error.

That's the mentioned stack overflow elsewhere in the API. When XPCOM is
used (i.e. on all platforms besides Windows), the stack of each thread
is limited to 128K, to maximize the scalability. That's not a lot...

Post by a.
VirtualBox (XP)COM Server 4.1.12_Ubuntu r77245 linux.amd64 (Jun 30 2014
16:40:09) release log

[...]

Post by a.
00:00:00.200 nspr-2 Loading settings file "/mnt/storage/vm.vbox" with
version "1.12-linux"

Rather clear hint that the processing of this settings file fails after
the XML parsing step.

Post by a.
Any help would be immensely appreciated.

2014-07-01 13:10:35 UTC

Permalink

Thanks for your reply!

We have created a testing environment which relies rather heavily on the
Snapshot feature including, afaik, the nesting feature. As we've read
elsewhere, the actual snapshot count is supposed to be unlimited which
is why we did not expect this problem.

For the time being we have decided to stop this feature when reaching a
particular Snapshot count to clean up and merge existing snapshots and
then carry on. Although that would mean losing particular states. In any
case we already do have one VM that has exceeded that limit and are now
facing the issue that we cannot clone or modify it, because we cannot
register the vm. Do you see any way how we can register that VM so we
can bring it into a working format?

Also personally I am confused how the snapshot count is supposed to be
unlimited although too much nesting results in stack overflows or forced
breaks. Maybe I've misunderstood something about the usages of Snapshots
in VBox: Is there a possibility to reduce the Snapshot-Disk-Nesting
without losing the ability to revert back to a previous Snapshot-State?
Even if we'd had, say, 300 Snapshots, would it still be possible to
revert back to Snapshot #58? That is basically the feature we tried to
use here.

Thanks!
Alex.

Post by Klaus Espenlaub
not sure if this is something worth fixing, as a huge number of
snapshots causes trouble elsewhere (stack overflows). That's the reason
why the snapshot nesting depth has been limited to 250 with VirtualBox
4.3.0.

I've checked the code of libxml 2.6.31, and the only occurrence of
XML_PARSE_HUGE is in some documentation. This means your patch will
break our builds by referring to an undefined symbol.
Actually I never saw trouble parsing big XML files with the libxml2
version we're using in all our builds, 2.6.31. It sets the variable
xmlParserMaxDepth to 1024, which is plenty.

XML_PARSE_HUGE was apparently introduced with version 2.7.3, and thus
this needs proper version checking to make it an acceptable patch.

Would only compile with a new enough libxml2...

Post by a.
VirtualBox (XP)COM Server 4.1.12_Ubuntu r77245 linux.amd64 (Jun 30 2014
16:40:09) release log

[...]

Post by a.
00:00:00.200 nspr-2 Loading settings file "/mnt/storage/vm.vbox" with
version "1.12-linux"

Rather clear hint that the processing of this settings file fails after
the XML parsing step.

Post by a.
Any help would be immensely appreciated.

Do you truly have a use case which can't exist without such a deep
snapshot nesting (the total snapshot count is still unlimited)? It costs
some efficiency to have data spread across so many disk images.
Klaus
_______________________________________________
vbox-dev mailing list
vbox-dev at virtualbox.org
https://www.virtualbox.org/mailman/listinfo/vbox-dev

Klaus Espenlaub

2014-07-03 15:59:02 UTC

Permalink

Alex,

Post by a.
Thanks for your reply!
We have created a testing environment which relies rather heavily on the
Snapshot feature including, afaik, the nesting feature. As we've read
elsewhere, the actual snapshot count is supposed to be unlimited which
is why we did not expect this problem.
For the time being we have decided to stop this feature when reaching a
particular Snapshot count to clean up and merge existing snapshots and
then carry on. Although that would mean losing particular states. In any
case we already do have one VM that has exceeded that limit and are now
facing the issue that we cannot clone or modify it, because we cannot
register the vm. Do you see any way how we can register that VM so we
can bring it into a working format?

That's a really tough one... you could try with an old (before 4.3.0)
generic linux package, because those use a libxml2 version which
predates the change in default nesting limit (the one we use allows
1024, which really is plenty).

Post by a.
Also personally I am confused how the snapshot count is supposed to be
unlimited although too much nesting results in stack overflows or forced
breaks.

Some of the snapshot processing (especially operations on the entire
tree) is done with recursion, and that means each nesting needs a
certain amount of stack space. Given that the stack is limited to 128K
there isn't much each nesting can use... this is the real cause why we
limited the nesting depth. As mentioned, I wasn't aware of libxml2
making silly default changes. Never ran into them.

If someone wants to contribute a complete (and tested!) change which
increases the snapshot depth to say 500, we'd be listening. It's a
rather low priority item, because the limit is very high, and having
hundreds of them can decrease the VM performance (especially disk
access). Hard to tell by how much.

Post by a.
Maybe I've misunderstood something about the usages of Snapshots
in VBox: Is there a possibility to reduce the Snapshot-Disk-Nesting
without losing the ability to revert back to a previous Snapshot-State?
Even if we'd had, say, 300 Snapshots, would it still be possible to
revert back to Snapshot #58? That is basically the feature we tried to
use here.

I can't truly tell from this description if you have any means of
controlling the depth. My guess is you can't easily, because you're
recording the past of a particular VM run, or probably a sequence of VM
runs (with some limited restoring). This makes the tree deep. Making it
wider only works by restoring a rather old snapshot and using this for
an "alternative" run.

Klaus

Post by a.
Thanks!
Alex.

I've checked the code of libxml 2.6.31, and the only occurrence of
XML_PARSE_HUGE is in some documentation. This means your patch will
break our builds by referring to an undefined symbol.
Actually I never saw trouble parsing big XML files with the libxml2
version we're using in all our builds, 2.6.31. It sets the variable
xmlParserMaxDepth to 1024, which is plenty.

XML_PARSE_HUGE was apparently introduced with version 2.7.3, and thus
this needs proper version checking to make it an acceptable patch.

Would only compile with a new enough libxml2...

Post by a.
VirtualBox (XP)COM Server 4.1.12_Ubuntu r77245 linux.amd64 (Jun 30 2014
16:40:09) release log

[...]

Post by a.
00:00:00.200 nspr-2 Loading settings file "/mnt/storage/vm.vbox" with
version "1.12-linux"

Rather clear hint that the processing of this settings file fails after
the XML parsing step.

Post by a.
Any help would be immensely appreciated.