Thank you for the trick Alex. We should put these information somewhere, maybe in a tuning chapter of Pharo Enterprise book ? Unfortunately even with the trick, I was unable to parse a 10Mb XML file ... Pharo memory grows up until 1.9 Go and crash after that.
I will try now with the pull parser.
On Sun, Jun 8, 2014 at 11:05 PM, Alexandre Bergel alexandre.bergel@me.com wrote:
Serge, you are using a mac don’t you? Have you tried to augment the memory of the image?
Open using a text editor the Info.plist file, contained in the VM folder. By checking on the internet, 1880000000 is apparently the biggest value possible. http://forum.world.st/OSX-squeak-crash-maximal-size-of-image-td2952312.html
Cheers, Alexandre
On Jun 8, 2014, at 4:35 PM, Serge Stinckwich serge.stinckwich@gmail.com wrote:
Thank you Doru for the help.
First case: 74Mb XML file. I'm able to load the string in memory but the XML parser crash the image before the end. Second case: 10Mb XML file. I'm able to load the string in memory, parse the XML file. When I start to process the XML file in order to create an object structure, the panel "Space is too low" appear.
I would like to give a try with the XMLPullParser. I guess this is working correctly, because I find it with the Configuration Browser.
On Sat, Jun 7, 2014 at 10:56 PM, Tudor Girba tudor@tudorgirba.com wrote:
Hi,
Ok, I looked a bit, and here is a way to get it to work. It is not at all ideal given that it loads the entire string before parsing it, but it surely will work with an xml of your size:
contents := fileReference readStreamDo: [ :stream | stream contents ]. svnlog := (XMLDOMParser on: contents ) documentReadLimit: contents size; parseDocument.
Cheers, Doru
On Fri, Jun 6, 2014 at 12:50 PM, Tudor Girba tudor@tudorgirba.com wrote:
I encountered a similar situation before, but there is a way to go beyond that limit.
I do not have the code at my disposal right now, but look at the XMLDOMParser constructor, and at some point you will see a hardcoded limit value. You should be able to pass another one in.
Cheers, Doru
On Fri, Jun 6, 2014 at 11:22 AM, Serge Stinckwich serge.stinckwich@gmail.com wrote:
Dear all,
I want to process large XML file (typically event logs with more than 80Mb data) and I'm not able to do that at the moment with the XML DOM parser (it says I reach the read limit after 3094 XML lines) and I guess I will have problem to manage such a large file in memory after that.
Should I switch to an event-driven XML parser in order to avoid loading all the XML file in memory ? Do we have such a parser for Pharo ?
-- Serge Stinckwich UCBN & UMI UMMISCO 209 (IRD/UPMC) Every DSL ends up being Smalltalk http://www.doesnotunderstand.org/ _______________________________________________ Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
-- www.tudorgirba.com
"Every thing has its own flow"
-- www.tudorgirba.com
"Every thing has its own flow"
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
-- Serge Stinckwich UCBN & UMI UMMISCO 209 (IRD/UPMC) Every DSL ends up being Smalltalk http://www.doesnotunderstand.org/ _______________________________________________ Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
