petit parser performance issues - Moose-dev

Tudor Girba

22 Aug 22 Aug

5:42 p.m.

Hi, Thanks for raising this. We did not have time to announce these changes properly. We integrated Jan's work on supporting island parsing and enhanced traceability. The features are really cool. To this end, the stream of petit parser was replaced with a first class context that holds the stream. We did expect a slight slowdown, but not a factor of 7 (which is what I get from your measurements) though. Jan? :) Doru On Fri, Aug 22, 2014 at 1:30 PM, Usman Bhatti <usman.bhatti(a)gmail.com> wrote:

...

Hi, Recent changes in PetitParser have caused parsing to slow down by a factor of 15-20. Here is an example run on freshly downloaded Moose image: [|string allStyles text| string := (PharoFilesOpener default changesFileOrNil contents copyFrom: 1 to: 60000). text := string asText. allStyles := { 'Announcement' -> TextColor green. 'Collections' -> TextColor blue. 'File' -> TextColor blue. 'Metacello' -> TextColor red. 'Monticello' -> TextColor magenta. 'Morphic' -> TextColor orange. 'Mooose' -> TextColor green. 'FAMIX' -> TextColor green. 'Roassal' -> TextColor green. }. allStyles do: [ :assoc | ((assoc key asParser, #newline asParser negate star) matchingRangesIn: string) do: [:aRange | text addAttribute: (assoc value) from: aRange first to: aRange last] ]. text] timeToRun ==> 15 sec ============== now do this: Gofer new url: 'http://smalltalkhub.com/mc/Moose/PetitParser/main'; version:'PetitParser-DiegoLont.233'; load. and run the example again. timeToRun ===> 2 sec So, we need to have a look at what is added recently in PetitParser that is making it slow down quite significantly because the for the larger systems, this can be enormous. regards, Usman P.S: Tests run on OS X 10.9.4, 2.3 GHz, 8 GB RAM, HD SSD. _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

-- www.tudorgirba.com "Every thing has its own flow"

Reply

Tudor Girba

4:02 p.m.

Hi Jan, Thanks a lot for looking into this. I checked, but I think the improvement you mention did not make it in the repo of PetitParser. Version JanKurs.238 is from August 20 (before the slowdown report): Name: PetitParser-JanKurs.238 Author: JanKurs Time: 20 August 2014, 10:13:55.838908 am UUID: a7316ed3-0751-4703-80c9-2790add1cc00 Ancestors: PetitParser-JanKurs.237 The latest version is JanKurs.239 and it is also from the same day: Name: PetitParser-JanKurs.239 Author: JanKurs Time: 20 August 2014, 12:13:26.196695 pm UUID: 8985ce0d-8d40-47ef-bd82-ead0d8695be2 Ancestors: PetitParser-JanKurs.238 Could you please double check? Cheers, Doru On Sun, Aug 24, 2014 at 3:56 PM, Jan Kurš <kurs.jan(a)gmail.com> wrote:

...

Hey, I can confirm the slowdown. The latest change (JanKurs.238, committed today) should return most of the performance back. Please, let me know, if you have any other issues Cheers, Jan On 22 August 2014 16:42, Tudor Girba <tudor(a)tudorgirba.com> wrote:

Hi, Thanks for raising this. We did not have time to announce these changes properly. We integrated Jan's work on supporting island parsing and enhanced traceability. The features are really cool. To this end, the stream of petit parser was replaced with a first class context that holds the stream. We did expect a slight slowdown, but not a factor of 7 (which is what I get from your measurements) though. Jan? :) Doru On Fri, Aug 22, 2014 at 1:30 PM, Usman Bhatti <usman.bhatti(a)gmail.com> wrote:

Hi, Recent changes in PetitParser have caused parsing to slow down by a factor of 15-20. Here is an example run on freshly downloaded Moose image: [|string allStyles text| string := (PharoFilesOpener default changesFileOrNil contents copyFrom: 1 to: 60000). text := string asText. allStyles := { 'Announcement' -> TextColor green. 'Collections' -> TextColor blue. 'File' -> TextColor blue. 'Metacello' -> TextColor red. 'Monticello' -> TextColor magenta. 'Morphic' -> TextColor orange. 'Mooose' -> TextColor green. 'FAMIX' -> TextColor green. 'Roassal' -> TextColor green. }. allStyles do: [ :assoc | ((assoc key asParser, #newline asParser negate star) matchingRangesIn: string) do: [:aRange | text addAttribute: (assoc value) from: aRange first to: aRange last] ]. text] timeToRun ==> 15 sec ============== now do this: Gofer new url: 'http://smalltalkhub.com/mc/Moose/PetitParser/main'; version:'PetitParser-DiegoLont.233'; load. and run the example again. timeToRun ===> 2 sec So, we need to have a look at what is added recently in PetitParser that is making it slow down quite significantly because the for the larger systems, this can be enormous. regards, Usman P.S: Tests run on OS X 10.9.4, 2.3 GHz, 8 GB RAM, HD SSD. _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

-- www.tudorgirba.com "Every thing has its own flow" _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

_______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

-- www.tudorgirba.com "Every thing has its own flow"

Reply

Tudor Girba

5:30 p.m.

Great! I tested it. The example of Usman went from 15s to 3s. @Usman: could you test and see if it is acceptable? Cheers, Doru On Sun, Aug 24, 2014 at 5:16 PM, Jan Kurš <kurs.jan(a)gmail.com> wrote:

...

Hey, sorry for a complication, the branching system is a too tricky for me. The version JanKurs.240 should do the thing. I just checked in a build Moose-50 #2222 (https://ci.inria.fr/moose/job/moose-5.0/2222/) Cheers, Jan On 24 August 2014 15:02, Tudor Girba <tudor(a)tudorgirba.com> wrote:

Hi Jan, Thanks a lot for looking into this. I checked, but I think the improvement you mention did not make it in the repo of PetitParser. Version JanKurs.238 is from August 20 (before the slowdown report): Name: PetitParser-JanKurs.238 Author: JanKurs Time: 20 August 2014, 10:13:55.838908 am UUID: a7316ed3-0751-4703-80c9-2790add1cc00 Ancestors: PetitParser-JanKurs.237 The latest version is JanKurs.239 and it is also from the same day: Name: PetitParser-JanKurs.239 Author: JanKurs Time: 20 August 2014, 12:13:26.196695 pm UUID: 8985ce0d-8d40-47ef-bd82-ead0d8695be2 Ancestors: PetitParser-JanKurs.238 Could you please double check? Cheers, Doru On Sun, Aug 24, 2014 at 3:56 PM, Jan Kurš <kurs.jan(a)gmail.com> wrote:

Hey, I can confirm the slowdown. The latest change (JanKurs.238, committed today) should return most of the performance back. Please, let me know, if you have any other issues Cheers, Jan On 22 August 2014 16:42, Tudor Girba <tudor(a)tudorgirba.com> wrote:

Hi, Thanks for raising this. We did not have time to announce these changes properly. We integrated Jan's work on supporting island parsing and enhanced traceability. The features are really cool. To this end, the stream of petit parser was replaced with a first class context that holds the stream. We did expect a slight slowdown, but not a factor of 7 (which is what I get from your measurements) though. Jan? :) Doru On Fri, Aug 22, 2014 at 1:30 PM, Usman Bhatti <usman.bhatti(a)gmail.com> wrote: > Hi, > > Recent changes in PetitParser have caused parsing to slow down by a > factor of 15-20. Here is an example run on freshly downloaded Moose image: > > > [|string allStyles text| > string := (PharoFilesOpener default changesFileOrNil contents > copyFrom: 1 to: 60000). > text := string asText. > allStyles := { > 'Announcement' -> TextColor green. > 'Collections' -> TextColor blue. > 'File' -> TextColor blue. > 'Metacello' -> TextColor red. > 'Monticello' -> TextColor magenta. > 'Morphic' -> TextColor orange. > 'Mooose' -> TextColor green. > 'FAMIX' -> TextColor green. > 'Roassal' -> TextColor green. > }. > allStyles do: [ :assoc | ((assoc key asParser, #newline asParser > negate star) matchingRangesIn: string) do: [:aRange | text addAttribute: > (assoc value) from: aRange first to: aRange last] ]. > text] timeToRun ==> 15 sec > > ============== > now do this: > > Gofer new url: 'http://smalltalkhub.com/mc/Moose/PetitParser/main'; > version:'PetitParser-DiegoLont.233'; load. > > and run the example again. > timeToRun ===> 2 sec > > So, we need to have a look at what is added recently in PetitParser > that is making it slow down quite significantly because the for the larger > systems, this can be enormous. > > regards, > > Usman > > P.S: Tests run on OS X 10.9.4, 2.3 GHz, 8 GB RAM, HD SSD. > > _______________________________________________ > Moose-dev mailing list > Moose-dev(a)iam.unibe.ch > https://www.iam.unibe.ch/mailman/listinfo/moose-dev > > -- www.tudorgirba.com "Every thing has its own flow" _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

_______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

-- www.tudorgirba.com "Every thing has its own flow" _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

_______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

-- www.tudorgirba.com "Every thing has its own flow"

Reply

Usman Bhatti

11:12 a.m.

On Fri, Aug 22, 2014 at 5:42 PM, Tudor Girba <tudor(a)tudorgirba.com> wrote:

...

Hi, Thanks for raising this. We did not have time to announce these changes properly. We integrated Jan's work on supporting island parsing and enhanced traceability. The features are really cool. To this end, the stream of petit parser was replaced with a first class context that holds the stream.

I understand that context is important when parsing a program for certain parsing related information. In my case, I am using PetitParser more as a syntax highlighter on the lines of PPTextHighlighter and in that case you dont need much of contextual information. All you need is token positions. Can we envisage, for such usage, a simplistic but more robust parser? usman

...

We did expect a slight slowdown, but not a factor of 7 (which is what I get from your measurements) though. Jan? :) Doru On Fri, Aug 22, 2014 at 1:30 PM, Usman Bhatti <usman.bhatti(a)gmail.com> wrote:

Hi, Recent changes in PetitParser have caused parsing to slow down by a factor of 15-20. Here is an example run on freshly downloaded Moose image: [|string allStyles text| string := (PharoFilesOpener default changesFileOrNil contents copyFrom: 1 to: 60000). text := string asText. allStyles := { 'Announcement' -> TextColor green. 'Collections' -> TextColor blue. 'File' -> TextColor blue. 'Metacello' -> TextColor red. 'Monticello' -> TextColor magenta. 'Morphic' -> TextColor orange. 'Mooose' -> TextColor green. 'FAMIX' -> TextColor green. 'Roassal' -> TextColor green. }. allStyles do: [ :assoc | ((assoc key asParser, #newline asParser negate star) matchingRangesIn: string) do: [:aRange | text addAttribute: (assoc value) from: aRange first to: aRange last] ]. text] timeToRun ==> 15 sec ============== now do this: Gofer new url: 'http://smalltalkhub.com/mc/Moose/PetitParser/main'; version:'PetitParser-DiegoLont.233'; load. and run the example again. timeToRun ===> 2 sec So, we need to have a look at what is added recently in PetitParser that is making it slow down quite significantly because the for the larger systems, this can be enormous. regards, Usman P.S: Tests run on OS X 10.9.4, 2.3 GHz, 8 GB RAM, HD SSD. _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

-- www.tudorgirba.com "Every thing has its own flow" _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

Reply

Usman Bhatti

4:11 p.m.

On Mon, Aug 25, 2014 at 12:50 PM, Thierry Goubier <thierry.goubier(a)gmail.com

...

wrote:

...

2014-08-25 11:12 GMT+02:00 Usman Bhatti <usman.bhatti(a)gmail.com>om>:

I understand that context is important when parsing a program for certain parsing related information. In my case, I am using PetitParser more as a syntax highlighter on the lines of PPTextHighlighter and in that case you dont need much of contextual information. All you need is token positions. Can we envisage, for such usage, a simplistic but more robust parser?

If you're more into that, then a lexer may be enough and a very efficient one can be built with SmaCC. What you're doing doesn't seems to require a parser: it is just matching

It was just a illustrative example to show the slowness of PParser due to the recent modifications. What I want to achieve is to be able to express keyword or lines matching (may be some day complete code blocks as well). We found that doing it with the parser is quite easy. It would be worth trying SmaCC to compare its performance with PP-based solution but I am not knowledgable about SmaCC. Any pointers to where should I start from? regards, Usman

...

Keyword [^\n] * isn't it? Thierry [|string allStyles text| string := (PharoFilesOpener default changesFileOrNil contents copyFrom: 1 to: 60000). text := string asText. allStyles := { 'Announcement' -> TextColor green. 'Collections' -> TextColor blue. 'File' -> TextColor blue. 'Metacello' -> TextColor red. 'Monticello' -> TextColor magenta. 'Morphic' -> TextColor orange. 'Mooose' -> TextColor green. 'FAMIX' -> TextColor green. 'Roassal' -> TextColor green. }. allStyles do: [ :assoc | ((assoc key asParser, #newline asParser negate star) matchingRangesIn: string) do: [:aRange | text addAttribute: (assoc value) from: aRange first to: aRange last] ]. text] timeToRun ==> 15 sec _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

Reply

Usman Bhatti

6:36 p.m.

Tx Thierry. I'd prefer to read and understand the basics of SmaCC first from the Pharo book for intelligent communication :). I'll let you know. Usman On Mon, Aug 25, 2014 at 4:48 PM, Thierry Goubier <thierry.goubier(a)gmail.com> wrote:

...

2014-08-25 16:11 GMT+02:00 Usman Bhatti <usman.bhatti(a)gmail.com>om>:

It was just a illustrative example to show the slowness of PParser due to the recent modifications. What I want to achieve is to be able to express keyword or lines matching (may be some day complete code blocks as well). We found that doing it with the parser is quite easy.

If there is little context around (deep tree structures, backtracking) then a lexer may be enough. If you give me a bit of context (or a petit parser description), I can work out the equivalent SmaCC grammar.

It would be worth trying SmaCC to compare its performance with PP-based solution but I am not knowledgable about SmaCC. Any pointers to where should I start from?

I updated the SmaCC chapter in the PharoForTheEnterprise book; it is also available as help inside Pharo itself. https://ci.inria.fr/pharo-contribution/job/PharoForTheEnterprise/ws/Smacc/S…

...

Thierry

regards, Usman

Keyword [^\n] * isn't it? Thierry [|string allStyles text| string := (PharoFilesOpener default changesFileOrNil contents copyFrom: 1 to: 60000). text := string asText. allStyles := { 'Announcement' -> TextColor green. 'Collections' -> TextColor blue. 'File' -> TextColor blue. 'Metacello' -> TextColor red. 'Monticello' -> TextColor magenta. 'Morphic' -> TextColor orange. 'Mooose' -> TextColor green. 'FAMIX' -> TextColor green. 'Roassal' -> TextColor green. }. allStyles do: [ :assoc | ((assoc key asParser, #newline asParser negate star) matchingRangesIn: string) do: [:aRange | text addAttribute: (assoc value) from: aRange first to: aRange last] ]. text] timeToRun ==> 15 sec _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

_______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

Reply

stepharo

27 Aug 27 Aug

11:12 p.m.

On 25/8/14 19:02, Thierry Goubier wrote:

...

Le 25/08/2014 18:36, Usman Bhatti a écrit :

Tx Thierry. I'd prefer to read and understand the basics of SmaCC first from the Pharo book for intelligent communication :). I'll let you know.

Well, I teach parsing with SmaCC, so I may help :)

Thierry do you have slides and related material? Stef

...

Thierry

Usman On Mon, Aug 25, 2014 at 4:48 PM, Thierry Goubier <thierry.goubier(a)gmail.com <mailto:thierry.goubier@gmail.com>> wrote: 2014-08-25 16:11 GMT+02:00 Usman Bhatti <usman.bhatti(a)gmail.com <mailto:usman.bhatti@gmail.com>>: It was just a illustrative example to show the slowness of PParser due to the recent modifications. What I want to achieve is to be able to express keyword or lines matching (may be some day complete code blocks as well). We found that doing it with the parser is quite easy. If there is little context around (deep tree structures, backtracking) then a lexer may be enough. If you give me a bit of context (or a petit parser description), I can work out the equivalent SmaCC grammar. It would be worth trying SmaCC to compare its performance with PP-based solution but I am not knowledgable about SmaCC. Any pointers to where should I start from? I updated the SmaCC chapter in the PharoForTheEnterprise book; it is also available as help inside Pharo itself. https://ci.inria.fr/pharo-contribution/job/PharoForTheEnterprise/ws/Smacc/S… Thierry regards, Usman Keyword [^\n] * isn't it? Thierry [|string allStyles text| string := (PharoFilesOpener default changesFileOrNil contents copyFrom: 1 to: 60000). text := string asText. allStyles := { 'Announcement' -> TextColor green. 'Collections' -> TextColor blue. 'File' -> TextColor blue. 'Metacello' -> TextColor red. 'Monticello' -> TextColor magenta. 'Morphic' -> TextColor orange. 'Mooose' -> TextColor green. 'FAMIX' -> TextColor green. 'Roassal' -> TextColor green. }. allStyles do: [ :assoc | ((assoc key asParser, #newline asParser negate star) matchingRangesIn: string) do: [:aRange | text addAttribute: (assoc value) from: aRange first to: aRange last] ]. text] timeToRun ==> 15 sec _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch <mailto:Moose-dev@iam.unibe.ch> https://www.iam.unibe.ch/mailman/listinfo/moose-dev _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch <mailto:Moose-dev@iam.unibe.ch> https://www.iam.unibe.ch/mailman/listinfo/moose-dev _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch <mailto:Moose-dev@iam.unibe.ch> https://www.iam.unibe.ch/mailman/listinfo/moose-dev _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

_______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

Reply

stepharo

29 Aug 29 Aug

10:06 a.m.

...

Thierry do you have slides and related material?

I should write some this year. At least if I manage to get some time between the current projects and the project proposals :(

ahh that..... I'm writing a new ERC proposals so I'm allowed to open Pharo for a couple of weeks. No fun.

Reply