Hi there
I'm trying to assess bottlenecks in the Moose importer. I think there are some hidden scalability issues especially in the InstallElementsOperator and the CandidateListOperator. Now I cant run a profiler on that because the image crashes before the end, so I just randomly pick some interruption from time to time (especially when I dont have feedback by a progress bar).
Some suspect code:
InstallElementsOperator >>basicRun model entityStorage forSetup. self container do: [:each | model add: each. self increment]. model entityStorage forRuntime <------- here: converting to a runtime storage takes a long time
CandidateListOperator >>setUp "dont call 'model allMethods' here because it sorts the group and we dont care at this level" groups := ((model allWithType: FAMIXMethod) reject: #isAbstract) groupedBy: #signature <----- groupedBy: create a dictionary with all method signatures, I suspect it becomes inefficient on large models
Any taker/suggestion on this?
-----------------------------
Also I think it would be good to launch the importer in a separate/ background thread, with the possibility to cancel it.
What we can do basically (for example): [MooseScripts createModelForPharo] forkAt: 20 (give it a low priority to not starve other process, esp. UI) and we can still interrupt the process in
Now if someone knowledgeable in Morphic can come with an enhanced progress bar with a cancel button, which would terminate the import process when clicked.
-- Simon
I have been working on a new profiler for Pharo. And I would like the moose importer to be a case study for it. But really, I need more time for this. Now, I am fixing bugs in Mondrian and adding features Jannik asked for. Johan is now putting pressure on me to get popupView: working.
Cheers, Alexandre
On 1 Dec 2009, at 12:17, Simon Denier wrote:
Hi there
I'm trying to assess bottlenecks in the Moose importer. I think there are some hidden scalability issues especially in the InstallElementsOperator and the CandidateListOperator. Now I cant run a profiler on that because the image crashes before the end, so I just randomly pick some interruption from time to time (especially when I dont have feedback by a progress bar).
Some suspect code:
InstallElementsOperator >>basicRun model entityStorage forSetup. self container do: [:each | model add: each. self increment]. model entityStorage forRuntime <------- here: converting to a runtime storage takes a long time
CandidateListOperator >>setUp
"dont call 'model allMethods' here because it sorts the group and we dont care at this level" groups := ((model allWithType: FAMIXMethod) reject: #isAbstract) groupedBy: #signature <----- groupedBy: create a dictionary with all method signatures, I suspect it becomes inefficient on large models
Any taker/suggestion on this?
Also I think it would be good to launch the importer in a separate/ background thread, with the possibility to cancel it.
What we can do basically (for example): [MooseScripts createModelForPharo] forkAt: 20 (give it a low priority to not starve other process, esp. UI) and we can still interrupt the process in
Now if someone knowledgeable in Morphic can come with an enhanced progress bar with a cancel button, which would terminate the import process when clicked.
-- Simon
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
this is really strange. I'm really wonder why the image crashes.
Hi there
I'm trying to assess bottlenecks in the Moose importer. I think there are some hidden scalability issues especially in the InstallElementsOperator and the CandidateListOperator. Now I cant run a profiler on that because the image crashes before the end, so I just randomly pick some interruption from time to time (especially when I dont have feedback by a progress bar).
Some suspect code:
InstallElementsOperator >>basicRun model entityStorage forSetup. self container do: [:each | model add: each. self increment]. model entityStorage forRuntime <------- here: converting to a runtime storage takes a long time
CandidateListOperator >>setUp
"dont call 'model allMethods' here because it sorts the group and we dont care at this level" groups := ((model allWithType: FAMIXMethod) reject: #isAbstract) groupedBy: #signature <----- groupedBy: create a dictionary with all method signatures, I suspect it becomes inefficient on large models
Any taker/suggestion on this?
Also I think it would be good to launch the importer in a separate/background thread, with the possibility to cancel it.
What we can do basically (for example): [MooseScripts createModelForPharo] forkAt: 20 (give it a low priority to not starve other process, esp. UI) and we can still interrupt the process in
Now if someone knowledgeable in Morphic can come with an enhanced progress bar with a cancel button, which would terminate the import process when clicked.
-- Simon
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
On 1 déc. 2009, at 12:17, Simon Denier wrote:
Hi there
I'm trying to assess bottlenecks in the Moose importer. I think there are some hidden scalability issues especially in the InstallElementsOperator and the CandidateListOperator. Now I cant run a profiler on that because the image crashes before the end, so I just randomly pick some interruption from time to time (especially when I dont have feedback by a progress bar).
Some suspect code:
InstallElementsOperator >>basicRun model entityStorage forSetup. self container do: [:each | model add: each. self increment]. model entityStorage forRuntime <------- here: converting to a runtime storage takes a long time
CandidateListOperator >>setUp
"dont call 'model allMethods' here because it sorts the group and we dont care at this level" groups := ((model allWithType: FAMIXMethod) reject: #isAbstract) groupedBy: #signature <----- groupedBy: create a dictionary with all method signatures, I suspect it becomes inefficient on large models
Any taker/suggestion on this?
Also I think it would be good to launch the importer in a separate/ background thread, with the possibility to cancel it.
What we can do basically (for example): [MooseScripts createModelForPharo] forkAt: 20 (give it a low priority to not starve other process, esp. UI) and we can still interrupt the process in
ok so it finally works in less than one hour (pharo 10493 dev image). Thanks to Jannik for the tip on increasing VM memory at start up (it's strange that it didnt complain about low memory in previous runs), it was the key. Running in a separate thread is also a good idea and the UI is still responsive despite the heavy computation. I think I will open a request for enhancement :)
As said, there is something to be done on the methods above. Perhaps it's just the lack of feedback, but computing candidates feels faster than installing/setting up the model for that.
So here is a picture of 142 packages in pharo 10493, with packages belonging to the same big cycle sharing the same color
Now if someone knowledgeable in Morphic can come with an enhanced progress bar with a cancel button, which would terminate the import process when clicked.
-- Simon
-- Simon
This is excellent Simon! 1 hour to import the whole Pharo is nothing.
Alexandre
On 1 Dec 2009, at 16:01, Simon Denier wrote:
On 1 déc. 2009, at 12:17, Simon Denier wrote:
Hi there
I'm trying to assess bottlenecks in the Moose importer. I think there are some hidden scalability issues especially in the InstallElementsOperator and the CandidateListOperator. Now I cant run a profiler on that because the image crashes before the end, so I just randomly pick some interruption from time to time (especially when I dont have feedback by a progress bar).
Some suspect code:
InstallElementsOperator >>basicRun model entityStorage forSetup. self container do: [:each | model add: each. self increment]. model entityStorage forRuntime <------- here: converting to a runtime storage takes a long time
CandidateListOperator >>setUp
"dont call 'model allMethods' here because it sorts the group and we dont care at this level" groups := ((model allWithType: FAMIXMethod) reject: #isAbstract) groupedBy: #signature <----- groupedBy: create a dictionary with all method signatures, I suspect it becomes inefficient on large models
Any taker/suggestion on this?
Also I think it would be good to launch the importer in a separate/ background thread, with the possibility to cancel it.
What we can do basically (for example): [MooseScripts createModelForPharo] forkAt: 20 (give it a low priority to not starve other process, esp. UI) and we can still interrupt the process in
ok so it finally works in less than one hour (pharo 10493 dev image). Thanks to Jannik for the tip on increasing VM memory at start up (it's strange that it didnt complain about low memory in previous runs), it was the key. Running in a separate thread is also a good idea and the UI is still responsive despite the heavy computation. I think I will open a request for enhancement :)
As said, there is something to be done on the methods above. Perhaps it's just the lack of feedback, but computing candidates feels faster than installing/setting up the model for that.
So here is a picture of 142 packages in pharo 10493, with packages belonging to the same big cycle sharing the same color
<PharoPackageCycles.png>
Now if someone knowledgeable in Morphic can come with an enhanced progress bar with a cancel button, which would terminate the import process when clicked.
-- Simon
-- Simon
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
On Dec 1, 2009, at 8:38 PM, Alexandre Bergel wrote:
This is excellent Simon! 1 hour to import the whole Pharo is nothing.
Seriously 1 hour? We can still import the complete VW in a couple of minutes in VW
Alexandre
On 1 Dec 2009, at 16:01, Simon Denier wrote:
On 1 déc. 2009, at 12:17, Simon Denier wrote:
Hi there
I'm trying to assess bottlenecks in the Moose importer. I think there are some hidden scalability issues especially in the InstallElementsOperator and the CandidateListOperator. Now I cant run a profiler on that because the image crashes before the end, so I just randomly pick some interruption from time to time (especially when I dont have feedback by a progress bar).
Some suspect code:
InstallElementsOperator >>basicRun model entityStorage forSetup. self container do: [:each | model add: each. self increment]. model entityStorage forRuntime <------- here: converting to a runtime storage takes a long time
CandidateListOperator >>setUp
"dont call 'model allMethods' here because it sorts the group and we dont care at this level" groups := ((model allWithType: FAMIXMethod) reject: #isAbstract) groupedBy: #signature <----- groupedBy: create a dictionary with all method signatures, I suspect it becomes inefficient on large models
Any taker/suggestion on this?
Also I think it would be good to launch the importer in a separate/background thread, with the possibility to cancel it.
What we can do basically (for example): [MooseScripts createModelForPharo] forkAt: 20 (give it a low priority to not starve other process, esp. UI) and we can still interrupt the process in
ok so it finally works in less than one hour (pharo 10493 dev image). Thanks to Jannik for the tip on increasing VM memory at start up (it's strange that it didnt complain about low memory in previous runs), it was the key. Running in a separate thread is also a good idea and the UI is still responsive despite the heavy computation. I think I will open a request for enhancement :)
As said, there is something to be done on the methods above. Perhaps it's just the lack of feedback, but computing candidates feels faster than installing/setting up the model for that.
So here is a picture of 142 packages in pharo 10493, with packages belonging to the same big cycle sharing the same color
<PharoPackageCycles.png>
Now if someone knowledgeable in Morphic can come with an enhanced progress bar with a cancel button, which would terminate the import process when clicked.
-- Simon
-- Simon
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
1 hour is very slow. I just tested and my image crash...
Jannik
On Dec 2, 2009, at 12:28 , Stéphane Ducasse wrote:
On Dec 1, 2009, at 8:38 PM, Alexandre Bergel wrote:
This is excellent Simon! 1 hour to import the whole Pharo is nothing.
Seriously 1 hour? We can still import the complete VW in a couple of minutes in VW
Alexandre
On 1 Dec 2009, at 16:01, Simon Denier wrote:
On 1 déc. 2009, at 12:17, Simon Denier wrote:
Hi there
I'm trying to assess bottlenecks in the Moose importer. I think there are some hidden scalability issues especially in the InstallElementsOperator and the CandidateListOperator. Now I cant run a profiler on that because the image crashes before the end, so I just randomly pick some interruption from time to time (especially when I dont have feedback by a progress bar).
Some suspect code:
InstallElementsOperator >>basicRun model entityStorage forSetup. self container do: [:each | model add: each. self increment]. model entityStorage forRuntime <------- here: converting to a runtime storage takes a long time
CandidateListOperator >>setUp
"dont call 'model allMethods' here because it sorts the group and we dont care at this level" groups := ((model allWithType: FAMIXMethod) reject: #isAbstract) groupedBy: #signature <----- groupedBy: create a dictionary with all method signatures, I suspect it becomes inefficient on large models
Any taker/suggestion on this?
Also I think it would be good to launch the importer in a separate/background thread, with the possibility to cancel it.
What we can do basically (for example): [MooseScripts createModelForPharo] forkAt: 20 (give it a low priority to not starve other process, esp. UI) and we can still interrupt the process in
ok so it finally works in less than one hour (pharo 10493 dev image). Thanks to Jannik for the tip on increasing VM memory at start up (it's strange that it didnt complain about low memory in previous runs), it was the key. Running in a separate thread is also a good idea and the UI is still responsive despite the heavy computation. I think I will open a request for enhancement :)
As said, there is something to be done on the methods above. Perhaps it's just the lack of feedback, but computing candidates feels faster than installing/setting up the model for that.
So here is a picture of 142 packages in pharo 10493, with packages belonging to the same big cycle sharing the same color
<PharoPackageCycles.png>
Now if someone knowledgeable in Morphic can come with an enhanced progress bar with a cancel button, which would terminate the import process when clicked.
-- Simon
-- Simon
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
--- Jannik Laval PhD Student - Rmod Team - INRIA Certified Project Management Associate (IPMA) http://www.jannik-laval.eu http://rmod.lille.inria.fr ---
It is indeed slow. But crashing is plain bad :).
Doru
On 2 Dec 2009, at 14:14, Laval Jannik wrote:
1 hour is very slow. I just tested and my image crash...
Jannik
On Dec 2, 2009, at 12:28 , Stéphane Ducasse wrote:
On Dec 1, 2009, at 8:38 PM, Alexandre Bergel wrote:
This is excellent Simon! 1 hour to import the whole Pharo is nothing.
Seriously 1 hour? We can still import the complete VW in a couple of minutes in VW
Alexandre
On 1 Dec 2009, at 16:01, Simon Denier wrote:
On 1 déc. 2009, at 12:17, Simon Denier wrote:
Hi there
I'm trying to assess bottlenecks in the Moose importer. I think there are some hidden scalability issues especially in the InstallElementsOperator and the CandidateListOperator. Now I cant run a profiler on that because the image crashes before the end, so I just randomly pick some interruption from time to time (especially when I dont have feedback by a progress bar).
Some suspect code:
InstallElementsOperator >>basicRun model entityStorage forSetup. self container do: [:each | model add: each. self increment]. model entityStorage forRuntime <------- here: converting to a runtime storage takes a long time
CandidateListOperator >>setUp
"dont call 'model allMethods' here because it sorts the group and we dont care at this level" groups := ((model allWithType: FAMIXMethod) reject: #isAbstract) groupedBy: #signature <----- groupedBy: create a dictionary with all method signatures, I suspect it becomes inefficient on large models
Any taker/suggestion on this?
Also I think it would be good to launch the importer in a separate/background thread, with the possibility to cancel it.
What we can do basically (for example): [MooseScripts createModelForPharo] forkAt: 20 (give it a low priority to not starve other process, esp. UI) and we can still interrupt the process in
ok so it finally works in less than one hour (pharo 10493 dev image). Thanks to Jannik for the tip on increasing VM memory at start up (it's strange that it didnt complain about low memory in previous runs), it was the key. Running in a separate thread is also a good idea and the UI is still responsive despite the heavy computation. I think I will open a request for enhancement :)
As said, there is something to be done on the methods above. Perhaps it's just the lack of feedback, but computing candidates feels faster than installing/setting up the model for that.
So here is a picture of 142 packages in pharo 10493, with packages belonging to the same big cycle sharing the same color
<PharoPackageCycles.png>
Now if someone knowledgeable in Morphic can come with an enhanced progress bar with a cancel button, which would terminate the import process when clicked.
-- Simon
-- Simon
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
Jannik Laval PhD Student - Rmod Team - INRIA Certified Project Management Associate (IPMA) http://www.jannik-laval.eu http://rmod.lille.inria.fr
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
-- www.tudorgirba.com
"Speaking louder won't make the point worthier."
Hi Simon,
The forRuntime is actually supposed to be an optimization that comes from the VW time. However, I do not know if it holds true in Pharo.
The groupedBy: sounds like a possible problem indeed.
What was the VM trick?
Cheers, Doru
On 1 Dec 2009, at 20:01, Simon Denier wrote:
On 1 déc. 2009, at 12:17, Simon Denier wrote:
Hi there
I'm trying to assess bottlenecks in the Moose importer. I think there are some hidden scalability issues especially in the InstallElementsOperator and the CandidateListOperator. Now I cant run a profiler on that because the image crashes before the end, so I just randomly pick some interruption from time to time (especially when I dont have feedback by a progress bar).
Some suspect code:
InstallElementsOperator >>basicRun model entityStorage forSetup. self container do: [:each | model add: each. self increment]. model entityStorage forRuntime <------- here: converting to a runtime storage takes a long time
CandidateListOperator >>setUp
"dont call 'model allMethods' here because it sorts the group and we dont care at this level" groups := ((model allWithType: FAMIXMethod) reject: #isAbstract) groupedBy: #signature <----- groupedBy: create a dictionary with all method signatures, I suspect it becomes inefficient on large models
Any taker/suggestion on this?
Also I think it would be good to launch the importer in a separate/ background thread, with the possibility to cancel it.
What we can do basically (for example): [MooseScripts createModelForPharo] forkAt: 20 (give it a low priority to not starve other process, esp. UI) and we can still interrupt the process in
ok so it finally works in less than one hour (pharo 10493 dev image). Thanks to Jannik for the tip on increasing VM memory at start up (it's strange that it didnt complain about low memory in previous runs), it was the key. Running in a separate thread is also a good idea and the UI is still responsive despite the heavy computation. I think I will open a request for enhancement :)
As said, there is something to be done on the methods above. Perhaps it's just the lack of feedback, but computing candidates feels faster than installing/setting up the model for that.
So here is a picture of 142 packages in pharo 10493, with packages belonging to the same big cycle sharing the same color
<PharoPackageCycles.png>
Now if someone knowledgeable in Morphic can come with an enhanced progress bar with a cancel button, which would terminate the import process when clicked.
-- Simon
-- Simon
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
-- www.tudorgirba.com
"Value is always contextual."
On 1 déc. 2009, at 16:50, Tudor Girba wrote:
Hi Simon,
The forRuntime is actually supposed to be an optimization that comes from the VW time. However, I do not know if it holds true in Pharo.
The rationale behind switching from Setup to Runtime is good I think, although difficult to assess. The problem here is that it just takes a loooong time to copy all entities from one collection to the other. One could try #primitiveChangeClassTo: but I dont think it will work given the restrictions on this method.
The groupedBy: sounds like a possible problem indeed.
What was the VM trick?
Launch VM from command line with parameter -memory
Squeak\ VM\ Opt -memory 1500m ~/Pharo/Moose-112009/moose10493c.pharo. 1.image
Cheers, Doru
On 1 Dec 2009, at 20:01, Simon Denier wrote:
On 1 déc. 2009, at 12:17, Simon Denier wrote:
Hi there
I'm trying to assess bottlenecks in the Moose importer. I think there are some hidden scalability issues especially in the InstallElementsOperator and the CandidateListOperator. Now I cant run a profiler on that because the image crashes before the end, so I just randomly pick some interruption from time to time (especially when I dont have feedback by a progress bar).
Some suspect code:
InstallElementsOperator >>basicRun model entityStorage forSetup. self container do: [:each | model add: each. self increment]. model entityStorage forRuntime <------- here: converting to a runtime storage takes a long time
CandidateListOperator >>setUp
"dont call 'model allMethods' here because it sorts the group and we dont care at this level" groups := ((model allWithType: FAMIXMethod) reject: #isAbstract) groupedBy: #signature <----- groupedBy: create a dictionary with all method signatures, I suspect it becomes inefficient on large models
Any taker/suggestion on this?
Also I think it would be good to launch the importer in a separate/ background thread, with the possibility to cancel it.
What we can do basically (for example): [MooseScripts createModelForPharo] forkAt: 20 (give it a low priority to not starve other process, esp. UI) and we can still interrupt the process in
ok so it finally works in less than one hour (pharo 10493 dev image). Thanks to Jannik for the tip on increasing VM memory at start up (it's strange that it didnt complain about low memory in previous runs), it was the key. Running in a separate thread is also a good idea and the UI is still responsive despite the heavy computation. I think I will open a request for enhancement :)
As said, there is something to be done on the methods above. Perhaps it's just the lack of feedback, but computing candidates feels faster than installing/setting up the model for that.
So here is a picture of 142 packages in pharo 10493, with packages belonging to the same big cycle sharing the same color
<PharoPackageCycles.png>
Now if someone knowledgeable in Morphic can come with an enhanced progress bar with a cancel button, which would terminate the import process when clicked.
-- Simon
-- Simon
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
-- www.tudorgirba.com
"Value is always contextual."
Moose-dev mailing list Moose-dev@iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev
-- Simon
-
Alexandre Bergel -
Laval Jannik -
Simon Denier -
Stéphane Ducasse -
Tudor Girba