Hi Moose users,
as you may know, at Synectique we experiment lots of issues about memory usage. That's why i had a look at the number of unused instance variables in one of our system loaded with a big model from one of our main customer.
Total instance variables: 26.852.653 Used instance variables: 17.393.938 Empty collections: 5.023.508 Recoverable instance variables: 14.482.223
The recoverable instance variables are those with nil or an empty collection (or MultiValueLink)
As you can see, we can save a lot of memory :-)
Here is my (dirty) code to get that:
countUsedInstanceVariableInForSubInstances: aClass | usedInstNbr instNbr emptyCollNbr | usedInstNbr := 0. instNbr := 0. emptyCollNbr := 0. aClass allSubInstancesDo: [ :anEntity | instNbr := instNbr + anEntity class allInstVarNames size. anEntity class allInstVarNames doWithIndex: [ :e :i | (anEntity instVarAt: i) ifNotNil: [ :content | ([ content isEmpty. emptyCollNbr := emptyCollNbr + 1. false ] on: MessageNotUnderstood do: [ false ]) ifFalse: [ usedInstNbr := usedInstNbr + 1 ] ] ] ]. ^ {('Used instance variables' -> usedInstNbr). ('empty collections' -> emptyCollNbr). ('Recoverable instance variables' -> (instNbr - (usedInstNbr - emptyCollNbr))). ('Total instance variables' -> instNbr)}
Thanks Guillaume for sharing your code.
Alexandre
On Nov 7, 2016, at 8:16 AM, Guillaume Larcheveque guillaume.larcheveque@gmail.com wrote:
Hi Moose users,
as you may know, at Synectique we experiment lots of issues about memory usage. That's why i had a look at the number of unused instance variables in one of our system loaded with a big model from one of our main customer.
Total instance variables: 26.852.653 Used instance variables: 17.393.938 Empty collections: 5.023.508 Recoverable instance variables: 14.482.223
The recoverable instance variables are those with nil or an empty collection (or MultiValueLink)
As you can see, we can save a lot of memory :-)
Here is my (dirty) code to get that:
countUsedInstanceVariableInForSubInstances: aClass | usedInstNbr instNbr emptyCollNbr | usedInstNbr := 0. instNbr := 0. emptyCollNbr := 0. aClass allSubInstancesDo: [ :anEntity | instNbr := instNbr + anEntity class allInstVarNames size. anEntity class allInstVarNames doWithIndex: [ :e :i | (anEntity instVarAt: i) ifNotNil: [ :content | ([ content isEmpty. emptyCollNbr := emptyCollNbr + 1. false ] on: MessageNotUnderstood do: [ false ]) ifFalse: [ usedInstNbr := usedInstNbr + 1 ] ] ] ]. ^ {('Used instance variables' -> usedInstNbr). ('empty collections' -> emptyCollNbr). ('Recoverable instance variables' -> (instNbr - (usedInstNbr - emptyCollNbr))). ('Total instance variables' -> instNbr)}
-- Guillaume Larcheveque
Moose-dev mailing list Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
Well yes, there was a reason we introduced FMNullMultivalueLink. The state handling of MooseEntity and the deep inheritance with lots of unused instVars of Famix are the other main memory users
Stephan
Thanks
A task of Pavel will be to see how slots usage can reduce that space lost.
For the empty collection having a kind of - sharing an #() in a classVar - creating a new collection on use could be a solution.
Le 7/11/16 à 12:16, Guillaume Larcheveque a écrit :
Hi Moose users,
as you may know, at Synectique we experiment lots of issues about memory usage. That's why i had a look at the number of unused instance variables in one of our system loaded with a big model from one of our main customer.
Total instance variables: 26.852.653 Used instance variables: 17.393.938 Empty collections: 5.023.508 Recoverable instance variables: 14.482.223
The recoverable instance variables are those with nil or an empty collection (or MultiValueLink)
As you can see, we can save a lot of memory :-)
Here is my (dirty) code to get that:
countUsedInstanceVariableInForSubInstances: aClass | usedInstNbr instNbr emptyCollNbr | usedInstNbr := 0. instNbr := 0. emptyCollNbr := 0. aClass allSubInstancesDo: [ :anEntity | instNbr := instNbr + anEntity class allInstVarNames size. anEntity class allInstVarNames doWithIndex: [ :e :i | (anEntity instVarAt: i) ifNotNil: [ :content | ([ content isEmpty. emptyCollNbr := emptyCollNbr + 1. false ] on: MessageNotUnderstood do: [ false ]) ifFalse: [ usedInstNbr := usedInstNbr + 1 ] ] ] ]. ^ {('Used instance variables' -> usedInstNbr). ('empty collections' -> emptyCollNbr). ('Recoverable instance variables' -> (instNbr - (usedInstNbr - emptyCollNbr))). ('Total instance variables' -> instNbr)}
-- *Guillaume Larcheveque*
Moose-dev mailing list Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
Reviving this old mail from Guillaume.
I did an analysis on a huge model (>2M entities) we have: - .mse file: 372.5 MB - .image file without the model: 86.9 MB - image with model: 692.8 MB (thus apparently 605,9 MB used by the model)
Then I did a small experiment: I removed the four instance variables of FAMIXSourcedEntity (sourceAnchor, sourceLanguage, comments, declaredSourceLanguage) and replaced them by variables in the privateState
for example:
FAMIXSourcedEntity>>comments "Returns the list of comments defined for the entity."
<MSEProperty: #comments type: #FAMIXComment opposite: #container> <multivalued> <derived> <MSEComment: 'list of comments defined in the entity'>
^comments
becomes
FAMIXSourcedEntity>>comments "Returns the list of comments defined for the entity."
<MSEProperty: #comments type: #FAMIXComment opposite: #container> <multivalued> <derived> <MSEComment: 'list of comments defined in the entity'> "^ comments" ^ self privateState cacheAt: #comments ifAbsentPut: nil
As a result, the .image file is 673.3 MB (thus apparently 586,4 MB used by the model) Not huge, but almost 20 MB saved, 3% of the initial memory
nicolas
On 10/11/2016 20:49, stepharo wrote:
Thanks
A task of Pavel will be to see how slots usage can reduce that space lost.
For the empty collection having a kind of - sharing an #() in a classVar - creating a new collection on use could be a solution.
Le 7/11/16 à 12:16, Guillaume Larcheveque a écrit :
Hi Moose users,
as you may know, at Synectique we experiment lots of issues about memory usage. That's why i had a look at the number of unused instance variables in one of our system loaded with a big model from one of our main customer.
Total instance variables: 26.852.653 Used instance variables: 17.393.938 Empty collections: 5.023.508 Recoverable instance variables: 14.482.223
The recoverable instance variables are those with nil or an empty collection (or MultiValueLink)
As you can see, we can save a lot of memory :-)
Here is my (dirty) code to get that:
countUsedInstanceVariableInForSubInstances: aClass | usedInstNbr instNbr emptyCollNbr | usedInstNbr := 0. instNbr := 0. emptyCollNbr := 0. aClass allSubInstancesDo: [ :anEntity | instNbr := instNbr + anEntity class allInstVarNames size. anEntity class allInstVarNames doWithIndex: [ :e :i | (anEntity instVarAt: i) ifNotNil: [ :content | ([ content isEmpty. emptyCollNbr := emptyCollNbr + 1. false ] on: MessageNotUnderstood do: [ false ]) ifFalse: [ usedInstNbr := usedInstNbr + 1 ] ] ] ]. ^ {('Used instance variables' -> usedInstNbr). ('empty collections' -> emptyCollNbr). ('Recoverable instance variables' -> (instNbr - (usedInstNbr - emptyCollNbr))). ('Total instance variables' -> instNbr)}
-- *Guillaume Larcheveque*
Moose-dev mailing list Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
Moose-dev mailing list Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
Wow!
Alexandre
On Mar 13, 2017, at 9:53 AM, Nicolas Anquetil Nicolas.Anquetil@inria.fr wrote:
Reviving this old mail from Guillaume.
I did an analysis on a huge model (>2M entities) we have:
- .mse file: 372.5 MB
- .image file without the model: 86.9 MB
- image with model: 692.8 MB (thus apparently 605,9 MB used by the model)
Then I did a small experiment: I removed the four instance variables of FAMIXSourcedEntity (sourceAnchor, sourceLanguage, comments, declaredSourceLanguage) and replaced them by variables in the privateState
for example:
FAMIXSourcedEntity>>comments "Returns the list of comments defined for the entity."
<MSEProperty: #comments type: #FAMIXComment opposite: #container> <multivalued> <derived> <MSEComment: 'list of comments defined in the entity'> ^commentsbecomes FAMIXSourcedEntity>>comments "Returns the list of comments defined for the entity."
<MSEProperty: #comments type: #FAMIXComment opposite: #container> <multivalued> <derived> <MSEComment: 'list of comments defined in the entity'> "^ comments" ^ self privateState cacheAt: #comments ifAbsentPut: nilAs a result, the .image file is 673.3 MB (thus apparently 586,4 MB used by the model) Not huge, but almost 20 MB saved, 3% of the initial memory
nicolas
On 10/11/2016 20:49, stepharo wrote:
Thanks
A task of Pavel will be to see how slots usage can reduce that space lost. For the empty collection having a kind of - sharing an #() in a classVar - creating a new collection on use could be a solution.
Le 7/11/16 à 12:16, Guillaume Larcheveque a écrit :
Hi Moose users,
as you may know, at Synectique we experiment lots of issues about memory usage. That's why i had a look at the number of unused instance variables in one of our system loaded with a big model from one of our main customer.
Total instance variables: 26.852.653 Used instance variables: 17.393.938 Empty collections: 5.023.508 Recoverable instance variables: 14.482.223
The recoverable instance variables are those with nil or an empty collection (or MultiValueLink)
As you can see, we can save a lot of memory :-)
Here is my (dirty) code to get that:
countUsedInstanceVariableInForSubInstances: aClass | usedInstNbr instNbr emptyCollNbr | usedInstNbr := 0. instNbr := 0. emptyCollNbr := 0. aClass allSubInstancesDo: [ :anEntity | instNbr := instNbr + anEntity class allInstVarNames size. anEntity class allInstVarNames doWithIndex: [ :e :i | (anEntity instVarAt: i) ifNotNil: [ :content | ([ content isEmpty. emptyCollNbr := emptyCollNbr + 1. false ] on: MessageNotUnderstood do: [ false ]) ifFalse: [ usedInstNbr := usedInstNbr + 1 ] ] ] ]. ^ {('Used instance variables' -> usedInstNbr). ('empty collections' -> emptyCollNbr). ('Recoverable instance variables' -> (instNbr - (usedInstNbr - emptyCollNbr))). ('Total instance variables' -> instNbr)}
-- Guillaume Larcheveque
Moose-dev mailing list
Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
Moose-dev mailing list
Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
-- Nicolas Anquetil -- MCF (HDR) Project-Team RMod
Moose-dev mailing list Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
Hi,
Nice! But here we are saving memory. What about the performances for access the variables? Because, when we try to minimize one, we are always changing the other.
Cheers, Vincent
-----Message d'origine----- De : Moose-dev [mailto:moose-dev-bounces@list.inf.unibe.ch] De la part de Alexandre Bergel Envoyé : lundi 13 mars 2017 23:08 À : Moose-related development Cc : Philippe Valenza; christophe.allegrini@synectique.eu Objet : [Moose-dev] Re: A quick analyzis of the usage of instance variables in Moose
Wow!
Alexandre
On Mar 13, 2017, at 9:53 AM, Nicolas Anquetil Nicolas.Anquetil@inria.fr
wrote:
Reviving this old mail from Guillaume.
I did an analysis on a huge model (>2M entities) we have:
- .mse file: 372.5 MB
- .image file without the model: 86.9 MB
- image with model: 692.8 MB (thus apparently 605,9 MB used by the
model) Then I did a small experiment: I removed the four instance variables of FAMIXSourcedEntity (sourceAnchor, sourceLanguage, comments, declaredSourceLanguage) and replaced them by variables in the privateState
for example:
FAMIXSourcedEntity>>comments "Returns the list of comments defined for the entity."
<MSEProperty: #comments type: #FAMIXComment opposite:#container> <multivalued> <derived>
<MSEComment: 'list of comments defined in the entity'> ^commentsbecomes FAMIXSourcedEntity>>comments "Returns the list of comments defined for the entity."
<MSEProperty: #comments type: #FAMIXComment opposite:#container> <multivalued> <derived>
<MSEComment: 'list of comments defined in the entity'> "^ comments" ^ self privateState cacheAt: #comments ifAbsentPut: nilAs a result, the .image file is 673.3 MB (thus apparently 586,4 MB used by the model) Not huge, but almost 20 MB saved, 3% of the initial memory
nicolas
On 10/11/2016 20:49, stepharo wrote:
Thanks
A task of Pavel will be to see how slots usage can reduce that space lost. For the empty collection having a kind of - sharing an #() in a classVar - creating a new collection on use could be a solution.
Le 7/11/16 à 12:16, Guillaume Larcheveque a écrit :
Hi Moose users,
as you may know, at Synectique we experiment lots of issues about
memory usage.
That's why i had a look at the number of unused instance variables in
one of our system loaded with a big model from one of our main customer.
Total instance variables: 26.852.653 Used instance variables: 17.393.938 Empty collections: 5.023.508 Recoverable instance variables: 14.482.223
The recoverable instance variables are those with nil or an empty collection (or MultiValueLink)
As you can see, we can save a lot of memory :-)
Here is my (dirty) code to get that:
countUsedInstanceVariableInForSubInstances: aClass | usedInstNbr instNbr emptyCollNbr | usedInstNbr := 0. instNbr := 0. emptyCollNbr := 0. aClass allSubInstancesDo: [ :anEntity | instNbr := instNbr + anEntity class allInstVarNames
size.
anEntity class allInstVarNames doWithIndex: [ :e :i | (anEntity instVarAt: i) ifNotNil: [ :content | ([ content isEmpty. emptyCollNbr :=emptyCollNbr + 1.
false ] on:MessageNotUnderstood
do: [ false ]) ifFalse: [usedInstNbr := usedInstNbr + 1 ] ] ] ].
^ {('Used instance variables' -> usedInstNbr). ('empty collections' -> emptyCollNbr). ('Recoverable instance variables' -> (instNbr - (usedInstNbr -
emptyCollNbr))).
('Total instance variables' -> instNbr)}
-- Guillaume Larcheveque
Moose-dev mailing list
Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
Moose-dev mailing list
Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
-- Nicolas Anquetil -- MCF (HDR) Project-Team RMod
Moose-dev mailing list Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
Moose-dev mailing list Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
!!!************************************************************************************* "Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis.
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.!!!"
1st, no idea about time performances. It would involve more sophisticated experiment that I cannot perform for now
2nd, I did a bit more experiments on the same line. Each experiment builds on the previous one, so the last one touched 6 classes - initial Famix MetaModel [692.8M] - FAMIXSourcedEntity: sourceAnchor, sourceLanguage, comments, declaredSourceLanguage [673.3M] - FAMIXAssociation: next previous [665.7M] - FAMIXNamedEntity: isStub/modifiers/parentPackage/annotationInstances/receivingInvocations [?] - FAMIXAccess: isWrite [?] - FAMIXInvocation: receiver/receiverSourceCode/signature [652.9M] - FAMIXContainerEntity: outgoingReferences/types/incomingReferences/functions [644.2M]
Substracting 87M to all numbers for the actual model size, we get 605.8M initially and 557.2M for the last one, which is about 50M save ( 8%)
Of course as we go deeper in the inheritance hierarchy, the memory save will be smaller, but I believe there is still room for improvement
nicolas
On 14/03/2017 11:02, Blondeau Vincent wrote:
Hi,
Nice! But here we are saving memory. What about the performances for access the variables? Because, when we try to minimize one, we are always changing the other.
Cheers, Vincent
-----Message d'origine----- De : Moose-dev [mailto:moose-dev-bounces@list.inf.unibe.ch] De la part de Alexandre Bergel Envoyé : lundi 13 mars 2017 23:08 À : Moose-related development Cc : Philippe Valenza; christophe.allegrini@synectique.eu Objet : [Moose-dev] Re: A quick analyzis of the usage of instance variables in Moose
Wow!
Alexandre
On Mar 13, 2017, at 9:53 AM, Nicolas Anquetil Nicolas.Anquetil@inria.fr
wrote:
Reviving this old mail from Guillaume.
I did an analysis on a huge model (>2M entities) we have:
- .mse file: 372.5 MB
- .image file without the model: 86.9 MB
- image with model: 692.8 MB (thus apparently 605,9 MB used by the
model) Then I did a small experiment: I removed the four instance variables of FAMIXSourcedEntity (sourceAnchor, sourceLanguage, comments, declaredSourceLanguage) and replaced them by variables in the privateState
for example:
FAMIXSourcedEntity>>comments "Returns the list of comments defined for the entity."
<MSEProperty: #comments type: #FAMIXComment opposite:#container> <multivalued> <derived>
<MSEComment: 'list of comments defined in the entity'> ^commentsbecomes FAMIXSourcedEntity>>comments "Returns the list of comments defined for the entity."
<MSEProperty: #comments type: #FAMIXComment opposite:#container> <multivalued> <derived>
<MSEComment: 'list of comments defined in the entity'> "^ comments" ^ self privateState cacheAt: #comments ifAbsentPut: nilAs a result, the .image file is 673.3 MB (thus apparently 586,4 MB used by the model) Not huge, but almost 20 MB saved, 3% of the initial memory
nicolas
On 10/11/2016 20:49, stepharo wrote:
Thanks
A task of Pavel will be to see how slots usage can reduce that space lost. For the empty collection having a kind of - sharing an #() in a classVar - creating a new collection on use could be a solution.
Le 7/11/16 à 12:16, Guillaume Larcheveque a écrit :
Hi Moose users,
as you may know, at Synectique we experiment lots of issues about
memory usage.
That's why i had a look at the number of unused instance variables in
one of our system loaded with a big model from one of our main customer.
Total instance variables: 26.852.653 Used instance variables: 17.393.938 Empty collections: 5.023.508 Recoverable instance variables: 14.482.223
The recoverable instance variables are those with nil or an empty collection (or MultiValueLink)
As you can see, we can save a lot of memory :-)
Here is my (dirty) code to get that:
countUsedInstanceVariableInForSubInstances: aClass | usedInstNbr instNbr emptyCollNbr | usedInstNbr := 0. instNbr := 0. emptyCollNbr := 0. aClass allSubInstancesDo: [ :anEntity | instNbr := instNbr + anEntity class allInstVarNames
size.
anEntity class allInstVarNames doWithIndex: [ :e :i | (anEntity instVarAt: i) ifNotNil: [ :content | ([ content isEmpty. emptyCollNbr :=emptyCollNbr + 1.
false ] on:MessageNotUnderstood
do: [ false ]) ifFalse: [usedInstNbr := usedInstNbr + 1 ] ] ] ].
^ {('Used instance variables' -> usedInstNbr). ('empty collections' -> emptyCollNbr). ('Recoverable instance variables' -> (instNbr - (usedInstNbr -
emptyCollNbr))).
('Total instance variables' -> instNbr)}
-- Guillaume Larcheveque
Moose-dev mailing list
Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
Moose-dev mailing list
Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
-- Nicolas Anquetil -- MCF (HDR) Project-Team RMod
Moose-dev mailing list Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
Moose-dev mailing list Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
!!!************************************************************************************* "Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis.
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.!!!" _______________________________________________ Moose-dev mailing list Moose-dev@list.inf.unibe.ch https://www.list.inf.unibe.ch/listinfo/moose-dev
-
Alexandre Bergel -
Blondeau Vincent -
Guillaume Larcheveque -
Nicolas Anquetil -
Stephan Eggermont -
stepharo