23 Feb
2012
23 Feb
'12
3:33 p.m.
Hi,
Magritte 3 is now available for Gemstone - with tests green. The
configuration has been updated and I loaded Magritte 3 with:
Gofer it
squeaksource: 'MetacelloRepository';
package: 'ConfigurationOfMagritte3';
load.
MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
ConfigurationOfMagritte3 project stableVersion load: #('Seaside Tests') ].
However the tests run extremely slowly - in the test suite I've measured
20x slowdown between Magritte 2 and Magritte 3 - making the tests a pain to
execute.
I've run a series of tests to characterise where the problem might be. The
tests are as follows:
-----
1) Test-suite benchmark:
| testSuite |
testSuite := (OBClassCategoryNode on: 'Magritte-Tests-Model-Description')
classes
inject: TestSuite new
into: [:suite :classNode |
(classNode theClass inheritsFrom: TestCase)
ifTrue: [suite addTests: classNode theClass suite tests].
suite].
Time millisecondsToRun: [ testSuite run ].
2) Description benchmark:
Magritte 2:
Time millisecondsToRun:
[100 timesRepeat: [ MABooleanDescription description ] ].
Magritte 3:
Time millisecondsToRun:
[100 timesRepeat: [ MABooleanDescription new magritteDescription ] ].
3) Micro-benchmark
Magritte 2:
Time millisecondsToRun:
[100 timesRepeat: [ MABooleanDescription class allSelectors select: [
:each | each isDescriptionSelector ] ] ].
Magritte 3:
Time millisecondsToRun:
[100 timesRepeat: [ Pragma allNamed: MAPragmaBuilder
magritteDescriptionPragma from: MABooleanDescription to: Object ] ].
* The micro-benchmark is illustrative of the difference between Magritte 2
and Magritte 3 - Magritte 2 uses a naming convention to retrieve
descriptions, whereas Magritte 3 uses Pragmas.
-----
Results:
Pharo Magritte 2: (1) 436 (2) 0 (3) 85
Pharo Magritte 3: (1) 743 (2) 58 (3) 12
Gemstone Magritte 2: (1) 988 (2) 30 (3) 2229
Gemstone Magritte 3: (1) 21663 (2) 1575 (3) 444
Cavet: Although Gemstone and Pharo are running on the same machine -
Gemstone is running virtualised in a VMWare Linux installation, so the
results are not directly comparable, although the 30x performance decrease
between Pharo and Gemstone surprised me.
First the good news:
For Magritte the pragma description look-up is more performant than the
named-based lookup (micro-benchmark (3)).
Now the bad news:
In Pharo Magritte 2 is faster then Magritte 3 when running a description
lookup based test-suite containing 1416 tests. My feeling is that in the
context of a real application performing a smaller number of lookups this
is probably not significant.
However in Gemstone the performance degradation is more noticeable and if
nothing else makes running the tests painful.
How to proceed:
Magritte 2's performance advantage is gained from caching the descriptions.
However I think the benefits of having dynamic (non-caching) descriptions
outweigh the disadvantages and I'd be reluctant to add a generic layer of
caching. In Pharo I don't think this is an issue. However in Gemstone I
feel that these numbers need some further investigation.
Any thoughts how to proceed?
Nick