On Jan 19, 2011, at 9:53 AM, Tudor Girba wrote: I was thinking that we could allocate some months of Cyrille in that task. And Mariano is expert on DB so we could ask him some feedback and advices Stef _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

What do you hope to achieve by having these models in a relational database?

Hi, I strongly recommend NOT to use a relational database. It is a very bad solution for OO (for several reasons, the best example I read ever is: "is like disassemble your car every night before sleep, and reassemble it every morning, before going to work"). I worked with relational databases a lot and I can ensure that it is a pain in the a**. Yes, with Mariano we did SqueakDBX, but that was just because in lots of jobs the database is not an option, you must use a relational db (and usually oracle), not because we think it is good for programming. Of course, gemstone can be a better solution... but if we want scalability for free, a nosql solution here can be a good choice (and there are some implementations in pharo, to choose one) btw...a document oriented database (like MongoDB, already implemented for Pharo) is a good approach for non-regular structures. my 2c. best, Esteban El 20/01/2011, a las 8:22a.m., Stéphane Ducasse escribió: _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

De: "Tudor Girba" <tudor.girba(a)gmail.com> À: "Moose-related development" <moose-dev(a)iam.unibe.ch> Envoyé: Jeudi 20 Janvier 2011 21:32:40 Objet: [Moose-dev] Re: what is the status of FameDB Hi Norbert, The goal of Moose is to help us analyze data. This means: modeling, mining, measuring, querying, visualizing, browsing etc. To do this, the prerequisite is being able to manipulate the data. Right now, we have all objects in memory. To be able to scale we need database support. So, all in all, it's not for a specific use case, but for any model that is described by Fame. Cheers, Doru On 20 Jan 2011, at 21:06, Norbert Hartl wrote: -- www.tudorgirba.com "Beauty is where we see it." _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

On 20 jan 2011, at 21:32, Tudor Girba wrote: Currently, the models have to fit into 32 bit address space. Modern machines support much more than that (data points: 16 GB @ 160 Euro for my current machine, standard workstations support 192GB). Do you have many models that wouldn't fit in 192 GB? The kinds of analysis Moose does are not supported efficiently by standard relational databases at all. They are optimized for a very different access scheme: selecting a very small subset of data and changing that. That means that they are only able to provide reasonable results for datasets that (nearly) fit into memory. In short: they allow you to avoid using a 64 bit Pharo image, and are able to use more cores. What you lose is having to copy data from and to the database and having to generate queries that don't fit the object model well. They are unlikely to provide better performance than a straightforward 64 bit Pharo image would, but can provide a short-term solution. Datawarehouse style databases (Vertical) and object oriented databases (Gemstone) are probably able to do better. Datawarehouse databases by pregenerating all kinds of cross sections and projections of the data, and oodbs by navigating instead of joining (and Gemstone by being able to use all memory). But even there, a lot of the Moose analysis seem to touch a large part of the model, and the interactivity needed means that disk based models will never become popular. Scaling of Moose is more likely to come from going 64 bit and distributing the model over multiple vms. Stephan _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

Thanks for this very sensible analysis. I would definitely give any database support for an image that can scale to use the entire promise of 64 bits and of multiple VMs. However, it looks like it will take a while until we will get that. In the meantime, it is more practical to use what exists. Even if relational databases are not at all my preferred option, we know that for Glorp there was an implementation in VW that served the purpose of storing the objects and enabling mining algorithms (even at the expense of high interaction). So, it would be great to salvage this effort and have a similar support in Pharo. In any case, I definitely would like to start an effort of looking into Gemstone and into other object-oriented databases. Anybody interested in joining the effort? Just a question: Would the new slate disks not alleviate the problem of the disk speed?

Cheers, Doru On 21 Jan 2011, at 12:15, Stephan Eggermont wrote: mining, measuring, querying, visualizing, browsing etc. To do this, the prerequisite is being able to manipulate the data. Right now, we have all objects in memory. To be able to scale we need database support. machines support much more than that (data points: 16 GB @ 160 Euro for my current machine, standard workstations support 192GB). Do you have many models that wouldn't fit in 192 GB? standard relational databases at all. They are optimized for a very different access scheme: selecting a very small subset of data and changing that. That means that they are only able to provide reasonable results for datasets that (nearly) fit into memory. In short: they allow you to avoid using a 64 bit Pharo image, and are able to use more cores. What you lose is having to copy data from and to the database and having to generate queries that don't fit the object model well. They are unlikely to provide better performance than a straightforward 64 bit Pharo image would, but can provide a short-term solution. (Gemstone) are probably able to do better. Datawarehouse databases by pregenerating all kinds of cross sections and projections of the data, and oodbs by navigating instead of joining (and Gemstone by being able to use all memory). But even there, a lot of the Moose analysis seem to touch a large part of the model, and the interactivity needed means that disk based models will never become popular. distributing the model over multiple vms. -- www.tudorgirba.com "We cannot reach the flow of things unless we let go." _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev

Just a question: Would the new slate disks not alleviate the problem of the disk speed?

On 24 jan 2011, at 23:47, Norbert Hartl wrote: Not when using an OO model and following the navigation. Objects that are created together are close together. A simple example: An order has orderlines. In an OODB you would expect the orderlines to be on the same (or next) page on disk. With a RDBMS, you need to access two tables, and have two seeks. A full table scan over orders though is of course slower with an OODB, as more pages need to be loaded.

Stef, are there any specific requirements on the db? Is this for long time storage of a lot of things

or just to extend the main memory for a definite amount of time? What will be the main actions to do? Searching for symbols/methods/class names or traversing stuff?

In case you want to add over time a lot of code and want to have general search capabilities that can be stored in a central location it might be a good idea to ask gemstone to sponsor a full license and install this somewhere in inria.

Or do I missing the point? Norbert _______________________________________________ Moose-dev mailing list Moose-dev(a)iam.unibe.ch https://www.iam.unibe.ch/mailman/listinfo/moose-dev