[Moose-dev] Report about the D4D datathon challenge

Dear all, this is a small report about the RESILIENCE team using Pharo&Roassal for the D4D datathon challenge: http://www.d4d.orange.com/en/home The idea was to use mobile network data from Senegal provided by Orange company in order to solve some development problems. The amount of data provided by Orange was really huge, something like 66 Gb: we have the number and the durations of calls between each pairs of GSM antennas in Senegal (1666 different antennas) and also some information about the mobility of people. You can find more information about the data here: http://arxiv.org/abs/1407.4885 We try to extract some data from August 2013, because we know that some floods occurs at that time in Dakar. Our main objective was to to be able to visualise mobility and communications patterns with ROASSAL to see if there is any differences before and after the floods. We had a lot of problems to solve in a short period of time :-) At the beginning, we couldn't open huge files with Pharo. Only open a file that is around 3.4Go is impossible. Pharo says that the file does not exist. Thierry tells us that us that one must compile the VM with -D_FILE_OFFSET_BITS=64. (the Cog VMs are also built with -D_GNU_SOURCE). Why this is not done by default ? After that we decide to split the files in many smaller files and we try to use a mySQL database and various other tools but still have many troubles because of the size of the data ... At the end, we have done some visualisations but only on a tiny portion of the data on one day (one day is around 100Mb of data). You could find some visualisations done by our team here: https://twitter.com/sergestinckwich/status/586178094215606273 Managing huge amount in Pharo and also ROASSAL is still a challenge and we definitively need more tools in this area. We had some problems with ROASSAL: bugs with RTEdgesBuilder (Alvaro can you send a report to Alex about that ?), to much time to process some viz, ... At the end, our team receive the honorary price: https://twitter.com/sergestinckwich/status/585866625745887232 This is a brief description of our project. ========================================================= Project name: RESILIENCE Team members: Clémence Douard (ENSCI), Onil Goubier (CIRELA), Alvaro Peralta (LabU / Chile University), Aurélie Thouron (ENSCI), Serge Stinckwich (IRD / UMMISCO) The main concern of this project is to improve resilience after an environmental crisis in a southern city. During the preparedness phase before crisis, records of inhabitants habits (like patterns of mobility or communications) are done in order to characterize the situation before the crisis. Assessments of the improvements will then be done by providing suitable visualizations of the resilience dynamic. The goal is to help citizens to represent these improvements and to take advantage of them in their daily lives. We took as an example the city of Dakar during the flooding that occurred in August 2013. The data used are those of the dataset 1 (communication between antennas) and 2 (user mobility) provided Orange. We use this information, coupled to OSM, and other data related to floods of August in Dakar to construct maps and indicators to show resilience. Visualizations are made with agile visualization platform Roassal: http://agilevisualization.com/ ========================================================= I would like to thank all the team members and also the people who help us remotely: Alexandre Bergel (ObjectProfile/Chile University) and Thierry Goubier (CEA) I would like to thank also the SIMPLON team and ORANGE for organizing such a nice event. Regards, -- Serge Stinckwich UCBN & UMI UMMISCO 209 (IRD/UPMC) Every DSL ends up being Smalltalk http://www.doesnotunderstand.org/