BIM Documentation

Data export and preprocessing

Data, used by widgets that display data on dashboards is prepared for statistical calculations in two steps.

First, data is exported from Magento Database into csv files. This ensures that database will not get extra overload if someone is performing extraction of data for data analysis. SQL databases such as MySQL are good optimized for short transactions, but not for full table extractions and aggregations. Data export could be performed once per day at the moment, when web store does not have high traffic.

Second, exported data is preprocessed. During preprocessing, data is extracted from csv files, optimized for calculations and stored in binary formats and in SQLite database.

For managing and controlling data actualization, back-end user can use menu Reports → BIM → Data Actualization in order to open Data Actualization page.

Data Actualization page displays user data export profiles and data preprocessing profiles. User can see indication if data was exported, last export date, size of exported file. Also for preprocessing, user can see indication if data preprocessing was performed and date and time of last data preprocessing.

From Actualization page, user is able to run data export and/or data preprocessing. Data export should be run before data preprocessing. As result of data export and data preprocessing, all export and preprocessing profiles should be marked as successfully processed.

During debug mode user will be getting notifications on dashboards page if some R packages needs to be installed/updated. Also, errors and warnings from R scripts will be presented to the user in most of cases.