Price Monitoring Tool The Business Need A leading Australian marketplace organization wished to monitor…
Mobius developed a complete automated pipelines setup in Azure cloud to process historical data as one time and then to do the delta processing on daily basis using Python, PySpark, Azure data factory, Azure Data Lake, Azure Synapse, Azure SQL and Power BI. Data will be pulled from ERP, CRM, Operational and Third party server. Data will be extracted from different listed sources using APIs / flat files from ERP/CRM system SharePoint services. Extracted output will be converted into flat files and placed in the Azure blob. We build Azure data factory pipelines to load the flat files from the blob to Azure SQL database. Using store procedures in Azure SQL, delta processing, data validation will be completed.
The unstructured/semi structured data (JSONs) from APIs will be stored in Azure data lake which then can be consumed by Azure Synapse Analytics as and when required. Flat files data will be loaded to the Azure SQL data warehouse from there data marts will be developed for the reporting needs. BI reports were developed using Power BI on different dimensions of Sales order, Inventory and forecasting information across business verticals providing very helpful business insights.
Data ware house output also been shared with AI/ML bots as input from which further analytics carried out on client end. Based on the internal and quality audit review comments, we will add the automated rules for ensuring data quality before loading into data ware house system.
The solution provided a single robust system which can be highly scalable as required in the future. Also single point for getting all the data sets for BI reports and AI/ML processing data sets achieved and thereby improving the operational efficiency in the client end. Increased data quality activities before ingestion into data ware house system made the data readily consumed for analytics purpose.
Using scalable services in Azure infra and go with pay as you use
Azure SQL can be scaled up/down as required which helps the cost management
Data storage in blob can be archived and can be stored for any number of years as per client requirement. No separate archive mechanism needed in this case
Highly scalable model developed
Improved operational efficiency and ready to analytics data delivered.