Upcoming Webinar : Leveraging Web Data For Advanced Analytics

On 6th Dec, 11.00 AM to 12.00 PM ( EST) 4.00 PM to 5.00 PM ( GMT )

Data warehouse using Azure Cloud Implementation

Data Warehouse using Azure Cloud Implementation

Data Analytics and BI Case Study

Client

A US Based Product Manufacturing Engineering Entity

The Business Need

Based out of US, a manufacturing engineering product entity which deals with selling of engineering materials for manufacturing sector had urgent requirement to build a data warehouse system to track the inventory, sales, order, forecasting details of the products across different categories. The workflow solution thus developed would enable the data analyst to get the daily track of order, sales, and forecasting details to get the insight out of that using BI reports.

Challenges

The major challenge was to fully automate large-scale sales, order and inventory tracking of the products across various industries. Delta processing should be done on daily basis, data validation, any post processing need to be done post that. We have to deliver very high quality of data from the different API / ERP enterprise sources and so building up the validation and data quality layer was very challenging in the automation process.

Contact us for a solutions demo:

Our Solution

TechMobius developed a complete automated pipelines setup in Azure cloud to process historical data at one time and then to do the delta processing daily using Python, PySpark, Azure data factory, AzureDataLake, AzureSynapse, AzureSQL, and PowerBI. Data will be pulled from ERP, CRM, Operational, and Third-party servers.

Data will be extracted from different listed sources using APIs/flat files from ERP/CRM system SharePoint services. The extracted output will be converted into flat files and placed in the Azure blob. We build Azure data factory pipelines to load the flat files from the blob to the Azure SQL database. Using store procedures in AzureSQL, deltaprocessing, and data validation will be completed.

The unstructured/semistructured data(JSONs) from APIs will be stored in Azure datalake which then can be consumed by Azure Synapse Analytics as and when required. Flatfiles data will be loaded to the AzureSQL data warehouse from the data marts and will be developed for reporting needs. BI reports were developed using Power BI on different dimensions of Sales order, Inventory, and forecasting information across business verticals providing very helpful business insights.

Data warehouse output has also been shared with AI/ML bots as input from which further analytics are carried out on the client end. Based on the internal and quality audit review comments, we will add the automated rules for ensuring data quality before loading into the data warehouse system.

Highlights

Result

The solution provided a single robust system that can be highly scalable as required in the future. Also, a single point of forgetting all the datasets for BI reports and AI/ML processing data sets was achieved thereby improving the operational efficiency on the client end. Increased data quality activities before ingestion into the data warehouse system made the data readily consumed for analytics.

The Business Need

Based out of US, a manufacturing engineering product entity which deals with selling of engineering materials for manufacturing sector had urgent requirement to build a data warehouse system to track the inventory, sales, order, forecasting details of the products across different categories. The workflow solution thus developed would enable the data analyst to get the daily track of order, sales, fore casting details and get the insight out of that using BI reports.

Challenges

The Mobius Solution

Top Challengers/ How Can We help/ Why us?

Mobius developed a complete automated pipelines setup in Azure cloud to process historical data as one time and then to do the delta processing on daily basis using Python, PySpark, Azure data factory, Azure Data Lake, Azure Synapse, Azure SQL and Power BI. Data will be pulled from ERP, CRM, Operational and Third party server. Data will be extracted from different listed sources using APIs / flat files from ERP/CRM system SharePoint services. Extracted output will be converted into flat files and placed in the Azure blob. We build Azure data factory pipelines to load the flat files from the blob to Azure SQL database. Using store procedures in Azure SQL, delta processing, data validation will be completed.

The unstructured/semi structured data (JSONs) from APIs will be stored in Azure data lake which then can be consumed by Azure Synapse Analytics as and when required. Flat files data will be loaded to the Azure SQL data warehouse from there data marts will be developed for the reporting needs. BI reports were developed using Power BI on different dimensions of Sales order, Inventory and forecasting information across business verticals providing very helpful business insights.

Data ware house output also been shared with AI/ML bots as input from which further analytics carried out on client end. Based on the internal and quality audit review comments, we will add the automated rules for ensuring data quality before loading into data ware house system.

Highlights

Results

The solution provided a single robust system which can be highly scalable as required in the future. Also single point for getting all the data sets for BI reports and AI/ML processing data sets achieved and thereby improving the operational efficiency in the client end. Increased data quality activities before ingestion into data ware house system made the data readily consumed for analytics purpose.

Using scalable services in Azure infra and go with pay as you use

Azure SQL can be scaled up/down as required which helps the cost management

Data storage in blob can be archived and can be stored for any number of years as per client requirement. No separate archive mechanism needed in this case

Highly scalable model developed

Improved operational efficiency and ready to analytics data delivered.