Certain key components developed formed a comprehensive and integrated solution to meet our client’s needs.
● Rationalized Data Scraping: We implemented a centralized data aggregation framework that integrates data from multiple sources which are
the input websites obtained from the client. Site analysis is further processed on these sites to check for feasibility.
● Automated Crawler Development: Developed scripts in perl, python and selenium based approach to pull data from multitude of websites which handles 60+ attributes in different categories each. An end-to-end fully automated workflow has been enabled. This handled millions of data on a
daily basis by means of normalized database structure.
● Efficient Data Automation: Enabled automatic notifications and reporting to ensure that the crawl and following procedures are proceeding as intended. To solve the coverage and fill rate difficulties, we implemented automatic validations. We have also automated the creation of JSON manifest files that will provide output file metadata such as categories, file name, size, md5 value, and S3 location for each monthly delivery.
● Diligent Quality Checks and Quality Assessments: To ensure data correctness and dependability, comprehensive quality checks and validation processes were implemented. Full output validation was done before the output was provided to the client’s S3 location. Our accuracy and coverage percentage was above 95%.