Web Scraping is the process of extracting information from online sources. The methodology falls under the cutting-edge Robotic Process Automation (RPA) technologies that help organizations to automate data collection workflows with increased productivity. As such, many organizations use this methodology to collect data relevant to their focus areas in their own data sources, websites, or in public repositories, and often considered as an alternative approach for manual fact-checking processes for updates.

Problem

Development of an automated framework to cross-validate given data sources against the latest updates available on the internet or given target directory of intranet source. The framework should present certain intelligence to be selective in certain data fields, based on their category, availability, or completeness, and to be adaptive during the process and run in a self-sustainable loop. The proposed intelligent agent should be able to carry out the following 3 main functions:

• Cross-Validation of given data.
• Use the data over multiple parameters of search parameters to get the most accurate update or validation against the data.
• If the correct updates are found to store the updated data in a new data table followed by the original data formating/structure while notifying the newly updated fields of data.

Solution

Our Data Science team has developed an intelligent agent (bot) that can navigate various public online search sources and run through multiple phases of search parameters to get the most accurate match for cross-validation. The bot can determine the given data source as its own data and use those data in its search parameters in an orderly fashion, making it looks like a secondary copy of the previously provided primary data source which the bot used to work on. Our team has given additional functionalities to the bot to run in different states based on the connectivity or blockers that may arise during the process.