Web scraping service

This investment fund was looking for a review tracking system to discover how companies from their portfolio are performing online. Our online reputation management solution is benefiting this client by tracking mentions from major review websites. The web scraper is searching for positive and negative reactions from customer reviews and delivering results on a weakly basis. In such a case, the fund can track the performance of the startup and discover any issues with it almost instantly; way before an official report.

The request came from a pricing intelligence consulting firm with a rather typical task of price tracking. The goal was to collect pricing information from several e-commerce websites and deliver data in an Excel format. Some of these websites had anti-scraping protection, so our team implemented a proxy rotation and captcha solving. One of the sites was an auction house with continually changing prices, so the best solution was to monitor price change on an hourly basis.

This web scraping service is another example of a review tracking system implementation. A law firm was working on a case against a supplier who sold bad quality tractors for the farm. The law firm thought about filing a сlass action lawsuit rather than working on a single case. Our solution was to track down negative reviews of this tractor supplier and identify other companies that have suffered from the quality of the product. As a result, the law firm received a report containing information about other companies in the EU had the same issues with this supplier.

The goal of an insurance startup from Germany was to create a map containing disaster information for the last 100 years all over the EU. Our task was to collect data from Wikipedia about all significant disasters such as train and airplane crashes, car accidents and others. The collected data also had addresses, casualties and other info. The insurance agency dataset required a GPS location, so our team created an address-to-GPS converter. The dataset was delivered in JSON format. The overall goal of this project was to do online and offline advertisement of insurance services near the locations of the disasters. Such advertisement now contains accurate information about an accident or a disaster that happened nearby. With this data marketing communication campaign now shows higher conversion and retention rates. Such a case is an example of multi-step complicated data collection, enrichment, cleansing and management. Data from Wikipedia comes out in unstructured format and requires intelligent internet scraping.

Researchers and students often have limited budgeting. Regardless of that, there are still ways to outsource some if not all of the web scraping. Service included collecting data from an interactive map that contained minerals, coal and other resources of Indonesia. There was no English version of the website, so our web scraper also had to translate data on the go.

For this project, we have collected horse and greyhound stats. In the data set, we have included race venue, runners, participant's statistics, date, time, distance, grade, prize, dog or horse name, trainer's name and lots of other data. The analysis included information about each runner's previous races and performance stats, including dates, position, odds, split, and race comments.