Build a Real World Scrapper!

Build a Real World Scrapper!

Introduction

In our previous post we saw the building blocks of a web scrapper as well as a small example at the end. In almost all cases scraping one article is not enough. We have to be able to scrape large number of data, across different categories and sub categories. We will use the Reuters to scrape a large number of articles across the site.

Structure

The first thing to do is identify what you want and the structure of the website.