Web scraping is simple, IMHO there is two type of site :
- Server-Side Rendered page
easy to scrape with curl or any HTTP Client (+ HTML Parser) - Javascript Rendered page
A. You can use Headless / Browser Automation, but, it will be slow (+ HTML Parser)
B. Do “little bit” Reverse Engineering on their Web API (FASTER)
Important Point :
- Make sure your scrapper support Proxy Usage
- If your site target has anti-scraper / crawler / bot (Like your bot follow pagination, 1->2->3 and so on) and it block your IP, you can use IP Rotation Service like geosurf.com and luminati.io
- In some Country / Site, Web Scraping are prohibited
