Web scraping is simple
I’d say this is a very generalized statement that should be cleared up for future readers bumping into your post. Not being disrespectful, but saying something like that without some real meat could get a new person in trouble.
But trust me - If you find yourself scraping 5-10 million jobs a day, it quickly becomes “not simple”. The premise of crawling/scraping is not complex, but I can assure you that sustaining it for 7-8 years on end and returning data in a timely fashion to paying customers is not easy at all.
Some more tips from my view, having done this for so long (without a single legal issue):
- Try to form a personal relationship with an IP provider - yes you can use the publicly available providers you mention above and there are tons, but none of those will scale to the numbers I needed to hit in any reasonably economic way. Easier said than done, but ask around - exhaust friends in SEO and marketing fields.
- Start slow. If you only have like 100 IPs to work with, don’t touch a target more than 1 time per hour with the same IP to start unless you plan to treat it like a “smash and grab”.
- Link your IPs to user agents somehow. Meaning if you pull IP #1 and go off to hit a site and randomly grab a UA string to roll with it - make sure the next time you show that IP - you come with the same UA string.
- Have some controls for measuring data quality over time. This may need to be manual in your case. You’d be amazed at how many big sites now start throwing you trash data that looks correct at a glance.
- When you think you have a crawl script ironed out - be sure you toss it at a tool like the EFF Panopticlick (and others) to look for obvious fingerprinting you may have not plugged. https://panopticlick.eff.org/
- Make sure, if using a headless browser you’re plugging all the massive, truck-sized holes they all expose… Things like mocking the
navigator.platform(and it better match the UA string) are major gaps I see all the time.
