I’ve been using Scrapy for a while and it is a nice and mature framework. I am writing Xpath selectors for getting data from htmls and using the built in Scrapy system for following links and building pipelines. There are some things that can be improved in Scrapy though. You need to write your own spider management process, when you have many spiders in a projects this gets complex. Also, today the same url that is crawled can contain frequently updated data. Scrapy is designed to through a bunch of URLs and store the data somewhere, but today the web works differently, I need to know if a field on a web site is updated. Seems like things like these can be managed very well in Elixir. Using xpath selectors (or the mentioned Rust implementation) is easy. Some html diff logic on top of that in order to listen for updates would be nice to have instead of how Scrapy functions.
