I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’t hold back. Thanks.

[image] adammokan: This reminds me of the fun I had with geolocation/gps coords - again, depending on what you are going for. Most common case for that effort was a well-known map site. Some hints for geolocation - and this may be outdated, but it was always important to mock both navigator.geol…

Hey people. Just wanted to announce that I have made an article on Erlang Solutions blog, about using Crawly: Erlang Solutions Also I have made a couple of sample/tutorial projects to get started with: GitHub - oltarasenko/crawly-spider-example · GitHub GitHub - oltarasenko/products-advisor · GitH…

@oltarasenko Couple bits of feedback that are common mistakes I see people making when using Floki. Recommend a safer HTML parser than Floki+mochiweb_html. I know it’s nice to not need to start an Elixir article with “and then install Rust”, but web scraping is exactly the situation you do want …

Here is a version of your crawler that uses Meeseeks and fixes all of the above problems. defmodule Esl do @behaviour Crawly.Spider import Meeseeks.CSS @impl Crawly.Spider def base_url() do "https://www.erlang-solutions.com" end @impl Crawly.Spider def init() do [ sta…

[image] mischov: ot need to start a Hey @mischov ! Thanks for the feedback. Actually I was aware that parsing would happen 3 times, but decided not to do anything with it for now. At this point, I think that it’s better to update examples, so they are not misleading! Thanks for the hint. I d…

[image] mischov: Meeseeks Ok. Looks like the Meeseeks also have Xpath support. I will check that.

[image] oltarasenko: I don’t have problems with Rust. Parsing should be done by low level and fast languages! Basically, scrapy uses C parses in order to parse pages under the hood. We should do the same. I want to be clear that just because it’s Rust doesn’t mean it’ll be faster than the moc…

@oltarasenko Took me a little longer than planned, but put together the promised benchmark, which compares the version of the HTML parsing and extraction code from your post against a version using the optimizations I suggested and a Meeseeks translation of the optimized version. Name …

Hey @mischov . Thanks for sharing those. In the meantime I have tried to use the Meeseeks for Xpath related extractors. Looks quite great. Thanks for the work you’re doing!

Holy cow, only 420kb memory usage. Quite amazing.

Elixir Programming Language Forum

Web scraping tools

Questions & Help Questions

oltarasenko September 24, 2019, 11:48am 34

Ok. Looks like the Meeseeks also have Xpath support. I will check that.