I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’t hold back. Thanks.

[image] adammokan: This reminds me of the fun I had with geolocation/gps coords - again, depending on what you are going for. Most common case for that effort was a well-known map site. Some hints for geolocation - and this may be outdated, but it was always important to mock both navigator.geol…

Hey people. Just wanted to announce that I have made an article on Erlang Solutions blog, about using Crawly: Erlang Solutions Also I have made a couple of sample/tutorial projects to get started with: GitHub - oltarasenko/crawly-spider-example · GitHub GitHub - oltarasenko/products-advisor · GitH…

@oltarasenko Couple bits of feedback that are common mistakes I see people making when using Floki. Recommend a safer HTML parser than Floki+mochiweb_html. I know it’s nice to not need to start an Elixir article with “and then install Rust”, but web scraping is exactly the situation you do want …

Here is a version of your crawler that uses Meeseeks and fixes all of the above problems. defmodule Esl do @behaviour Crawly.Spider import Meeseeks.CSS @impl Crawly.Spider def base_url() do "https://www.erlang-solutions.com" end @impl Crawly.Spider def init() do [ sta…

[image] mischov: ot need to start a Hey @mischov ! Thanks for the feedback. Actually I was aware that parsing would happen 3 times, but decided not to do anything with it for now. At this point, I think that it’s better to update examples, so they are not misleading! Thanks for the hint. I d…

[image] mischov: Meeseeks Ok. Looks like the Meeseeks also have Xpath support. I will check that.

[image] oltarasenko: I don’t have problems with Rust. Parsing should be done by low level and fast languages! Basically, scrapy uses C parses in order to parse pages under the hood. We should do the same. I want to be clear that just because it’s Rust doesn’t mean it’ll be faster than the moc…

@oltarasenko Took me a little longer than planned, but put together the promised benchmark, which compares the version of the HTML parsing and extraction code from your post against a version using the optimizations I suggested and a Meeseeks translation of the optimized version. Name …

Hey @mischov . Thanks for sharing those. In the meantime I have tried to use the Meeseeks for Xpath related extractors. Looks quite great. Thanks for the work you’re doing!

Elixir Programming Language Forum

Web scraping tools

Questions & Help Questions

dimitarvp October 1, 2019, 8:31am 38

Holy cow, only 420kb memory usage. Quite amazing.