Hi How can I crawl a website content and save the contents to database? Any favorite library? Thanks, Ani

I usually use httpoison with floki . There has been some discussion about it [image] Web scraping tools Questions I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’t hold bac…

thank you for answering, I wondered how can I use meeseeks or html5ever, it seems like it built with other languages (rust) and how to spawn the process or crawl any link on the page? Thanks

You can have a pool ( poolboy , tutorial ) of scraping processes (may be genservers), each of which would accept a url to scrape (in handle_call), save the contents to a database, and return the urls that it has found (in {:reply, urls, state}). Then those urls would be fed back into the pool. I won…

[image] idi527: I wondered how can I use meeseeks or html5ever, it seems like it built with other languages (rust) I guess you would have to have a rust compiler installed. See GitHub - mischov/meeseeks: An Elixir library for parsing and extracting data from HTML and XML with CSS or XPath sel…

[image] suryaniwati: thank you for answering, I wondered how can I use meeseeks or html5ever, it seems like it built with other languages (rust) and how to spawn the process or crawl any link on the page? It has an elixir front-end. The dependency itself manages the rust part (all you need …

@suryaniwati : Let me summary everything: Similar topic We have already talked about tools for collect (scrape) data from web pages here: [image] Web scraping tools Questions I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hop…

[image] Eiji: Example script I really need to give asdf a try sometime… ^.^;

[image] OvermindDL1: I really need to give asdf a try sometime… ^.^; It’s not a “tool from god”, but it’s very easy to setup and it have very simple commands, so I think that’s best answer for all members (including beginners).

Plus it supports so many different languages. I was so tired of having separate version managers for each language, it’s super nice to have just one version manager.

Crawler Data

Questions & Help Questions

mischov February 7, 2018, 5:09pm 5

Couldn’t have said it better.

@suryaniwati

You might look at Crawler or Crawlie for examples of more complete crawling solutions.