You can have a pool (poolboy, tutorial) of scraping processes (may be genservers), each of which would accept a url to scrape (in handle_call), save the contents to a database, and return the urls that it has found (in {:reply, urls, state}). Then those urls would be fed back into the pool.
I wondered how can I use meeseeks or html5ever, it seems like it built with other languages (rust)
I guess you would have to have a rust compiler installed. See GitHub - mischov/meeseeks: An Elixir library for parsing and extracting data from HTML and XML with CSS or XPath selectors. · GitHub
