hello, i have the following problem:
I need to make http requests for a given site with thousands of pages. just stop the requests when the response returns {_page,}
I was able to make the code without using concurrency, and Tesla for requests
def fetch(page) do
{:ok, response} = get(“/api/res”, query: [page: page])
{page, response.body[“res”]}
end
def fetchAll(curr \\ 1 , list \\ ) do
result = fetch(curr)
case result do
{_page, } →
list
{curr, nil} →
fetchAll(curr,list)
{_page, res} →
curr = curr + 1
fetchAll(curr,list++res)
end
end
however I wanted to use Task to increase speed, but i don’t know how to do it.
if the number of pages was known, it would be easy to implement it using Task.async_stream
1..numberPages
|> Task.async_stream(fn page ->fetch(page) end, ordered: false,
max_concurrency: System.schedulers_online()*3)
|> Enum.reduce(, fn {:ok, {_page, res}}, j → res++ j end)
but as the number of pages depends on the return of the request, I don’t know how to do this
