Thank you, the CSV.parse_stream worked!
Final code:
alias NimbleCSV.RFC4180, as: CSV
ExAws.S3.download_file("testfiles", "file_upload.csv", :memory)
|> ExAws.stream!()
|> CSV.to_line_stream()
|> CSV.parse_stream ()
|> Stream.chunk_every(10_000)
|> Stream.map(& process_stream(&1)
|> Enum.to_list()
