Hi,
I’m working on an elixir application that processes json files(contain transactions) from an upstream server. The average file size is 10mb. The upstream server will generate the file(s) and push it to my server. I need to parse the json file and generate alerts where applicable based on threshold configuration. A lot of files will be generated by the upstream server and I’m expecting to process about 10GB in total everyday. I don’t have control over the rate at which the files are generated and pushed by the upstream server.
-
What will be the most efficient solution for the above. (I’ve heard of FLOW/BROADWAY/GENSTAGE - which one will be ideal)
-
Is there any filesystem monitoring api that can monitor the filesystem and notify my application once a new file is pushed by the upstream server.
-
What are some of the gotchas I should be aware of, especially if I want to persist the file content to a database(Postgresql)
Thanks for your continuous guidance.
