# Processing json files **URL:** \ **Category:** Questions\ **Tags:** development\ **Created:** [October 20, 2019, 6:50am UTC](https://forum.elixirforum.com/t/processing-json-files/26210 "2019-10-20T06:50:56Z")\ **Posts on this page:** 1\ **Showing post:** 2
**Author:** ![Eiji](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/eiji/32/36743_2.png) [@Eiji](https://forum.elixirforum.com/u/Eiji)\ **Post date:** [October 20, 2019, 9:44am UTC](https://forum.elixirforum.com/t/processing-json-files/26210/2 "2019-10-20T09:44:23Z")
@kodepett I have working in a bit similar case. I was doing a migration from old `JSON` BigData (thousands of small files) to `PostgreSQL` database. Therefore I did not need to make it fastest ever, but of course it should not run for hours. For that case simply `Flow` was enough. From what you said you should have about 1000 files per day which is not that big number. If you will make parsing each file enough fast then number of files should not be a problem for you. > [@kodepett](#): > > What will be the most efficient solution for the above. (I’ve heard of FLOW/BROADWAY/GENSTAGE - which one will be ideal) I’m sure about only one thing: there is nothing ideal 😃. It may depend on your logic. Look that simply putting `JSON` to `jsonb` column without even parsing `JSON` to `Elixir` structs is way different comparing to a big process of parsing and processing data. If working with single file will be really short then maybe you do not even need to think about putting an extra dependency you don’t know about just for this case. However typically the whole process is longer than just reading file and it may be worth to think about splitting job into few stages. For this case I would recommend `Broadway`. > You may also be interested in [Flow by Plataformatec](https://github.com/plataformatec/flow). Both Broadway and Flow are built on top of GenStage. Flow is a more general abstraction than Broadway that focuses on data as a whole, providing features like aggregation, joins, windows, etc. Broadway focuses on events and on operational features, such as metrics, automatic acknowledgements, failure handling, and so on. > Source: [GitHub - elixir-broadway/broadway: Concurrent and multi-stage data ingestion and data processing with Elixir · GitHub](https://github.com/plataformatec/broadway#comparison-to-flow) > [@kodepett](#): > > Is there any filesystem monitoring api that can monitor the filesystem and notify my application once a new file is pushed by the upstream server. Yes, it is. Even `phoenix_live_reload` is using it, so it may be worth to check it’s source. The library is called `file_system`. For this please make sure that your `backend` is prepared as in some cases it’s a **must have**. > [@kodepett](#): > > What are some of the gotchas I should be aware of, especially if I want to persist the file content to a database(Postgresql) There are lots of gotchas. From basic `Elixir` up to specific to your use case. First of all for a well known gotchas in `Elixir` you may read this forum topic: > [@Elixir's biggest gotchas?](https://forum.elixirforum.com/t/elixirs-biggest-gotchas/796): > > Being new to Elixir would be very valuable to learn what you guys consider to be the biggest gotchas in Elixir land? When working in file you are doing few operations (especially with a big number of files) then it’s worth to use different `File.open` or sometimes also `File.stream`. Therefore in some cases `Jaxon` may be interesting for you: [https://moboudra.com/intro-to-jaxon-json-parser-for-elixir/](https://moboudra.com/intro-to-jaxon-json-parser-for-elixir/) Finally you should be aware of typical overusing some features of `Elixir` like `GenServer` as there are not good for every use case: [https://learn-elixir.dev/dangers-of-genservers](https://learn-elixir.dev/dangers-of-genservers) --- _[View the full topic](https://forum.elixirforum.com/t/processing-json-files/26210)._