Greetings, everyone!
When learning elixir, I faced up with the next task.
I have mongodb with stored entities that have id, created_at and type fields, count of records around ~ 500_000.
So, I need to create an elixir app that fetch this records in batches and pass them (one by one) by REST API interface to other application. Moreover, I need to trackdown progress of entity migration with possible restarts/retries. How I should design this app in OTP terms?
Looks like I need a supervisor with single genserver, that pool data from mongodb in smaller batches (about 100 or more), after that, genserver pushes records via REST API (using Task module), until every record successfully sent and after that proceed next batch, right?
And if I need restarts, when application is stopped or something other happened, should I save every step of batch processing to local db, like ID range of entities and state of batch process, for example progressing / failed / succeded?
And how add to genserver functionality to stop processing? Save genserver status to its state and change status from client code?
If processed entity is duplicated, for REST API it’s OK situation. How should I design this app? Please, share your opinion! Thanks for answers, in advice! ![]()






















