Using Elixir to Monitor Multiple Interface Process/Task

[Woody88]

Hey thanks a lot for your reply!

@OvermindDL1

Big first question is how can those other systems be monitored? Is it some polling that has to be done? Can they broadcast events to some existing infrastructure that you can hook in to? Regardless of how it is done it can be done on the EVM, just need to know how to be able to describe better.

Well for an example we have an interface(Boomi) that is in the middle of two platforms. First platform is where the end user creates for example the vendors, then sends an outbound to our interface which process the data and creates the info to another platform(for e.g with got NiFi but i don’t think that will be one of the systems) that we got. Within the interface/platforms I can make http requests. The idea would be that the interface would push some data to notify my EVM server/client to notify that a task that has been executed, and would that info that I receive would let me know what is the current condition or next step that the task will need to go through. Wither it has been created successfully on one of the platform or not.

@OvermindDL1

GenServer is fine if you want the system to perform any processing on the EVM side. Unsure what you mean by animation, are you wanting to display a webpage of the overall system monitoring? If so then that would best be done by ‘channels’ in Phoenix, designed just for that kind of purpose.

The thing is that we have multiple platforms which a data might need to go through, and I would like to build a (yes with some html/javascript) specific monitoring system. So what I was thinking is that my server/client would be receiving chunks of data just containing small information, and then update the webpage based on the incoming info(so same as a server event). So do I really need to keep a constant channel open? If I’m not losing anything then that’s fine.But, the idea is that I would have to define the step of each task that I want to monitor and hold it somewhere, so that when the data comes it knows what task its reflecting. So I could be monitoring different type of tasks for the same systems(so multiple channel?).Does that makes sense?
e.g:
Task 1 : create vendor(platform 1) → (interface) process data → create data to another platform(platform 2)
Task 2 : platform(2) prepare to sync data to platform(1) → interface receives data check if platform(1) ready for receive → platform(1) save new data
Same systems but different task that i could display on a web page.(So each type of task would have is own page. Maybe?)

@OvermindDL1

For user accounts and such you can use DETS or Mnesia fine (Mnesia over DETS though in 90% of cases), however for generic data storage especially of logging data that can potentially exceed 2gigs over time, definitely use a real database like PostgreSQL or something (and Ecto works natively with it so you’d have a very nice interface in to it).

Well i would really like to use Mnesia or DETS because it would help me understand them better and give me a reason to touch them . PostgreSQL I’m already familiar and used a lot in Rails. But because I need to define the structure of my task and maybe I would also like to have user accounts(which is not really needed now) I was thinking that I might definitely need a real database. The thing is that I dont want to have to manage a database later on if this project fits perfectly. haha.

@OvermindDL1

If you are just holding known state and not needing to store anything past restarts then a simple web interface with a fairly trivial no-state login would be quite easy to do, but again that all builds up based on how you get the information from what you are monitoring. And yes a Channel is one of the most efficient ways to do the client-updates of a webpage.

Well right now just to find out where something went wrong they gotta backtrace and they can only hold 2 weeks old info. So I was being nice when I said that I would hold like a month past of log info. If later on I feel that I’d rather use a database I will switch it to a database. But, if its clearly pushing it using DETS or Mnesia then ill just go with a database now.

@OvermindDL1 Thanks a lot for your detailed reply. Replying back to you took me a least 5 -7 mins to write. I appreciate you taking the time to reply with such detail!