Idea for TWX architecture

[SaranKarthick]

Hi Developers,

I'm working on a TWX architecture for a use case as shown below.

There are manufacturing plants across different geographical locations. In each plant there are multiple machines and each machine has its own FTP server, where after 10000 recordings are collected from the machine, a csv file with these recordings are generated in the local FTP server. There is only read access to these FTP servers and no static ip is provided.

The objective is to provide a cloud based architecture for data visualization of the csv file data. 

I need help on the following to get me started on the initial architecture:

1. Since static ip can't be given for individual FTP servers, what is the best possible architecture for the use case.
2. With one TWX cloud instance will it be possible to collect data from the FTP servers across different locations or do we need a separate TWX instance for each location.

Any help will be much appreciated.

Regards,

Saran 

[PaiChung]

Have you thought about putting the Thingworx Agent (Edge Micro Server) on the Local system and then use that to do a file transfer to get the csv files up to the cloud.

[Constantine]

Hello Saran,

This looks like a decent use case for EDGE MicroServer. The idea would be to install it on each machine and let it send CSV contents to the master ThingWorx instance. By reversing data flow this way you can avoid thinking about firewalls and dynamic IP addresses. The EMS is cross-platform and fairly lightweight, so it shouldn't be a major issue to pass it through IT approval.

Regards,
Constantine

[SaranKarthick]

Thanks @Constantine  and @PaiChung  for your valuable suggestions.

I haven't got much exposure on Thingworx Agent (EMS) so based on your suggestion I have come up with the below architecture.

The Thingworx agent will be installed and running in the local system which will collect the csv files (pushed from the FTP servers to their respective local systems) and push it to a plant specific folders existing in the system repository of the cloud TWX instance.

Once the csv files are available in the repository folders we can write subscription service to parse the csv files and update the data storage component (e.g. streams or value streams or external DB). Then we can create mashups for Data visualization as well.

Hope I'm in the right track w.r.t the architecture.

One more clarification i need is if i decide to add more machines/plants to the above architecture i need to add the TWX agent correspondingly or how will the architecture scale up.

Also it would be helpful if you could share any reference articles/tutorials related to file transfer activity achieved by EMS (The EDGE Microserver article was very broad and it will be helpful if there is any dedicated tutorials/exercises available to do a quick PoC) 

Thanks and Regards,

Saran Karthick K R 

[raluca_edu]

Hi,

The architecture is on the right track from my point of view. You can add agents/servers later in case you need to scale up.

I am recommending you to check this course: https://www.ptcu.com/enrollment/student/iot-connectivity-with-thingworx

In this course, you will learn basic of LUA script that is running on Edge agent  and as far as I remember there is also a section where it is showing you how to read/save files.

Hope it helps,

Raluca Edu

[Constantine · Accepted solution]

Hello,

The architecture looks good, the only thing I can add is that if needed you can also put the EMS directly on the Machine (for example, in case if Machine doesn't support pushing files to remote FTP servers). As I mentioned, it's a very lightweight component and doesn't require its own dedicated server, so it is commonly found at the edge.

In addition to the course that @raluca_edu suggests, here's an article that may be a good starting point for the file transfers. Depending on certain criteria (amount of data, performance, security, etc.) you might want to parse your CSVs directly on the edge and instead of sending files to ThingWorx just execute remote services (EMS also allows doing it).

Regards,
Constantine

[SaranKarthick]

Hi Developers,

Appreciate your inputs and articles shared.

@Constantine ,

I understand your idea on deploying the EMS on the machine itself. But currently there is uncertainty w.r.t the access to the machines. So decided to go with the above architecture (push files to sys repo and parse) where i'll have much handle. 

In future based on the accessibility provided I'll certainly try to implement the way you suggested. I believe it shouldn't take much efforts if i move from the local system architecture to device level architecture.

Regards,

Saran