Hello,
We are working on a project where we aim to collect PhD data using Kepware and transfer it to Apache Kafka. Since Kepware doesn't have a direct integration with Apache Kafka, we designed an architecture where we send the data to an MQTT broker (Mosquitto) and then use Node-Red to forward it to Apache Kafka. However, we're unsure if this is the best approach.
We're currently in the testing phase, but as the number of tags increases, we’re encountering some issues with Node-Red. The data is not being transferred properly to Kafka, and we are getting some buffer errors in Node-Red. By the way, we are using Node-Red as a MQTT connector, and while there are many other connector services available, We preferred the open source one..
We are working with approximately 60,000 tags, and our goal is to send the data to Kafka in 1-minute intervals.
How can we successfully implement this project? Have there been any similar projects done before? If you need more information about the project, I can provide it.
We would appreciate your help.
Best
