Getting duplicate data from various devices

[Devang_Singh]

Is there a way to change settings/configuration in Kepware which can allow us to remove duplicate data coming from devices? Its being fed into ADX and there we are getting lot of duplicate rows with same timestamp, telemetry value etc.

[AKshara]

Hi there,

We appreciate your query.

Regarding Kewpware, logging duplicate data coming from devices may be due to duplicate entries, which can be expected based on trigger conditions, however, there is no way to guarantee zero duplicate records.

There may be other reasons as well, like:

• Datalogger is using different caches for log items and monitor item, which are not in sync.
• When the monitor item gets a new value, it triggers the logging, but sometimes the log group cache still holds the old value for that item because it has not been updated yet, which leads to logging the old value that has already been logged at last cycle => logging duplicate and skipping the actual value

I hope this clarifies your question. Nevertheless, we welcome your feedback on feature enhancements and encourage you to create a case through MyKepare.

-Akshara

[Devang_Singh]

Hi Akshara,

Thanks for your reply. From the screenshot below, it looks like we don't have Datalogger configured yet. Do we need to this first? Secondly if its not configured yet, does that mean the duplicate issue is coming from somewhere else?

[AKshara]

Thank you for adding details; however, it is hard for me to imagine where you have been seeing the duplicate entries as there is no DataLogger configured.

Is there a third-party logger installed? 

Certainly, I have developed an interest in analyzing the situation in depth, and it would be highly appreciated if you would log a case using My.Kepware for further analysis.

Don't forget to attach a copy of the project. A basic application report will also add value. Please follow the steps mentioned in the manual to generate a basic application report.

-Akshara

[Devang_Singh]

Yes. Actually we have an ADX integration which let us know about the data coming in. Running the query on it is how we got to know that there is multiple instances of duplicate data running into millions. Before we had 'every scan' selected and started receiving data every sec for every item across all our devices. We started receiving too much data then since ADX was not equipped to handle this much data. We changed this now to publish only when change occurs but there is still duplicate data coming in (with same telemetry value against same timestamp) thats why wanted to know if this is something that can be avoided.

 

[AKshara · Accepted solution]

Hi Devang,

I would suggest going through the manual about Format on page 22.

If that doesn't answer your question, then I believe you need to upgrade the Kepware to its latest version, which is 6.15.154.

Please follow the steps mentioned in the article to upgrade Kepware.

-Akshara