Influx DB Delay Logging Property Value

[JK_10744682]

Hi all,

I have a subscription that occurs on a property data change. The subscription queries the most recent property history data. Sometimes the most recent value isn't logged yet so when I query in the data change subscription the most recent value does not show up. Any way around this slight delay so I can query the most recent data without missing the most recent value?

[TonyZhang]

Hi @JK_10744682,

What is the purpose of this subscription? Are you verifying the new property value is indeed logged into the value stream through this subscription by comparing the current value with the logged record?

I doubt if this is necessary but I guess you can write the timestamp and new value into a separate storage via the data change subscription and compare the data between the one you manually instructed to write in the subscription and the one automatically recorded in Value Stream at a later point.

Also, I found this article: https://www.ptc.com/en/support/article/CS371327, not sure if it's relevant to your case.

Best,

[JK_10744682]

The use case we have is about every 40 minutes we have a data change and want to take the mean of the last 10 values to see if it is outside of the threshold, and if it is then we want to write a calculated value to a PLC. I have tested adding a pause() of a few seconds and it seems to be working to query the new data change, would this be an appropriate approach to you?

[Rocko]

This seems to be an easy fix, however keep in mind you block an event-processing thread longer than needed, so might give you adverse effects in high-load scenarios. Vlad's approach would work well, but I think the easiest fix would be you query the 10 last values, and if you notice the current value - which is available in the event data - is missing from the query, just add it manually and then do your computation. So basically you add one if statement.

[VladimirRosu]

Hi @JK_10744682 ,

We need a bit more details on the use-case you're trying to achieve.

For example, Influx DB and any such software systems will always incur a delay from when you send the data to them until when you can query it back. It's the nature of the system and the batching process that causes this.

In your situation however, we have a far better mechanism for this and this is the capability to store data in-memory between different executions of a subscription. If you access https://support.ptc.com/help/thingworx/platform/r9.7/en/index.html#page/ThingWorx/Help/Composer/Things/ThingSubscriptions/ThingSubscriptions.html please read the chapter at the end about the thisSub.JSONState. You can add on DataChange a property value to this object, which will be accessible to other executions of the same subscription in the future. You could store for example the last 10 values of that property in memory without any latency penalty - of course, memory consumption being the thing you should manage. To make sure this is not resulting in out of memory situations, we enforce memory size limits for thisSub.JSONState of a single subscription and the overall memory storage of all subscription states (in platform-settings.json).

[JK_10744682]

The use case we have is about every 40 minutes we have a data change and want to take the mean of the last 10 values to see if it is outside of the threshold, and if it is then we want to write a calculated value to a PLC. The issue with the thisSub.JSONState is that if the system were to go down or the machine is saved or something of that nature that the data will be erased from memory. If the system crashes the querying needs to continue. I have tested adding a pause() of a few seconds and it seems to be working to query the new data change, would this be an appropriate approach?

[VladimirRosu · Accepted solution]

Hi @JK_10744682 ,

Typically, in a subscription, when you use the thisSub.JSONState object, the very first thing you should do is to verify its state.

For example, you should verify if it's empty (or any other condition you would want that would be indicative of the system just being restarted) and in that if clause you should "refill" it with the relevant data (last 10 values). This means that ThingWorx will only query the external data store in these situations. Then, on subsequent executions of the subscription, that condition will not be triggered and you'll use only the in-memory values, therefore greatly reducing the external system impact.

Considering this, I would not recommend using pause (even if it works). If you really want to use this approach I would advise to test the impact of adding pause on the system (ideally via a load test). Reason for saying that is that if the pause is too long and you have too many DataChange events (eg: I don't know how many devices you have), that pause could pile up and start generating increasing numbers of events in the event subsystem, which is something you should definitely avoid.