Throttling OfflineMsgStore's messages when WebSocket looses connection with UpdatePropertyValues svc

[rad1]

I read this article: How to insert past-data to a Thing?

I want to handle Offline messages in a specific way when TW agent's WebSocket looses connection to TW server for some amount of time. 

I have these requirements:

• Real-time data cannot be delayed by historical data.  Real-time data takes priority.
• Cached data cannot be sent all at once but must be throttled.

I have these settings that configure Message Queue size to 1GB.

config.ThrowOnOfflineMesssageSave = true;
config.OfflineMsgQueueSize = 1048576; //16384 default
config.OfflineMsgStoreDir = "C:\OfflineMsgStoreFolder";

Questions:

1.  When WebSocket connection gets established again and we have a huge number of messages in the Message queue (let's say full 1GB) and we also have the latest collection of updated properties in the memory, will all historical messages in the queue be propagated the TW server before the latest update is sent?

2. Is there a way to first sent the latest change and send historical data later?

3. Is there a way to control the speed of sending historical data so it doesn't affect outbound Internet traffic, i.e. can we throttle that tragic?

4.I am wondering if I can turn turn off offline message store? Maybe OfflineMsgQueueSize=0 would to the trick.

I would expect some settings to control that. Then I can do some custom solution to satisfy requirements?

5. If I am able to do custom offline message handling can I use VirtualThing.getPendingPropertyUpdates() method to obtain the list of PropetyValue changes, write it (serialize it to a local disk) and then clear/drain PropertyValueList  (by calling drainPendingPropertyUpdates() to release Memory usage)?

Then I could use manual push in a controlled way to sent historical data using UpdatePropertyValues service to re-create history (as explained in How to insert past-data to a Thing?)

However the number of PropertyValues can be different each during each scanning cycle on the agent so how can I handle various number of properties I have in each update?

I would like to reduce the number of properties I am sending to the server.

6. Any other recommendation on how to do custom offline message handling?

Can you interact with saved offline messages via TW SDK API and handle sending to TW server in custom way by preventing automatic sending, i.e. when WS connection gets established once again, can I intercept that event and ready messages from offline message store and send them slowly over time, but in the meantime I want to send the latest updates multiple times so I don't loose visibility on how Properties change at the current movement.

Thanks

Rad

[AdamR]

Hi Rad,

Going to try my best to answer each question in a way that should help.  What you describe is possible with some custom structure but is not the default behavior for property updates.

1.  Not with the normal property update structure from the edge device.  This is by design since many application structures rely on updated in ordered time.

2. This would be custom code, and it is dependent on what you want to do with the historical data (store it in streams, use it for a calculation, etc.)

3. Setting would allow smaller blocks of data to be sent by the property update, but it would do this as fast a it could process.  So there is no "bandwidth" throttling built in.  As before, dependent on use a custom implementation could slow the sending of historical data.

4.Yes that is possible.  You can also set the property update to "fold" all updates.  Meaning ignore all previous updates and only send the latest cached in the offline message store.

5. Assuming you are just looking to add the historical values to the ValueStream that is configured for the thing, I would suggest instead calling the Add(dataType)ValueStreamEntry service to record the data.  This eliminated additional server side processing and would achieve the same result.

I do not have additional info off the top of my head but will check to see if there is a recommended way to handle offline update history from the edge device.

Thanks,

Adam

[rad1]

Thanks AdamR,

Could you add a little snippets of code that would accomplish some of things I want to accomplish.

Rad