Asyncronous Streams and Delayed Calculations Execution?

[CarlesColl]

Hi All,

I'm importing on batch old data ( in blocks ), this data it's automatically stored on a Stream ( either Value Stream or Stream ), and I need to do a calculation based on current value and previous value (accumulated consumption counter). I don't have previous value in memory, hence I have to go to the Stream and get previous value, and here the async stream problem arises: if I do too fast the previous value ( which I'm importing in batch "super fast" ) won't be stored on the Stream it's on the write queue.

As I control Data Change event on the property ( accumulated consumption counter ), I would like to execute a delayed Data Change event, there's an easy way to do it? All the ways I think of are too much or I don't like it:

• Build a Delayed Events queue, which will execute events after a while. I think this is the best way, but It's a lot of work to build it correctly, also I will need a timer to trigger events once in a while and this will add load to the server,...
• Do the calculations once a day . I don't like this solution, as not always I will load data on batch, for the same property on another Thing may I have "real time" data and I want on that case that the calculations are also in "real time"
• Delay a service call ( something like setTimeout() ) --> I think this is not possible on TW? maybe a combination of Async Service call plus a pause(), but I don't know the performance impact of such a solution on the TW Threading model, something like this

  • Async Service:  void SetTimeout(thing,service,params,milliseconds) {
       pause(milliseconds);
       Things[thing][service](params);
    }

Best Regards,

Carles.

[paic]

Thingworx supports the pause service

basically

pause(1000) would be pause for 1000 miliseconds.

[CarlesColl]

Hi Pai,

Yes, already it's on my proposals, but the question about using pause(n) it's how will this affect on the threading model... I mean, let's say I import 100k registers in batch, which in turn does 100k SetTimout ( function as shown above ) what will happen to TW?

Best Regards,

Carles.

[CarlesColl]

I'm doing test with the fake SetTimeout function ( which works ) , first limitation: pause(n) it's limited to 60000 milliseconds . Yes I know I can iterate over pause(n) calls, but it's awful.

[AdamR]

Hi Carles,

Is there a reason that you are not keeping the previous value you need in memory?  Since I do not know the full scope of the transactions I was not sure if the data value was large or some further reason.  If you can store the previous value it should eliminate the need to pause?

Thanks,

Adam

[CarlesColl]

Hi All,

This post got lost on the community ( it went to another place instead of Developer Community ).

Just for the record, and if someone has the same needs, or maybe anyone can think on other options...

• Build a Delayed Events queue, which will execute events after a while. I think this is the best way, but It's a lot of work to build it correctly, also I will need a timer to trigger events once in a while and this will add load to the server,...
  • I followed this approach:
    • Created a DataTable where Events are pushed with adding a Timeout, and with a timer I trigger delayed events when Timeout it's reached.
• Do the calculations once a day . I don't like this solution, as not always I will load data on batch, for the same property on another Thing may I have "real time" data and I want on that case that the calculations are also in "real time"
• Delay a service call ( something like setTimeout() ) --> I think this is not possible on TW? maybe a combination of Async Service call plus a pause(), but I don't know the performance impact of such a solution on the TW Threading model, something like this
  • I Tested this approach, but it ends ups consuming lots of RAM, as it creates a new Thread for each async SetTimeout call. If someone want's it, here it's, it can be useful but not for lot's of calls ( 10k calls = 1Gb RAM ),

async function SetTimeout(thing,service,params,milliseconds) {
if (milliseconds>=60000) {
    var nTimes = parseInt(milliseconds/60000,10);
    for(var i=0;i<nTimes;i++) pause(59999);
    // -- As the following instruction doesn't works and it should: case: 12808137
    // -- we will just trigger half a minute more 
    //pause(parseInt(milliseconds%60000,10));
    pause(30000);
} else {
    pause(milliseconds);
}
if (thing) {
    Things[thing][service]((params?JSON.parse(params):null));
} else {
    me[service]((params?JSON.parse(params):null));
}
}

Best Regards,

Carles.