Subscriptions are not getting invoked, or delay in the Invoke

[pshashipreetham]

Hi,

There are subscriptions on EventAnyDataChange, but when ever there is any change in the Property the subscription are not getting invoked. There is no issue in logging the data in to the DB, that's working fine, but the subscriptions are not invoking.

Restarted the VM and also the TomCat.

@slangley , @PaiChung 

Thanks,
Shashi.

[VladimirRosu]

I would doubt that the subscriptions do not start execution.

Can you offer a bit more specificity, like:

• What ThingWorx version displays this?
• what type of property exhibits this?
• what is the DataChange value for that property definition?
• modify the subscription code to write in the logs the start of the subscription, ie: first line: logger.warn("Subscription execution started for Thing"+me.name);

With the correct DataChange settings you should not have this issue.

[pshashipreetham]

Hi @VladimirRosu ,

Thanks for the reply,

• Thingworx version: 9.3.2
• Number/String DataType Property, PFA:
  
• I did put the Logger on the first line of subscription for testing, that's how got to know the subscriptions are not invoking.
  
  Thanks,
  Shashi

[VladimirRosu]

There are a couple of things which are not obvious when you try to execute subscriptions.

First, if a property value has a timestamp older than the current property value, the *DataChange events will not be triggered. Instead, the HistoricalPropertyLogged event will be triggered.

Second, if a new property value comes in the system, and has timestamp >= the current property timestamp, and has the same value, then the *DataChange events will not be triggered if the DataChange type is set to "Value".

Both topics above are the default behavior of ThingWorx, and they are documented in the Help Center.

I do not know which of the ones above apply to you, but I can give you a potential situation when that might happen: when you have a large batch of properties that are sent to ThingWorx (either via EMS or via REST, through UpdatePropertyValues), due to the parallel nature of the threads in the Event Subsystem, you might write in ThingWorx a value with a newer timestamp before you wrote the older properties in the DataSet.

The solution in the scenario I explained is just to use both events (*DataChange and HistoricalPropertyLogged) and point them to the same processing service.

While I'm very confident one of the above might be the case, or something relatively similar/default behavior,I do not know if this is really your situation.

In this situation, I think the investigation it is better suited for a Technical Support case (just to make sure we're not overlooking any other reasons which might really be bugs). Could you please open a case with Technical Support, meaning you do have access to our Support portal and can do that?

[pshashipreetham · Accepted solution]

Hi @VladimirRosu ,

Thanks for the reply, found the issue, this is due to Subscriptions are taking more time to execute.

As per below pic, there were more than 1000+ task awaiting to be executed, that's the reason the subscriptions are not invoking or invoking late. The Tasks are pending because, our DataTables have more than 20,000 entries, querying them and applying the filters is causing delay in execution of services present in the Subscriptions.

Purging the DataTable Entries and restarting the Thingworx fixed this issue.

Thanks,
Shashi.

[VladimirRosu]

Hi @pshashipreetham 

I'm writing this because I believe the root cause of this issue deserves more explanation than what is here, and also because the observed issue here is the fact that subscriptions are delayed - but not completely forgot as part of your post title suggested.

When a subscription executes, it hits the Event Subsystem. By default, that uses 16 threads, and has (by default) a limit of 200.000 tasks that can be added in it before it adds a new thread to process the incoming tasks.

If the subscriptions you execute are slow (due to whatever reason), they will indeed add up as tasks in this subsystem, but they will execute at some point in time, which can not be calculated and depends on the code complexity and the state of the system.

What concerns me though, is that purging the Data Table you mentioned should not be considered a resolution for this issue (I assume you have data required for the solution proper behavior?). I think this issue will manifest itself again, considering that probably you will have new entries in that Data Table adding up.

Instead, you should focus on making sure the execution time of your code is fast, and I suggest you would switch to an SQL table in this situation, if indeed that is the core issue.

Again, it all depends on what you actually do in that subscription, which we do not know, but the main point is that, at the level of detail I observed in this post, I do not recommend purging to be a reliable strategy (unless it was agreed with the customer or the users of that application and does not impact the application usage).

[pshashipreetham]

Hi @VladimirRosu ,

Yes, Purging will not solve this issue; to ensure that the delay is due to Querying the DataTable 20,000 entries causing this issue, I have purged and checked in Dev Env to confirm. The client will not agree to this, so discussions are going on how the Architecture should be for the next release.

Moving to Streams on the INFLUX DB as a secondary persistence provider is what I am researching. Also checking on the performance tuning of Postgres too.

Thanks,
Shashi.

[VladimirRosu]

I strongly believe that an SQL table will fit this scenario well (using the Database Thing with direct SQL queries).

One thing about Influx: if your code does frequent deletes or updates in Influx, those generally speaking are not good for Influx. Update for example, in Influx, is transformed in an Delete+Insert.

Remember that Influx can store Streams and Value Streams only, and in your original post you mentioned a DataTable, so take that into account.

[pshashipreetham]

@VladimirRosu ,

Sure, Will give a try with SQL Query with Database thing and present it to my Seniors !

Thanks
Shashi.