ThingWorx crashing due to JS code

[tk512]

We're running ThingWorx 8.2.2 and it's crashing due to JS code on data event. I logged a case but I see it did not go through (also it has a 1200 character limit).

What happens is that seemingly trivial JS code puts ThingWorx into a "poison message" mode where the following stack trace is repeated continuously until the server runs out of space. I consider this a TW bug. It does not seem to be related to whether the JS code is valid or not (we let our developers work on TW and play around with it, so I don't expect TW to "crash" regardless of whether that code is valid or not).

java.util.concurrent.RejectedExecutionException: Task com.thingworx.system.subsystems.eventprocessing.EventInstance@63e9304 rejected from com.thingworx.common.utils.MonitoredThreadPoolExecutor@a38e4d6[Running, pool size = 500, active threads = 500, queued tasks = 200000, completed tasks = 1226032]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
        at com.thingworx.system.subsystems.eventprocessing.EventRouter.dispatchEvent(EventRouter.java:122)
        at com.thingworx.store.subscription.impl.SubscriptionStoreInMemoryThreadSafe.fireEventByDescriptors(SubscriptionStoreInMemoryThreadSafe.java:222)
        at com.thingworx.store.subscription.impl.SubscriptionStoreInMemoryThreadSafe.fireEvent(SubscriptionStoreInMemoryThreadSafe.java:187)
        at com.thingworx.things.Thing.dispatchEvent(Thing.java:7464)
        at com.thingworx.things.Thing.rawSetPropertyVTQ(Thing.java:4028)
        at com.thingworx.things.Thing.rawSetPropertyVTQ(Thing.java:3872)
        at com.thingworx.things.Thing.handleSetPropertyWithUpdate(Thing.java:4113)
        at com.thingworx.things.Thing.internalSetPropertyVTQ(Thing.java:4106)
        at com.thingworx.things.Thing.setPropertyVTQ(Thing.java:4230)
        at com.thingworx.things.Thing.setPropertyValue(Thing.java:4155)
        at com.thingworx.dsl.engine.adapters.ThingworxEntityAdapter.put(ThingworxEntityAdapter.java:296)
        at org.mozilla.javascript.ScriptableObject.putProperty(ScriptableObject.java:2347)
        at org.mozilla.javascript.ScriptRuntime.setObjectProp(ScriptRuntime.java:1598)
        at org.mozilla.javascript.ScriptRuntime.setObjectProp(ScriptRuntime.java:1592)
        at org.mozilla.javascript.gen.Event_AnyDataChange_2._c_script_0(Event.AnyDataChange:27)
        at org.mozilla.javascript.gen.Event_AnyDataChange_2.call(Event.AnyDataChange)
        at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
        at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3090)
        at org.mozilla.javascript.gen.Event_AnyDataChange_2.call(Event.AnyDataChange)
        at org.mozilla.javascript.gen.Event_AnyDataChange_2.exec(Event.AnyDataChange)
        at com.thingworx.dsl.engine.DSLProcessor.executeService(DSLProcessor.java:168)
        at com.thingworx.dsl.DSLServiceHandler.processService(DSLServiceHandler.java:38)
        at com.thingworx.things.Thing.handleEvent(Thing.java:7613)
        at com.thingworx.things.Thing.processEvent(Thing.java:7549)
        at com.thingworx.system.subsystems.eventprocessing.EventInstance.run(EventInstance.java:61)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

The on data event for a property follows:

var date1 = new Date();
me.EndDate = date1;
var date2 = me.StartDate;

Day = (dateDifference(date1,date2))/86400000;
me.Days = 20 - Day;

var date3 = new Date();
me.EndDateSeal = date3;
var date4 = me.StartDateSeal;

DaySeal = (dateDifference(date3,date4))/86400000;
me.DaysSeal = 10 - DaySeal;

var date5 = new Date();
me.EndDateImpeller = date5;
var date6 = me.StartDateImpeller;
DayImpeller = (dateDifference(date5,date6))/86400000;
me.DaysImpeller = 100 - DayImpeller;

[AdamR]

What is the event type that subscription code is running on?  AnyDataChange, data change on the specific property?? Could help pin down the issue.

[tk512]

Hi Adam - yes, a data change on a specific property. But it's not easy to reproduce.   I've enabled the JS code again now and we're waiting for the same problem to re-occur.

[AdamR · Accepted solution]

From the error message posted it looks like there is a loop in the event processing (without looking at the application directly I cannot say 100%).  The error message notes an AnyDataChange event...

at org.mozilla.javascript.gen.Event_AnyDataChange_2._c_script_0(Event.AnyDataChange:27)

 Then in the service script I also note a set local property values, which would again trigger this same service.  This will cause the event processing subsystem to eventually overload and use all available JVM memory.

[tk512]

Adam - that does make sense.   That means of course that a developer would be able to take down the entire TW platform just by doing something like that. Is there a way to configure max iterations?   To avoid endless loops like that.

[tk512]

Adam - but how does one change properties from the JavaScript handler?  Do we have to use another type of event ?

[AdamR]

You would need to use just the DataChange event as opposed to the AnyDataChange.  This can be confusing (I had it mixed up at first too).  AnyDataChange fires on change to Any Property (not any change to a specific property).  

The only thing you need to watch for in that case is that there is not a loop across subscriptions.  For example, Property1 has a Datachange, which sets the value of Property2.  Property2 also has a DataChange event, it cannot update Property1 value or there is a loop between the two subscriptions.

[tk512]

Adam you are a legend - I think that did the trick!