Data is not reaching the Thingworx platform

[Abhiram]

Hi,

We have connected assets which is sending data to Thingworx platform from last two weeks, but from yesterday data is not reaching platform.

It's showing all assets isconnected=true but none of the assets data is not reaching platform.

I have checked the logs on device, its' giving me error code 1114(TW_GATEWAY_TIMEOUT), but the device is sending data.

Below are the set of Application logs keep on getting on the thingworx:

 - NULL MessageSynchronizationContext! Request either timed-out waiting for this response, or it was received by mistake: ResponseMessage [requestId: 4518085, endpointId: -1, sessionId: 30071, code: STATUS_SUCCESS, multipart: false, packet #: 0, total packets: 0]

- Could not dispatch async service request [NotifyPropertyUpdate] : Timed out APIRequestMessage [requestId: 4519346, endpointId: -1, sessionId: -1, method: POST, entityName: *8902A, characteristic: Services, target: NotifyPropertyUpdate]

- Could NOT send response for request 148738 on /Things/*1004613/Services/GetPropertySubscriptions/ [java.lang.Exception: No open connections were available on endpoint 26908]

- ERROR sending initial response: java.lang.Exception: No open connections were available on endpoint 26871.  Preparing to re-send response for request 148789 on /Things/*1004507/Services/GetPropertySubscriptions/

After tomcat restart everything started working fine, but this is happening many times.

can you guys help me how to fix this issue without tomcat restart, what is the root cause of these errors?

Thanks,

Abhiram Kuncham

[PaiChung]

If you can, I would open a support ticket for this at support.ptc.com

[Soumya15]

Yeah, I have already raised the ticket.

[Soumya15]

You should try checking on the activeThread count inWSExecutionProcessingSubsystem. If the thread count is over the limit , I have observed these types of behavior.

[slangley]

Hi @Soumya15.

If your issue has been resolved, please either post the solution or select one of the previous responses and mark it as the Accepted Solution for the benefit of others who may have the same question.

Regards.

--Sharon

[Soumya15]

Hi ,

The issue is on hold as of now as the logs shared were insufficient for the Support team. As the issue is not reproducible easily, waiting for a error scene. In the mean time we have been working on optimizing the PG writes and queries and other associated optimizations.

Hope it helps. Please share exact details if you feel I can be of help.

Regards,

Soumya

[sbt]

Hello Soumya,

We are able to reproduce it and very often we are getting this message.

Below message is for your reference.

2018-11-02 02:36:47.0655 | ERROR | Error sending property updates to the Platform (updateSubscribedProperties failed).
2018-11-02 02:36:47.0655 | ERROR | System.AggregateException: One or more errors occurred. ---> com.thingworx.common.exceptions.TwApiException: Error invoking Service for Entity. Gateway Timeout. (Code 1114)
at com.thingworx.communications.client.TwApiWrapper.InvokeService(ThingworxEntityTypes entityType, String entityName, String service, InfoTable parameters, InfoTable& result, Int32 timeout, Boolean forceConnect, Boolean throwOnOfflineSave)
at com.thingworx.communications.client.BaseClient.invokeService(ThingworxEntityTypes entityType, String entityName, String service, ValueCollection parameters, Int32 timeout)
at com.thingworx.communications.client.things.VirtualThing.updateSubscribedProperties(Int32 timeout)
at TWInjector.ThingworxClient.<TWSendData>b__1(IGrouping`2 group) in ....\ThingworxClient.cs:line 358
at System.Threading.Tasks.Parallel.<>c__DisplayClass42_0`2.<PartitionerForEachWorker>b__1()
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object )
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at System.Threading.Tasks.Parallel.PartitionerForEachWorker[TSource,TLocal](Partitioner`1 source, ParallelOptions parallelOptions, Action`1 simpleBody, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, Action`1 body)
at TWInjector.ThingworxClient.TWSendData(JArray messages) in ..\ThingworxClient.cs:line 330
---> (Inner Exception #0) com.thingworx.common.exceptions.TwApiException: Error invoking Service for Entity. Gateway Timeout. (Code 1114)
at com.thingworx.communications.client.TwApiWrapper.InvokeService(ThingworxEntityTypes entityType, String entityName, String service, InfoTable parameters, InfoTable& result, Int32 timeout, Boolean forceConnect, Boolean throwOnOfflineSave)
at com.thingworx.communications.client.BaseClient.invokeService(ThingworxEntityTypes entityType, String entityName, String service, ValueCollection parameters, Int32 timeout)
at com.thingworx.communications.client.things.VirtualThing.updateSubscribedProperties(Int32 timeout)
at TWInjector.ThingworxClient.<TWSendData>b__1(IGrouping`2 group) in ...\ThingworxClient.cs:line 358
at System.Threading.Tasks.Parallel.<>c__DisplayClass42_0`2.<PartitionerForEachWorker>b__1()
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object )<---

for every 30 seconds we are updating values of multiple assets(around 2000) with each asset having (15 properties) to there respective remote thing.

all the values are updating in parallel task. 
thing.updateSubscribedProperties(20000);
thing.updateSubscribedEvents(60000);

Thanks.

[Soumya15]

Hi sbt,

I believe you would be persisting these properties ? Is it postgres DB writes. To handle the DB writes are you writing all the properties for a single thing in a batch ? You can also try grouping the DB writes based on a group of things to optimize DB writes.

Also another workaround would be to increase the DB max connections to handle the large incoming data surge and associated DB writes/queries.

Regards,

Soumya

[sbt]

Hello Soumya,

Thingworx doesnot allow the default persistent provider "Postgres" to change the settings. Can you please tell me the way to update the settings.

Also on the data transfer, we are first grouping the data by thing and sending data in parallel on to the thingworx platform. (meaning sending multiple thing updates at the same time).

Below is the settings what i have currently. 

Thanks for all your support.

[Soumya15]

Hi sbt,

I recall you can set these parameters for the 1st time you create a persistence provider. During that time you can edit the "Max Connection Pool Size" to something larger than the default 100 value, e.g. 5000. Also you can update at a later point of time if its a RDS instance from the AWS dashboard or PGAdmin interface. But just to let you know this is a workaround and the problem would be still there if the number of writes increase.

Also,"sending data in parallel" is not a good way if you don't know if the parallel writes are more than the max connections you have available to you.

You can observe the "max connection" count when the issue occurs. I suspect this would be maxing out already and the server is busy performing PG writes and the incoming data is queued internally to be processed later(hypothetically giving the illusion that the server is ON and receiving data, whereas issue start happening on the edge WS connections).

Hope it helps.

Regards,

Soumya