EMS service invoke/property change timesOut from Thingworx Platform

[Ramkumar]

We have EMS deployed in remote devices and are connected to Thingworx platform via connection server

There are instances where remote device shows as connected but any request to device timesOut with below error in logs

JavaException: java.util.concurrent.TimeoutException: Timed out APIRequestMessage [requestId: 1878, endpointId: -1, sessionId: -1, method: POST, entityName: *deviceName, characteristic: Services, target: serviceName]

NULL MessageSynchronizationContext! Request either timed-out waiting for this response, or it was received by mistake: ResponseMessage [requestId: 1879, endpointId: 856, sessionId: 733353036, code: STATUS_SUCCESS, multipart: false, packet #: 0, total packets: 0]

We do not have physical access to remote device, so we cannot access EMS logs

Any information on the below could be helpful
- does network connectivity going on and off causes a lot of open sockets(lot of connection were in FIN_WAIT2) causing this issue?
- is there any service to restart EMS from Thingworx platform or Connection service?(could not find existing services under remote* entities)
- is there any service to track if EMS in device is running as expected or hanged(i.e a service to perform health check)?
- will restarting Connection Server messes existing other connections?
- is there any way to track particular device connection status from connection server?

We have tried
- invoking BrowseDirectory service from FileSystemService(cant access EMS logs as it is in a different directory)
- update timeout setting in WSCommunicationsSubsystem/Connection Server as requests were failing after 45 seconds

Environment
Thingworx platform :8.3.3
EMS Version : 5.4.7

Tried few articles(cs235581  and Data-is-not-reaching-the-Thingworx-platform ) and still no direction

Regards

[slangley]

Hi @Ramkumar.

To answer your questions:

- If you are unable to access the EMS logs, it sounds like the BrowseDirectory service has not been set up correctly.  It needs to point to the folder where the logs exist.

- The network dropping will leave a lot of open sessions until they time out

- For restarting the EMS, here is a list of services that can be called.  Notice there is one for Restart.

- For a service to check the status of a remote device, you can use the Remote monitor in ThingWorx.  However, if a disconnect has occurred, you may have to wait for it to timeout to confirm the current status.

- Restarting Connection server will drop all existing connections.

- For determining the status of a device, as long as Connection Server is up and running and connected to ThingWorx, you should be able to see the status.  You can also use the Connection Server console for validating status of the connection server.

You can access the EMS Help Center here.

Regards.

--Sharon

[Ramkumar]

Hi @slangley ,

Thanks for the updates.

The issue that we are currently facing is that EMS connection running in the remote embedded device shows online (isConnected=true) but any request to invoke remote service/ property writes fails. In such a scenario, we are do not have any way to connect to remote device nor troubleshoot it.

"The network dropping will leave a lot of open sessions until they time out"

- we suspect network drop causes a number of not properly terminated sessions resulting platform not able to reach remote device.Again, we do not have any evidence to conclude on the same.

"For restarting the EMS, here is a list of services that can be called. Notice there is one for Restart."

Our understanding was these REST services cannot be invoked from platform (Will look into documentation) onto a remote service.

"For a service to check the status of a remote device, you can use the Remote monitor in ThingWorx. However, if a disconnect has occurred, you may have to wait for it to timeout to confirm the current status."

Issue being status shows as connected, but cannot invoke services/property update from platform.

Please recommend any best practices or directions to troubleshoot, if any, when the device is not reachable but still shows as connected in platform

[slangley]

Hi @Ramkumar.

If the network isn't reliable resulting in a lot of connection failures, you will need to tune your system for the most appropriate timeout setting.  Setting the timeout shorter may help but setting it too short could be prone to more problems.  It's basically a trial and error activity to find the right setting.

As far as calling a REST service for the EMS, take a look at the ContentLoaderFunctions under Resources.  These should provide all the functions needed to manage your EMS remotely.

Regards.

--Sharon 

[slangley]

Hi @Ramkumar.

If you found one of the previous responses helpful, please mark the appropriate one as the Accepted Solution for the benefit of others with the same issue.

Regards.

--Sharon