SAP OData Connector - ERROR c.t.i.r.IntegrationEngine

[otluk]

The integration engine fails with status [ServerErrorException - Internal Server Error] 

How can the integtration engine be stabilized to handle these kind of error?

2020-11-08 18:18:43.651 [ClientProcessor-1] ERROR c.t.i.r.IntegrationEngine - Exchange process failed with exception thrown : Your route exchange has failed.  Service operation failed with status [ServerErrorException - Internal Server Error]
java.lang.RuntimeException: Your route exchange has failed.  Service operation failed with status [ServerErrorException - Internal Server Error]
        at com.twx.integration.route.exceptions.ExceptionHelper.convertToRuntimeException(ExceptionHelper.java:77)
        at com.twx.integration.route.IntegrationEngine.dispatchRoute(IntegrationEngine.java:159)
        at com.twx.integration.client.things.IntegrationRuntimeServer.Invoke(IntegrationRuntimeServer.java:72)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.thingworx.common.processors.ReflectionProcessor.processService(ReflectionProcessor.java:261)
        at com.thingworx.communications.client.things.VirtualThing.invokeService(VirtualThing.java:1120)
        at com.thingworx.communications.client.things.VirtualThing.handleServiceRequest(VirtualThing.java:1028)
        at com.thingworx.communications.client.ConnectedThingClient.handleRequest(ConnectedThingClient.java:315)
        at com.thingworx.communications.client.processor.ClientProcessorTask.run(ClientProcessorTask.java:55)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

 Someone has some hints ideas what went wrong? 

Followed the instructions here: https://support.ptc.com/help/thingworx/platform/r9/en/index.html#page/ThingWorx/Help/Composer/IntegrationConnectors/ConnectorOview.html

Thanks!

[slangley]

Hi @otluk.

Which version of ThingWorx are you running?  Are you saying that the connection is sometimes successful?  There could be external factors affecting the ability to connect, such as a network issue, if it appears to be random.

In the most recent releases, we recommend the use of ThingWorx Flow for establishing external connections.

Regards.

--Sharon

[otluk]

Hi @slangley ,

Running: integration-runtime-8.0.7-b870 (from marketplace) with ThingWorx 8.4.9 .

Prefer to use the SAP-ODATA connector Thing directly in ThingWorx.

Network topic seems to be unlikely as both machines run in the same network.

Full log of integration runtime: 

2020-11-08 17:50:48.907 [vert.x-eventloop-thread-0] INFO  c.t.i.IntegrationRuntimeConfiguration - Reading from config file D:\IntegrationRuntime\integrationRuntime-settings.json
2020-11-08 17:50:48.923 [vert.x-worker-thread-0] INFO  c.t.i.r.IntegrationEngine - Starting Integration Engine
2020-11-08 17:50:49.048 [vert.x-worker-thread-1] INFO  c.t.c.c.p.ClientMessageProcessor - Created new ClientMessageProcessor [minPoolSize: 4, maxPoolSize: 10, maxThreadLife: 10000, maxQueueSize: 1000]
2020-11-08 17:50:49.079 [vert.x-worker-thread-1] INFO  c.t.i.ThingworxEdgeVerticle - Waiting for client to connect
2020-11-08 17:50:49.079 [Client-EndpointMonitor-1] INFO  c.t.c.c.e.m.ConnectivityMonitorTask - Endpoint not connected [name: IR-DEDCB3242-ddfa34db-bd73-402d-b141-45ff018f04da, id: 0]. Attempting [re]connect in 0 seconds...
2020-11-08 17:50:49.079 [Client-EndpointMonitor-1] INFO  c.t.c.c.e.ClientCommunicationEndpoint - Needed to refill connections on client endpoint [id: 0] : [active: 0, max: 1]
2020-11-08 17:50:49.376 [vert.x-worker-thread-0] INFO  o.a.c.i.c.DefaultTypeConverter - Type converters loaded (core: 195, classpath: 0)
2020-11-08 17:50:49.391 [vert.x-worker-thread-0] WARN  o.a.c.i.c.DefaultTypeConverter - Overriding type converter from: StaticMethodTypeConverter: public static java.lang.String org.apache.camel.converter.IOConverter.toString(java.io.InputStream,org.apache.camel.Exchange) throws java.io.IOException to: InstanceMethodTypeConverter: public java.lang.String com.twx.integration.route.converter.InputStreamToStringTypeConverter.toString(java.io.InputStream,org.apache.camel.Exchange) throws org.json.JSONException,java.io.IOException
2020-11-08 17:50:49.391 [vert.x-worker-thread-0] WARN  o.a.c.i.c.DefaultTypeConverter - Overriding type converter from: StaticMethodTypeConverter: public static com.thingworx.types.InfoTable com.twx.integration.route.converter.HttpConnectionResponseTypeConverter.toInfoTable(org.apache.camel.Exchange) to: StaticMethodTypeConverter: public static com.thingworx.types.InfoTable com.twx.integration.route.converter.HttpResponseHeadersConverter.toInfoTable(org.apache.camel.Exchange)
2020-11-08 17:50:49.391 [vert.x-worker-thread-0] WARN  o.a.c.i.c.DefaultTypeConverter - Overriding type converter from: InstanceMethodTypeConverter: public org.w3c.dom.Document org.apache.camel.converter.jaxp.XmlConverter.toDOMDocument(java.lang.String,org.apache.camel.Exchange) throws java.io.IOException,org.xml.sax.SAXException,javax.xml.parsers.ParserConfigurationException to: InstanceMethodTypeConverter: public org.w3c.dom.Document com.twx.integration.route.converter.XMLDocumentTypeConverter.toDocument(java.lang.String) throws javax.xml.parsers.ParserConfigurationException,java.io.IOException,org.xml.sax.SAXException
2020-11-08 17:50:49.391 [vert.x-worker-thread-0] WARN  o.a.c.i.c.DefaultTypeConverter - Overriding type converter from: InstanceMethodTypeConverter: public org.w3c.dom.Document org.apache.camel.converter.jaxp.XmlConverter.toDOMDocument(java.io.InputStream,org.apache.camel.Exchange) throws java.io.IOException,org.xml.sax.SAXException,javax.xml.parsers.ParserConfigurationException to: InstanceMethodTypeConverter: public org.w3c.dom.Document com.twx.integration.route.converter.XMLDocumentTypeConverter.toDocument(java.io.InputStream) throws java.io.IOException,org.xml.sax.SAXException,javax.xml.parsers.ParserConfigurationException
2020-11-08 17:50:49.735 [Client-EndpointMonitor-1] INFO  c.t.c.c.e.ClientCommunicationEndpoint - Preparing new Connection Authentication Request: DispatchingClientEndpoint [id: 0, isConnected: false, open connections: 0, max connections: 1]
2020-11-08 17:50:49.782 [Client-EndpointMonitor-1] INFO  c.t.c.c.e.ClientCommunicationEndpoint - Authentication Request was SUCCESSFUL [tworx session id: 136060142, connection group id: 5]
2020-11-08 17:50:49.782 [Client-EndpointMonitor-1] INFO  c.t.c.c.e.ClientCommunicationEndpoint - Successfully reconnected client endpoint [id: 0] : [active: 1, max: 1]
2020-11-08 17:50:49.782 [Client-EndpointMonitor-1] INFO  c.t.c.c.ConnectedThingClient - Communications Endpoint Opened
2020-11-08 17:50:49.782 [Client-EndpointMonitor-1] INFO  c.t.c.c.e.m.ConnectivityMonitorTask - Endpoint reconnected [name: IR-DEDCB3242-ddfa34db-bd73-402d-b141-45ff018f04da, id: 0]
2020-11-08 17:50:49.813 [vert.x-worker-thread-0] INFO  o.a.c.i.DefaultCamelContext - Apache Camel 2.23.1 (CamelContext: camel-1) is starting
2020-11-08 17:50:49.829 [vert.x-worker-thread-0] INFO  o.a.c.m.ManagedManagementStrategy - JMX is enabled
2020-11-08 17:50:50.126 [vert.x-worker-thread-1] INFO  c.t.i.ThingworxEdgeVerticle - The client is now connected.
2020-11-08 17:50:50.173 [vert.x-worker-thread-1] INFO  c.t.i.ThingworxEdgeVerticle - Route Configurations Received, got 0 route configurations
2020-11-08 17:50:50.173 [vert.x-worker-thread-1] WARN  c.t.i.ThingworxEdgeVerticle - No routes could be configured from Thingworx Response
2020-11-08 17:50:50.204 [vert.x-worker-thread-1] INFO  c.t.i.ThingworxEdgeVerticle - #########
2020-11-08 17:50:50.204 [vert.x-worker-thread-1] INFO  c.t.i.ThingworxEdgeVerticle - Connection Established
2020-11-08 17:50:50.204 [vert.x-worker-thread-1] INFO  c.t.i.ThingworxEdgeVerticle - #########
2020-11-08 17:50:50.219 [vert.x-worker-thread-1] INFO  ROOT - Setting root logger level to WARN
2020-11-08 18:08:19.104 [ClientProcessor-1] ERROR c.t.i.r.IntegrationEngine - Exchange process failed with exception thrown : Your route exchange has failed.  Service operation failed with status [ServerErrorException - Internal Server Error]
java.lang.RuntimeException: Your route exchange has failed.  Service operation failed with status [ServerErrorException - Internal Server Error]
	at com.twx.integration.route.exceptions.ExceptionHelper.convertToRuntimeException(ExceptionHelper.java:77)
	at com.twx.integration.route.IntegrationEngine.dispatchRoute(IntegrationEngine.java:159)
	at com.twx.integration.client.things.IntegrationRuntimeServer.Invoke(IntegrationRuntimeServer.java:72)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.thingworx.common.processors.ReflectionProcessor.processService(ReflectionProcessor.java:261)
	at com.thingworx.communications.client.things.VirtualThing.invokeService(VirtualThing.java:1120)
	at com.thingworx.communications.client.things.VirtualThing.handleServiceRequest(VirtualThing.java:1028)
	at com.thingworx.communications.client.ConnectedThingClient.handleRequest(ConnectedThingClient.java:315)
	at com.thingworx.communications.client.processor.ClientProcessorTask.run(ClientProcessorTask.java:55)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
2020-11-08 18:09:10.738 [ClientProcessor-4] ERROR c.t.i.r.IntegrationEngine - Exchange process failed with exception thrown : Your route exchange has failed.  Service operation failed with status [ServerErrorException - Internal Server Error]
java.lang.RuntimeException: Your route exchange has failed.  Service operation failed with status [ServerErrorException - Internal Server Error]
	at com.twx.integration.route.exceptions.ExceptionHelper.convertToRuntimeException(ExceptionHelper.java:77)
	at com.twx.integration.route.IntegrationEngine.dispatchRoute(IntegrationEngine.java:159)
	at com.twx.integration.client.things.IntegrationRuntimeServer.Invoke(IntegrationRuntimeServer.java:72)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.thingworx.common.processors.ReflectionProcessor.processService(ReflectionProcessor.java:261)
	at com.thingworx.communications.client.things.VirtualThing.invokeService(VirtualThing.java:1120)
	at com.thingworx.communications.client.things.VirtualThing.handleServiceRequest(VirtualThing.java:1028)
	at com.thingworx.communications.client.ConnectedThingClient.handleRequest(ConnectedThingClient.java:315)
	at com.thingworx.communications.client.processor.ClientProcessorTask.run(ClientProcessorTask.java:55)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

Some hints how to stabilize this Apache Camel based runtime service? (currently just manually staring via command line)

[slangley]

Hi @otluk.

Does it consistently fail, or is the failure random?

Regards.

--Sharon

[otluk]

Hi Sharon,

Failure seems to be random. As the logs do not indicate why and when the runtime environment fails.

Regards,

--Luk

[bot1q]

Same Issue here sometimes.

Sometimes the restart on the Integration Runtime helps...

But on prod systems it is realy unstable ...

[slangley]

Hi @otluk.

I recommend opening a case for this issue. If you agree, I will be happy to open it on your behalf.

Regards.

--Sharon

[bot1q]

Our case is still open since 2 weeks ... maybe tommorow....

[slangley]

Hi @otluk.

Have you opened a case for this issue?  If so, please provide me with the case number under private email so I can follow up.

Regards.

--Sharon

[slangley]

Hi @otluk.

I have created case 15566647 on your behalf and assigned it to the appropriate support queue.  I will monitor for updates and once it is resolved, will post the solution here.

Regards.

--Sharon

[slangley · Accepted solution]

Hi @otluk.

Per the notes on the case, you assumed that the issues you were experiencing were network related. 

If you agree with that assessment, please mark this response as the Accepted Solution for the benefit of others with similar issues.

Regards.

--Sharon