cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Azure IoT Hub Connector issue with AzureIotSendMessage service!

Highlighted
Level 4

Azure IoT Hub Connector issue with AzureIotSendMessage service!

In the projects we have, we often use the AzureIoTSendMessage service to send commands to the machines. (C2D message)

after 5 January 2019 we started to have problems below.

It resolves briefly by restarting the connector but after a short time the problem occurs again.

I have both 8.1 production infrastructure and 8.3 for Development.

 

The problem is the folowing:

 

When I use the AzureIoTSendMessage service, dosen't work. Gives me the error below:

 

Execution error in service script [THGF_LALLAVAT1 IoTRestService] : Wrapped java.util.concurrent.TimeoutException: Timed out APIRequestMessage [requestId: 13762, endpointId: -1, sessionId: -1, method: POST, entityName: AzureIot-cxserver-3d164c0f-f22a-4a8c-80e4-48b576cd6c53, characteristic: Services, target: AzureIotSendMessage] Cause: Timed out APIRequestMessage [requestId: 13762, endpointId: -1, sessionId: -1, method: POST, entityName: AzureIot-cxserver-3d164c0f-f22a-4a8c-80e4-48b576cd6c53, characteristic: Services, target: AzureIotSendMessage]

 

 

 

My configuration:

 

PRODUCTION:

-Virtual machine front end (Windows server2016 Os Build 14393.2670 ): azure virtual machine B4m

-Thingworx 8.1

-Java JDK 1.8.0_92

 

-Virtual Machine with connector  (Linux Ubuntu 18.04 )

-Java JDK 1.8.0_191

-Azure IoT 2.0.0 Connector

 

DEV:

 

-Virtual machine front end (Windows server2012 R2 Os  ): azure virtual machine B4m

-Thingworx 8.3.0

-Java JDK 1.8.0_92

 

-Virtual Machine with connector  (Linux Ubuntu 18.04 )

-Java JDK 1.8.0_191

-Azure IoT 2.0.0 Connector

 

 

This configuration works until 5 of jannary.

 

I think that the problem is the Azure IoT connector 2.0.0.

 

Problably something has been updated on the azure side and now the connector is no longer compatible.

 

Can you help me please? 


Have you found same issue? Can you try the service? 

 

Thanks

 

ps: I attached the application log

 

 

 

6 REPLIES

Re: Azure IoT Hub Connector issue with AzureIotSendMessage service!

Hi @Gabriele1986.

 

We tested under ThingWorx 8.3.3 using the 2.0.0 version of the connector but were unable to recreate the issue. We don't believe this is the result of an upgrade/incompatibility since it works following a restart.

 

How many messages are being sent?  On the Azure side is logging reporting anything regarding the attempts being made to deliver messages?

 

Regards.

 

--Sharon

Re: Azure IoT Hub Connector issue with AzureIotSendMessage service!

Hi  Sharon,

 

To recreate the problem try to do this:

 

Create a Mashup and put a refresh of 5 seconds of the service AzureIoTSendMessage.

 

for me after 30-40 (more or less) messages,  gives me the error Timout.

 

 

Azure side I can not see anything because the messages do not arrive in azure.

 

I Attached the entity for testing,in the thing's service [LocalRestApiCall] make sure to change the thing name with your ThingName that implement the azure iot connector.

 

Then change or create[with name: iotaas] the thing with azure template.

 

Thank you

Re: Azure IoT Hub Connector issue with AzureIotSendMessage service!

I want to emphasize that yesterday night I used a new iot hub for a test : iohubNorthEurope

 

In the same way I call the service AzureIoTSendMessage in Thingworx (with the connector) that invoke a C2D message and at about 40-50 message sent gives me the error : Timed out APIRequestMessage, and the device stopping receiving message from this service in thingworx.

 

Then I tested with another iot hub iothubtestWestEurope and here no problem.

 

 

This is interesting because same environment, different region of iot hub, at one hand works: west Europe. On the other hand not: north Europe.

 

I tested iot hub directly without Thingworx with a program written in node.js (https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-node-node-c2d )

With this program I have no problems.

 

It seems that something has changed in iot hub northorpe that AzureConnector of thingworx no longer works.

 

I tested also locally in my local infrastructure with the same result: after 40-60 messages sent gives me the error below:

 

Error executing service LocalRestApiCall. Message :: Timed out APIRequestMessage [requestId: 6266, endpointId: -1, sessionId: -1, method: POST, entityName: AzureIot-cxserver-66dec122-0451-4d29-a37f-b2aba841023e, characteristic: Services, target: AzureIotSendMessage] -

 

 

Re: Azure IoT Hub Connector issue with AzureIotSendMessage service!

Hello @Gabriele1986,

 

At the thresholds you are reporting for when you notice this issue, it could be possible that you are running into a queue limit for your devices.  Note that there is a per-device C2D queue max of 50 messages within Azure.  Errors would result for any messages sent to the same device beyond this threshold.  After the messages from the queue have been processed, you should be able to begin sending additional messages.

 

For your IoT devices in Azure, you should be able to review their C2D message counts to see where you are sitting currently.  Also, a good test would be to see if the queue clears on its own with more time, as opposed to restarting the connector.  Can you review/test these items and let us know what you find?

 

Regards,

 

Stefan

Re: Azure IoT Hub Connector issue with AzureIotSendMessage service!

I had a consumer in node-red, so I consume all the messagges.

 

I noticied in the azure iot hub connector bash the folowing error:

log.png

The virtual machine is a Windows server 2012 R2 

4 GB Ram

Intel(R) Xenon CPU E5-2673 v3 @ 2.40 GHz 

java = jre1.8.0_92

 

After 1-2 hours of startup, the connector continues to require so many resources until the error occurs:

task.png

After the error I have to restart the connector in order to use C2D and D2C messages.

 

Any suggestions???? 

Thanks

Re: Azure IoT Hub Connector issue with AzureIotSendMessage service!

Can you please analyze the log of the connector?

 

 

I think there is the cause of the using a lot of resources.

 

18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: Failure creating client or receiver, retrying
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: Caught java.util.concurrent.ExecutionException: com.microsoft.azure.servicebus.ServiceBusException: The supplied offset '137460531720' is invalid. The last offset in the system is '1789592' TrackingId:bdff1d66-b344-47fa-992c-7ebfafabcb81_G17_B15, SystemTracker:iothub-ns-iothubprod-1202394-e704e8b244:EventHub:iothubproduction~8191|northeurope, Timestamp:2019-01-18T17:35:21 Reference:f6a37fb8-7192-493e-a2ff-34a87f7fd324, TrackingId:2a52f80c-1003-4bbc-b337-a23eed5711f3_B15, SystemTracker:iothub-ns-iothubprod-1202394-e704e8b244:eventhub:iothubproduction~8191|northeurope, Timestamp:2019-01-18T17:35:22 TrackingId:f779ab2f1a424441bcddd5f9a6d6df2c_G17, SystemTracker:gateway5, Timestamp:2019-01-18T17:35:22, errorContext[NS: iothub-ns-iothubprod-1202394-e704e8b244.servicebus.windows.net, PATH: iothubproduction/ConsumerGroups/northeurope/Partitions/0, REFERENCE_ID: fccec9_f2c_G17_1547832920447]
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: java.util.concurrent.CompletableFuture.get(Unknown Source)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: com.microsoft.azure.eventprocessorhost.EventHubPartitionPump.openClients(EventHubPartitionPump.java:126)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: com.microsoft.azure.eventprocessorhost.EventHubPartitionPump.specializedStartPump(EventHubPartitionPump.java:46)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: com.microsoft.azure.eventprocessorhost.PartitionPump.startPump(PartitionPump.java:70)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: com.microsoft.azure.eventprocessorhost.Pump.lambda$createNewPump$0(Pump.java:61)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: java.util.concurrent.FutureTask.run(Unknown Source)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: java.lang.Thread.run(Unknown Source)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: Inner exception com.microsoft.azure.servicebus.ServiceBusException: The supplied offset '137460531720' is invalid. The last offset in the system is '1789592' TrackingId:bdff1d66-b344-47fa-992c-7ebfafabcb81_G17_B15, SystemTracker:iothub-ns-iothubprod-1202394-e704e8b244:EventHub:iothubproduction~8191|northeurope, Timestamp:2019-01-18T17:35:21 Reference:f6a37fb8-7192-493e-a2ff-34a87f7fd324, TrackingId:2a52f80c-1003-4bbc-b337-a23eed5711f3_B15, SystemTracker:iothub-ns-iothubprod-1202394-e704e8b244:eventhub:iothubproduction~8191|northeurope, Timestamp:2019-01-18T17:35:22 TrackingId:f779ab2f1a424441bcddd5f9a6d6df2c_G17, SystemTracker:gateway5, Timestamp:2019-01-18T17:35:22, errorContext[NS: iothub-ns-iothubprod-1202394-e704e8b244.servicebus.windows.net, PATH: iothubproduction/ConsumerGroups/northeurope/Partitions/0, REFERENCE_ID: fccec9_f2c_G17_1547832920447]
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: com.microsoft.azure.servicebus.ExceptionUtil.toException(ExceptionUtil.java:43)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: com.microsoft.azure.servicebus.MessageReceiver.onClose(MessageReceiver.java:587)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: com.microsoft.azure.servicebus.amqp.BaseLinkHandler.processOnClose(BaseLinkHandler.java:75)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: com.microsoft.azure.servicebus.amqp.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:46)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:309)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:276)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: com.microsoft.azure.servicebus.MessagingFactory$RunReactor.run(MessagingFactory.java:375)
18:35:20.900 [pool-5-thread-4] WARN eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: java.lang.Thread.run(Unknown Source)
18:35:20.900 [pool-5-thread-4] DEBUG eventprocessorhost.trace - host AzureIot-9f448f05-3d8a-4c5b-9cb5-53661ee5a908: partition 0: Opening EH client

 

 

Thank you