Packet loss using MQTT subscribe

[dbologna]

Hello,

we have defined a Thing with MQTT Template, receiving packets from MQTT broker.

Packets are coming and stored in a stream, but sometimes packets are lost.

We know that these packets have been sent from the MQTT broker but we don't find them in the stream.

The packet is received into a JSON property, and the subscription (defined on data change) read the JSON and transform it to store into the stream.

Any idea ? Many thank Dimitri

[PaiChung]

how rapidly are these messages coming in at times?

anything with a timestamp within 3 ms will not go into the stream or overwrite the existing entry.

So especially if you are queueing a lot of messages to be processed at once you might run into this issue.

[dbologna]

H i Thanks.

That's what I found.
What solution can I adopt to avoid losing information.

BR

Dimitri

[PaiChung]

Either you bulk / cache the incoming messages and process them as 'one' or you may have to force a 3ms pause() before writing the subsequent messages to be certain.

[dbologna]

Hi PaiChung,

packets are submitted from many machines then the we receive then with high frequency.

How can i 'bulk / cache the incoming messages'  in order to avoid to loose packets.

I cannot set a pause because I don't manage the submit on the  Mqtt Broker.

I Just manage only the subscription on Thingworx.

BR 

Dimitri

[PaiChung · Accepted solution]

I would then first put messages into an in memory infotable and then have that picked up by a separate service to handle the messages.

you can use the addrow/deleterow to very real time population of the memory queue and unload.