Skip to main content
W
Willie16-Pearl
16-Pearl
September 29, 2020
Question

How to not lose data when adding stream entries rapidly

  • September 29, 2020
  • 2 replies
  • 4043 views

I've noticed that I lose data when stream entries are added very rapidly.  This occurs due to the entries being added simultaneously in different threads.  The stream cannot have data with the same timestamp, so only one of them gets added if there are multiple entries being added simultaneously.  To prevent this from occurring, I've added random milliseconds to the timestamp.  However, when so many entries are being added simultaneously, I lose data since the simultaneous entries still seems to occur even with the random milliseconds.  Is there a better way to add stream entries without losing data?

 

 

var timestamp = new Date();
timestamp.setMilliseconds(Math.floor(Math.random() * 1000));

Things[StreamName].AddStreamEntry({
	sourceType: undefined /* STRING */,
	values: values /* INFOTABLE */,
	location: undefined /* LOCATION */,
	source: me.name /* STRING */,
	timestamp: timstamp /* DATETIME */,
	tags: tags /* TAGS */
});

 

 

2 replies

N
nmilleson17-Peridot
17-Peridot
September 29, 2020

@Willie does your actual stream data that you are entering have unique timestamps?  If so, you could use that as your timestamp.  A couple other things I have done when I run into similar problems:

 

  • with streams, if it's not a service that I'm running often, I'll just use the built-in pause() function and pause for 2 or 3 milliseconds between row adds
  • you can loop through the data first and assign sequential timestamps/milliseconds/etc. before you run the insert function
  • if you really want to use random (which would likely throw off your queries), you could keep a JSON object with a record of all the timestamps you've used already and do a hasOwnProperty() test on each new timestamp to ensure it's unique (that's kind of an ugly way to do it though and I'm not sure that's thread-safe)

 

W
Willie16-PearlAuthor
16-Pearl
September 29, 2020

@nmilleson 

 

  • does your actual stream data that you are entering have unique timestamps?

         No, some of them have the same timestamp. 

         I have a separate Timestamp column defined in the Data Shape for the asset time.  

         Precision is to HH:mm:ss, so milliseconds difference doesn't really matter.

         I am using the stream timestamp for record keeping purposes for when it was added to the stream.

 

  • if it's not a service that I'm running often, I'll just use the built-in pause() function and pause for 2 or 3 milliseconds between row addsIt's a service that runs pretty often.  It runs based on a subscription to a data change in a string property.  Since Axeda Agents (edge) do not have infotable properties, I am passing a line of text as a string property and parsing the string using a service I wrote to create stream entries.  
  • you can loop through the data first and assign sequential timestamps/milliseconds/etc. before you run the insert function

         I can't loop it since I can't predict when and how much data is coming in from the edge.

  • if you really want to use random (which would likely throw off your queries), you could keep a JSON object with a record of all the timestamps you've used already and do a hasOwnProperty() test on each new timestamp to ensure it's unique (that's kind of an ugly way to do it though and I'm not sure that's thread-safe)

         This method sounds interesting but I'm not clear on this. 

 

Maybe I can do a do while loop when adding the stream entry. If the stream entry throws an error when adding an entry with the same timestamp, it will keep trying until it was able to add.  Do you know if the stream entry service throws an error when adding an entry with the same timestamp?  I have a feeling it just ignores and doesn't throw an error.

N
nmilleson17-Peridot
17-Peridot
September 29, 2020

@Willie 

 

I'm not sure if a stream entry with a timestamp conflict throws an error.  Could you just modify your timestamp on the Axeda edge to have milliseconds?

A
abhiramk1-Visitor
1-Visitor
September 30, 2020

@Willie 

I have observed this behavior earlier, I have handled it by using source field. While inserting the rows to stream, i have set source as actual source concatenate  with GUID. As source is primary key if the source changes it will write it to stream without any issues. It will allow duplicate timestamp as source changes. 

Tags which i mentioned earlier is indexed column but not primary key.

    W
    Willie16-PearlAuthor
    16-Pearl
    September 30, 2020

    @abhiramk 

     

    Does this mean it will allow duplicate timestamps since tags is also a primary key?

    Terms & ConditionsCookie settingsAccessibility statement