Skip to main content
17-Peridot
Feb 6, 2023 12:01 PM UTC
Solved

How to avoid data loss when adding multiple stream entries

Hi @Willie 

Were you able to find any solution for this?

I have come across similar issue where multiple entries are being updated simultaneously and UpdateStreamEntry leaks few of the entries.

 

Note: My Stream datashape has String,Number and Infotable type columns.

TanmeyTWXAnswer
17-Peridot
Apr 19, 2023 10:28 AM UTC

Hi Sathish,

 

I have segregated the nested infotables into separate infotables and using only AddStreamEntry and UpdateStreamEntry is not required anymore which actually was causing issue. Hence the issue is resolved.

4 replies

16-Pearl
Feb 6, 2023 7:41 PM UTC

Hi @TanmeyTWX 

The use case was scrapped so I did not need the stream anymore.  According to @abhiramk , you can have the same timestamp in a stream if the source is different, but in my case, I had the same source so I was adding random milliseconds.  It might be possible to make the milliseconds less random if you have a property on the stream to act as a counter for milliseconds that resets to 0 after 999, but I haven't tried it so I don't know if it would work.  The threads might (probably) access and increment the property simultaneously.  As @nmilleson mentioned, maybe you could have a staging area for your data before adding them to a stream so you could ensure they are added.  Stage the data then change the timestamp one by one before adding.

Translate
16-Pearl
Feb 10, 2023 7:21 PM UTC

Hi @TanmeyTWX 

I asked ChatGPT.   Here's the response:

Thingworx - 2023-02-10 - ChatGPT response to duplicate stream entry issue3.PNG

I think a combination of #1 and #2/#4 could work.

Translate
17-Peridot
Feb 13, 2023 4:24 AM UTC
  • How many records do you have in same timestamp? As you mentioned, I am passing a line of text as a string property and parsing the string using a service I wrote to create stream entries.  
  • What is the data format? How you are parsing it? Is it a JSON?
  • How frequently data change is happening?
Translate
TanmeyTWX17-PeridotAuthorAnswer
17-Peridot
Apr 19, 2023 10:28 AM UTC

Hi Sathish,

 

I have segregated the nested infotables into separate infotables and using only AddStreamEntry and UpdateStreamEntry is not required anymore which actually was causing issue. Hence the issue is resolved.

Translate