Skip to main content
T
TanmeyTWX17-Peridot
17-Peridot
February 6, 2023
Solved

How to avoid data loss when adding multiple stream entries

  • February 6, 2023
  • 2 replies
  • 1708 views

Hi @Willie 

Were you able to find any solution for this?

I have come across similar issue where multiple entries are being updated simultaneously and UpdateStreamEntry leaks few of the entries.

 

Note: My Stream datashape has String,Number and Infotable type columns.

Best answer by TanmeyTWX

Hi Sathish,

 

I have segregated the nested infotables into separate infotables and using only AddStreamEntry and UpdateStreamEntry is not required anymore which actually was causing issue. Hence the issue is resolved.

2 replies

W
Willie16-Pearl
16-Pearl
February 6, 2023

Hi @TanmeyTWX 

The use case was scrapped so I did not need the stream anymore.  According to @abhiramk , you can have the same timestamp in a stream if the source is different, but in my case, I had the same source so I was adding random milliseconds.  It might be possible to make the milliseconds less random if you have a property on the stream to act as a counter for milliseconds that resets to 0 after 999, but I haven't tried it so I don't know if it would work.  The threads might (probably) access and increment the property simultaneously.  As @nmilleson mentioned, maybe you could have a staging area for your data before adding them to a stream so you could ensure they are added.  Stage the data then change the timestamp one by one before adding.

W
Willie16-Pearl
16-Pearl
February 10, 2023

Hi @TanmeyTWX 

I asked ChatGPT.   Here's the response:

Thingworx - 2023-02-10 - ChatGPT response to duplicate stream entry issue3.PNG

I think a combination of #1 and #2/#4 could work.

    S
    17-Peridot
    Sathishkumar_C17-Peridot
    17-Peridot
    February 13, 2023
    • How many records do you have in same timestamp? As you mentioned, I am passing a line of text as a string property and parsing the string using a service I wrote to create stream entries.  
    • What is the data format? How you are parsing it? Is it a JSON?
    • How frequently data change is happening?
    T
    TanmeyTWX17-PeridotAuthorAnswer
    17-Peridot
    April 19, 2023

    Hi Sathish,

     

    I have segregated the nested infotables into separate infotables and using only AddStreamEntry and UpdateStreamEntry is not required anymore which actually was causing issue. Hence the issue is resolved.

    Terms & ConditionsAccessibility statement