How to avoid data loss when adding multiple stream entries

[TanmeyTWX]

Hi @Willie 

Were you able to find any solution for this?

I have come across similar issue where multiple entries are being updated simultaneously and UpdateStreamEntry leaks few of the entries.

Note: My Stream datashape has String,Number and Infotable type columns.

[Willie]

Hi @TanmeyTWX 

The use case was scrapped so I did not need the stream anymore.  According to @abhiramk , you can have the same timestamp in a stream if the source is different, but in my case, I had the same source so I was adding random milliseconds.  It might be possible to make the milliseconds less random if you have a property on the stream to act as a counter for milliseconds that resets to 0 after 999, but I haven't tried it so I don't know if it would work.  The threads might (probably) access and increment the property simultaneously.  As @nmilleson mentioned, maybe you could have a staging area for your data before adding them to a stream so you could ensure they are added.  Stage the data then change the timestamp one by one before adding.

[Willie]

Hi @TanmeyTWX 

I asked ChatGPT.   Here's the response:

I think a combination of #1 and #2/#4 could work.

[Sathishkumar_C]

• How many records do you have in same timestamp? As you mentioned, I am passing a line of text as a string property and parsing the string using a service I wrote to create stream entries.  
• What is the data format? How you are parsing it? Is it a JSON?
• How frequently data change is happening?

[TanmeyTWX · Accepted solution]

Hi Sathish,

I have segregated the nested infotables into separate infotables and using only AddStreamEntry and UpdateStreamEntry is not required anymore which actually was causing issue. Hence the issue is resolved.