Individual streams per asset

[TanmeyTWX]

Hi,

Is it recommended to create stream per device? any major impact or performance issues if create a independent stream for each device Thing.

example: for 10k devices there will be 10K streams for data storage of a device.

Please revert your suggestions.

TIA.

[Ciprian-Traian]

Hello @ TanmeyTWX,

The amount of historical data to be stored on the platform impacts both database and platform system sizing.  
Larger data sources (streams, data tables, and value streams)will require longer transactionsto query from the database.
These long transactions can also cause the stream and value stream queues to back up and utilize additional memory.

The ThingWorx Platform Sizing Guide it is a good reference to check in this situation.

It may be of help to check the Application Development Guide for ThingWorx 8 because it gives more details on

how ThingWorx handles data storage.
Choosing the correct data storage is very important since it affects the outcome of the project,its scalability and reliability, and the user experience.

[TanmeyTWX]

Hi,

I guess you didn't get my query. let me rephrase it.

--> I have a Stream X for all the assets data and each Asset will generate lot of data that will sumup the records around 10 million

--> Now I want to store each Asset's data to its corresponding Stream A for Asset A, Stream B for Asset B,etc rather than in a single Stream.

My concern is whether its a good approach to create separate streams for Assets for storing data OR single Stream for all assets data, when it is going to be huge data.

[nmilleson]

@TanmeyTWX ,

All of the stream data goes into one table on the ThingWorx database (tables are not created per Stream entity).  Using one stream per Thing should in theory make your queries faster.  There are a few metadata columns which include things like timestamp, source, and tags. The source will be your Entity name.  Hope that helps.

[TanmeyTWX]

nmilleson,

Thanks for writing back.

Can you tell me creating 20K STREAMS in composer will impact the application performance?

[nmilleson · Accepted solution]

@TanmeyTWX ,

I don't think the number of Things (in this case Streams) will impact the performance, as even the Small ThingWorx sizing shows a limit of about 30,000 Things (see https://www.ptc.com/support/-/media/BC2F28281E4142AFA6FC79EC49D9DBD3.pdf?sc_lang=en page 12)

However, when your streams have over 100,000 data points in them, it's recommended that you use an external data store (see https://support.ptc.com/help/thingworx_hc/thingworx_8_hc/en/index.html#page/ThingWorx/Help/Best_Practices_for_Developing_Applications/DataStore.html under "Best Practices for Choosing Your Data Store Option")

Hope that helps!

[slangley]

Hi @TanmeyTWX.

If you feel your question has been answered, please mark the appropriate post as the Accepted Solution for the benefit of others with the same question.

Regards.

--Sharon