How many value streams we need to create.

[MK_10775467]

Consider a hypothetical example of a steel manufacturing company. The company has 50 motors and 50 sensors. So lets say my thing model has 100 things with 10 different properties. Each thing will denote a physical object. Now is there any rule or formula on how many value streams I need to create to log the data. I can create one value stream for each asset or 1 VS for 2 or more assets etc. I want to know how can I make a stable solution architecture over here. Will it be good if I own one or more database. One for the thingworx and other for storing value stream data. I want to ask below question.

1. Is there any rule or formula on knowing how much value streams we need to create.
2. Can we store data in separate database or in separate tables.
3. How will storing and querying the database for showing values in mashup will have an impact on the performance

[Velkumar]

Hi @MK_10775467 

Please refer below links :

Storing Data in ThingWorx (ptc.com)

Determining the Correct Data Store Option (ptc.com)

Question 1 : Solved: Limits of Value Streams - PTC Community

Question 2 : Yes if you are going to perform complex logic or query large dataset I would recommend to store data in external database. 

Question 3: For larger/complex query, external database will be faster than in build tables.

If you are going to store high frequency timeseries data you can influx DB to store it - Using InfluxDB as the Persistence Provider (ptc.com)

/VR 

[VladimirRosu · Accepted solution]

For Q1: the reply from 2018 is no longer correct. It is best that you have a number of Value Streams <= number of threads assigned to the Persistence Provider in case of InfluxDB. If you have 1 VS or 10 VS this would result in typically the same performance. You should not have tens or hundreds of VSs in the system. My advice is to just have 10 VSs and go with this number forever.

For Q2: it all depends on your definition of "data". For historical data that originates from property updates, I find it hard to believe that somebody would be able to create fast a better system than the one of the Value Streams. Remember that a ValueStream must have a PersistanceProvider, and you can define multiple PPs each pointing to a different DB. This would in effect allow you to store historical data in different databases. However, with Influx, I would really want to understand the data size and frequency to see if I really need for example 10 different Influx instances for each of the 10 VSs I imagined at point 1. InfluxDB is really optimized on writing data and supports high write speeds and I believe that for most workloads a properly sized InfluxDB instance is enough.

If the data you mention is other type of data (mapping tables, historical alarms etc) you probably will need to use either Streams (optimized for writes), DataTables or maybe even pure SQL tables.

For 3: that question is in reality not tied to ThingWorx, since reading/writing that data results in database writes/reads, which at the end of the day need to be optimized in a manner specific to each database. Depending on the data you need, you will find that the ingestion is typically handled quite well by ThingWorx while reading, which might inquire complex SQL/SQL-like queries needs to be optimized based on the impact on the DB server itself.

Note that whenever I say DB server, this might mean InfluxDB or any other JDBC database out there, including the DB ThingWorx uses for its own needs (PostgreSQL, MSSQL, AzureSQL etc).

Overall you'd need to be more specific to what type of data you're querying or storing in order to make sure you get an efficient advice. ThingWorx has a few mechanisms in this area, that need to be well understood before deciding.

[VladimirRosu]

Hi @MK_10775467 .

For other's benefit, please make sure to mark down the Accepted Solution for the reply that helped you answer your questions.

Thanks!