Community Tip - Did you get an answer that solved your problem? Please mark it as an Accepted Solution so others with the same problem can find the answer easily. X
The platform is Thingworx + PostgreSQL + ConnectionServer.
I have 10,000 devices which update values every 100ms. Assuming one property per device, a total of 10,000 properties need to be submitted to Thingworx. Submitting data at this speed should be able to query 600 data per minute per property. However, at present I can only find about 300 data.
Don't know if submitting data to Thingworx at this speed is beyond its capabilities?
What parameters should I adjust to enable Thingworx to process high-frequency data uploads?
What I am currently doing is adjusting the following parameters in platform-settings.json.
But I don't understand the meaning of many parameters. For example, most of the parameters in ValueStreamProcessorSettings. These parameter's explanation in the Installing_ThingWorx_8_2_1_0_174097.pdf file is too brief and completely incomprehensible. Can someone help me to explain it?
{
"PlatformSettingsConfig": {
"BasicSettings": {
"BackupStorage": "C:/ThingworxBackupStorage2",
"Storage": "C:/ThingworxStorage2"
}
},
"PersistenceProviderPackageConfigs": {
"PostgresPersistenceProviderPackage": {
"ConnectionInformation": {
"acquireIncrement": 5,
"acquireRetryAttempts": 3,
"acquireRetryDelay": 10000,
"checkoutTimeout": 1000000,
"driverClass": "org.postgresql.Driver",
"fetchSize": 5000,
"idleConnectionTestPeriod": 60,
"initialPoolSize": 5,
"jdbcUrl": "jdbc:postgresql://localhost:5432/twdb8_2_5",
"maxConnectionAge": 0,
"maxIdleTime": 0,
"maxIdleTimeExcessConnections": 300,
"minPoolSize": 5,
"maxPoolSize": 500,
"maxStatements": 10000,
"numHelperThreads": 8,
"password": "newstar123",
"testConnectionOnCheckout": false,
"unreturnedConnectionTimeout": 60,
"username": "twadmin"
},
"ValueStreamProcessorSettings": {
"maximumBlockSize": 4000000,
"maximumQueueSize": 4000000,
"maximumWaitTime": 100,
"numberOfProcessingThreads": 20,
"scanRate": 1000,
"sizeThreshold": 4000000
}
}
}
}
Solved! Go to Solution.
100,000 property writes per second is beyond what the setup you described is certified to handle. Here is some info about other databases that can be used with ThingWorx that can handle that number of property writes:
You should talk to a sales engineer about options to handle your use case. I would guess they will suggest federating multiple ThingWorx instances behind a load balancer.
100,000 property writes per second is beyond what the setup you described is certified to handle. Here is some info about other databases that can be used with ThingWorx that can handle that number of property writes:
You should talk to a sales engineer about options to handle your use case. I would guess they will suggest federating multiple ThingWorx instances behind a load balancer.
I noticed that the database is really the bottleneck. Thank you!