Why TWX Platform could not receive/queue all value streams updated by edges?
We made one edge update a temperature property increased by 1 every 5 secs. If 300k ~ 400k edges update logged temperature properties(value streams) every 5 secs concurrently, the ThingWorx Platform(TWX) could only receive/queue 85% ~ 90% of properties. When the edges' connections are established stably, We observed the 'totalWritesQueued' value of Value Stream Substem every 5 secs and computed its minus, then divided this minus by the amount of edges.

My question is, why the TWX couldn't handle 100% of properties when more than 300k edges update properties concurrently?
+ TWX System Resource:
- CPU E5-2650 v3 @ 2.30GHz with 10 cores * HT * 2-way
- CPU usage: less than 50%
- Total Memory: 128GB
- JVM Memory: 96GB
+ TWX Subsystem Configuration:
- Default except:
> Value Stream Subsystem:
' maximumBlockSize: 50,000
' maximumQueueSize: 10, 000, 000
' maximumWaitTime: 1000
' numberOfProcessingThreads: 80
+ Logs:
- Tomcat:
com.thingworx.exceptions.CouldNotConnectException: 172.21.128.30-temp13270 is not connected
java.util.concurrent.TimeoutException: Timed out APIRequestMessage [requestId: 1398634, endpointId: -1, sessionId: -1, method: POST, entityName:
172.21.128.30-temp11300, characteristic: Services, target: NotifyPropertyUpdate]
- TWX:
[L: ERROR] [O: c.t.p.p.BatchStreamEntryProcessor] [I: ] [U: SuperUser] [S: ] [T: pool-8-thread-10] Error executing batch
[L: WARN] [O: c.t.s.s.w.WSCommunicationsSubsystem] [I: ] [U: ] [S: ] [T: WsCommunication-151] Could not dispatch async service request [NotifyPropertyUpdate] :
Timed out APIRequestMessage [requestId: 1405707, endpointId: -1, sessionId: -1, method: POST, entityName: 172.18.142.102-temp540, characteristic: Services,
target:NotifyPropertyUpdate]
[L: WARN] [O: c.t.c.s.e.DispatchingServerEndpoint] [I: ] [U: ] [S: ] [T: http-nio-8080-exec-68] NULL MessageSynchronizationContext! Request either timed-out waiting
for this response, or it was received by mistake: ResponseMessage [requestId: 1401202, endpointId: 175200, sessionId: 1406274, code: STATUS_SUCCESS,
multipart: false, packet #: 0, total packets: 0]
+ QueryPropertyHistory for One Thing: NOT CONTINUOUS
temperature | timestamp |
98 | 2017-01-04 03:49:27.962 |
93 | 2017-01-04 03:49:01.006 |
91 | 2017-01-04 03:48:49.861 |
87 | 2017-01-04 03:48:26.911 |
86 | 2017-01-04 03:48:21.909 |
85 | 2017-01-04 03:48:16.907 |
79 | 2017-01-04 03:47:28.103 |
77 | 2017-01-04 03:47:17.009 |
74 | 2017-01-04 03:47:00.646 |
72 | 2017-01-04 03:46:46.981 |
70 | 2017-01-04 03:46:34.941 |
69 | 2017-01-04 03:46:29.372 |
66 | 2017-01-04 03:46:11.139 |
65 | 2017-01-04 03:46:05.810 |
63 | 2017-01-04 03:45:54.443 |
62 | 2017-01-04 03:45:48.696 |
61 | 2017-01-04 03:45:42.753 |
60 | 2017-01-04 03:45:37.495 |
58 | 2017-01-04 03:45:26.171 |
53 | 2017-01-04 03:44:57.037 |
50 | 2017-01-04 03:44:37.082 |
48 | 2017-01-04 03:44:26.089 |
46 | 2017-01-04 03:44:13.205 |
44 | 2017-01-04 03:44:03.188 |
42 | 2017-01-04 03:43:52.557 |
38 | 2017-01-04 03:43:29.961 |
33 | 2017-01-04 03:42:55.120 |
31 | 2017-01-04 03:42:44.548 |
27 | 2017-01-04 03:42:21.796 |
26 | 2017-01-04 03:42:12.657 |
25 | 2017-01-04 03:42:07.615 |
24 | 2017-01-04 03:42:02.288 |
21 | 2017-01-04 03:41:42.045 |
19 | 2017-01-04 03:41:30.359 |
18 | 2017-01-04 03:41:24.820 |
16 | 2017-01-04 03:41:13.063 |
15 | 2017-01-04 03:41:04.834 |
14 | 2017-01-04 03:40:59.453 |
13 | 2017-01-04 03:40:54.440 |
12 | 2017-01-04 03:40:49.361 |
11 | 2017-01-04 03:40:40.500 |
10 | 2017-01-04 03:40:28.502 |
9 | 2017-01-04 03:40:23.246 |
8 | 2017-01-04 03:40:15.742 |
7 | 2017-01-04 03:40:10.398 |
6 | 2017-01-04 03:40:04.640 |
5 | 2017-01-04 03:39:49.397 |
4 | 2017-01-04 03:39:38.949 |
3 | 2017-01-04 03:39:33.756 |
2 | 2017-01-04 03:39:28.749 |
1 | 2017-01-04 03:39:17.345 |

