Troubleshooting example of extremely slow server performance with low CPU consumption
In this particular scenario, the server is experiencing a severe performance drop.The first step to check first is the overall state of the server -- CPU consumption, memory, disk I/O. Not seeing anything unusual there, the second step is to check the Thingworx condition through the status tool available with the Tomcat manager.
Per the observation:
Despite 12 GB of memory being allocated, only 1 GB is in use.
Large number of threads currently running on the server is experiencing long run times (up to 35 minutes)
Checking Tomcat configuration didn't show any errors or potential causes of the problem, thus moving onto the second bullet, the threads need to be analyzed.
That thread has been running 200,936 milliseconds -- more than 3 minutes for an operation that should take less than a second. Also, it's noted that there were 93 busy threads running concurrently.
Concurrency on writing session variable values to the server.
The threads are kept alive and blocking the system.
Tracing the issue back to the piece of code in the service recently included in the application, the problem has been solved by adding an IF condition in order to perform Session variable values update only when needed.
In result, the update only happens once a shift.
Conclusion: Using Tomcat to view mashup generated threads helped identify the service involved in the root cause. Modification required to resolve was a small code change to reduce the frequency of the session variable update.