ThingWorx 9.1.0 Tomcat 100% CPU Usage

[Gatulz]

My thingworx server(deployed in a virtual machine in Azure) become very slow suddenly because Thingworx apace tomcat high CPU Usage (until 100% in VM). This issue happens several times before and could be solved by restart the Thingworx tomcat services, but until now I didn't know the main reason why this issue happens. I have check thingworx application log, script log until the apache tomcat log but I'm still not found anything about the reason why the CPU suddenly increased. The CPU usage increased even when there is no user log in at that time (if user log in should be appeared in script log because of the logger that I put on the services). I don't have any subscription/scheduler/timer except for one scheduler that will be excecute daily at 00.00 AM, and the issue could happen any time, some time at 5PM, 6AM etc. the period of this issue id also random, sometime it happen 12 days, 7days, 20 days after I restart the Thingworx Apache tomcat. 

Below was some details on my thingworx server

===Versions===

Thingworx 9.1.0b + Postgresql 11.x + Windows server 2019

Java : OpenJDK - Amazon Corretto 11.0.10.9.1

========tomcat config======

-Dserver
-Dd64
-Djava.net.preferIPv4Stack=true
-Djava.library.path=C:\PROGRA~2\ThingWorxFoundation\tomcat\current\webapps\Thingworx\WEB-INF\extensions
-Dfile.encoding=UTF-8
-XX:+UseNUMA
-XX:HeapDumpPath=C:\PROGRA~2\ThingWorxFoundation\tomcat\current\logs
-Dcatalina.base=C:\PROGRA~2\ThingWorxFoundation\tomcat\current
-Dcatalina.home=C:\PROGRA~2\ThingWorxFoundation\tomcat\current
-Djava.io.tmpdir=C:\PROGRA~2\ThingWorxFoundation\tomcat\current\temp
-Djdk.tls.ephemeralDHKeySize=2048
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources
-Duser.timezone=UTC
-Xlog:gc:file=C:\PROGRA~2\ThingWorxFoundation\tomcat\current\logs\tomcat-twx-gc.log:time,level,tags
-Xms2500m
-Xmx3000m

Any idea about it ? 

Regards,

Gatulz

[PaiChung]

I would do a few things.

Monitor the memory heap and when it does GC and if that is when you see a spike or not.

You can also turn on system statistics and see what services run how often and how long.

[Tudor · Accepted solution]

For this type of performance issue, application logging is of limited value: by definition, several processes are running on the CPU in the background and these services (since they're running) will not report errors or print other information.

We need runtime diagnostics for this type of issue to identify which processes are involved (memory? events? timers?)

In our documentation we have outlined some troubleshooting methods for this type of scenario: http://support.ptc.com/help/thingworx_hc/thingworx_8_hc/en/#page/ThingWorx%2FHelp%2FBest_Practices_for_Developing_Applications%2Fmonitoring_thingworx_applications.html%23

Additionally, we have a timer utility that will help us capture runtime diagnostics if we have an event or timer issue that's driving this behavior: https://www.ptc.com/en/support/article/CS347564

In short:

* Ensure GC logging is enabled (http://support.ptc.com/help/thingworx_hc/thingworx_8_hc/en/#page/ThingWorx%2FHelp%2FBest_Practices_for_Developing_Applications%2Fmonitoring_tools.html%23wwID0E3NY3) or consider using a utility like VisualVM to track memory performance over time

* Consider a tool like VisualVM to see which processes are actively using the CPU and collecting a thread dump at the appropriate time

* Review subsystem metrics to see if there are any event processing spikes (metrics are printed in the logs, or can be viewed with the timer above)

* If you do have the timer installed, it will collect thread dumps as performance issues occur (or these can be collected manually before a restart)

We're happy to review any diagnostic data collected if you want to attach it here or to a support case.

[slangley]

Hi @Gatulz.

If you feel your question has been answered, please mark the appropriate response as the Accepted Solution for the benefit of others with the same question.

Regards.

--Sharon