Tomcat using 100% of CPU Memory

[pshashipreetham]

When checked in the Application Log, there is following error:

[L: ERROR] [O: c.t.s.s.w.p.WSExecutionInstance] [I: ] [U: Administrator] [S: ] [P: ] [T: WSExecutionProcessor-479] error executing APIRequest Message: Unable to dispatch [ uri = /Things/RemoteThingName/Services/UpdateSubscribedPropertyValues/]: Unable to Invoke Service UpdateSubscribedPropertyValues on RemoteThingName: java.lang.Exception: Invalid Property Name: [Property1], sending ERROR ResponseMessage to caller!

@c_lowy @paic @slangley @CarlesColl 

Thanks,
Shashi

[PaiChung]

Is your system using 100% CPU or is Tomcat using 100% Memory in the JVM?

Unless you have a very low amount of memory on your server, it will be more likely to hit 100% CPU, which would probably mean that you are spawning threads faster than that ThingWorx processes them.

For example your subscriptions fire every second, but your subscriptions actually take longer than 1 second to be processed so that the next set of subscriptions fire off and start to stack up.

[pshashipreetham]

Hi @PaiChung ,

how can I know Tomcat is using 100% Memory in the JVM ?

Thanks,
Shashi.

[PaiChung]

Not sure if you set a JVM Memory max limit, but that would show in the server memory usage.

Also you can look within ThingWorx by going to Monitoring/System Statistics

You'll also see an Event Queue there, if that is continuously going up or staying very high would indicate that your are 'overloading' the processor(s)

[pshashipreetham]

Hi @PaiChung ,

is this the issue ?

Thanks,
Shashi.

[PaiChung]

It looks like you are indeed running out of memory. 

Easiest would be to increase your server memory and JVM Max memory setting.

It will require a restart.

[pshashipreetham]

Hi @PaiChung ,

the same configuration is been using for an year now, suddenly this issue came up from last one week, may be there is some other root cause for this ?

Thanks,
Shashi.

[PaiChung]

I do not know what may have caused it sorry.

[VladimirRosu]

@pshashipreetham 

In order to diagnose such events you would need to have proper monitoring set in place. While PTC System Monitor was the solution I would have suggested in the past, today I would say Dynatrace, fully web based and with a good partnership with us are a very good choice.

At the end of the day, any monitoring solution would fit the need, but I must say that Dynatrace has a very nice feature that allows you to see exactly which Java classes consumed CPU execution time at a certain moment in time, making in turn very easy to understand what's consuming your CPU time (which service etc.)

But, to make this short, we also have plenty documentation about monitoring, and I suggest reading this article https://www.ptc.com/en/support/article/CS330379

As Pai said above, we will not be able to tell you what's the root cause - you need to investigate this on your own.

[pshashipreetham]

@VladimirRosu ,

Is there any article, where Dynatrace and Thingworx integration is described, I googled but, not able to find the proper one.

Thanks,
Shashi.

[slangley]

Hi @psp316r

Have you been able to find a solution to your performance related issues?  If not, I recommend opening a case.  I will be happy to open one on your behalf if you agree but you will need to private message me your PTC Support email address to do so.  A current support agreement will be required.

Regards.

--Sharon

[pshashipreetham · Accepted solution]

Got the issue,

So there are a few issues that got noticed.

1. There was a Software Antimalware Service Executable running 10+ Postgres Service Instances in the background - I googled and got rid of the software.
2. A few Developer-created test Schedulers and Timers were creating issues; got rid of them too.

Thanks,

Shashi.