~1000 processes java <defunct> when starting tomcat8 with Thingworx.war 7.4.9

[pjoly]

hello

i have installed a new server with Thingworx 7.4.9-B47 on ubuntu 14 LTS using postgres (following the installation procedures)

It works but it creates ~1000 processes for  tomcat8 which are not killed . I don't know where to search for root cause.

tomcat8 24113 21113 0 09:41 ? 00:00:00 [java] <defunct>

I did not noticed this issue with previous relases (7.3)

I see nothing in catalina.out :

15-Mar-2018 09:42:31.844 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/share/tomcat8/8.0.33/webapps/docs has finished in 28 ms
15-Mar-2018 09:42:31.858 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
15-Mar-2018 09:42:31.873 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-443"]
15-Mar-2018 09:42:31.877 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 100808 ms

and nithing in application.log for Thingworx except hourly warnings for metrics :

15-Mar-2018 09:42:31.844 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/share/tomcat8/8.0.33/webapps/docs has finished in 28 ms
15-Mar-2018 09:42:31.858 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
15-Mar-2018 09:42:31.873 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-443"]
15-Mar-2018 09:42:31.877 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 100808 ms

[cdovholuk]

A 1000 threads would be expected but not 1000 processes on the machine. If you take a thread dump using jstack and the tomcat pid you should see those threads are all of the same type.  Can you explain more what you mean by 'not killed'? Do you mean you use tomcat's "shutdown.sh" and the tomcat process never exits?  

Could you post your exact steps?

[pjoly]

Thanks for answer

I'm not a specialist but the command "ps -edf" is showing these threads (or processes)

I said "not killed" because of the status "defunct" I guess they should be killed (something like zombie processes)

I was warned by my monitoring system (zabbix) by "too many processes runing on server THX" so I detect these "1000 processes"

I have another server for Thingworx (but in 7.3.2) with same release of OS, tomcat8 & Postgres (and ~ the same application) and I don't have this issue

Could it be configuration parameter for the platform ?

I kept the default file when Installing (platform-settings.json ) just adjusting the password for Postgres)

[cdovholuk]

To be honest - I'm not sure what's happening or why. I would not expect ps to show you 1000 processes whatsoever. It seems like something is spawning a child process - an additional extension you imported or you wrote? You may need to create an actual support ticket to get better help here. :(

[pjoly]

Hello

Thanks for advices

I found the issue by corellation of the issue date & modifications don :

I put a value stream itself  for the value stream field which probably created the issue.

The problem is that I have this error in the application log for a value stream

Thing [M2WineValueStream] has logged properties but does not have a valid value stream assigned

Do we have to fill the field "value stream "for a value stream ?

what to put there ?

[cdovholuk]

I've never tried to add a property to a Thing of type ValueStream... Usually you will make a Thing and add a ValueStream to that thing (or thing template). I actually don't know what happens when you add a logged property to a Thing of type "ValueStream"... That's definitely interesting. If I get a chance to try that out I might.

[pjoly]

Hello

I got the  same issue again (850 java <defunct> processes) , just while modifying a mashup 

and I lost all maps in mashup (google & openstreet widget)

I'll try to contact support