Thingworx 6.03 running on Tomcat 8.0.24 Oracle 1.8.0_51 JDK on Linux

[aleckie]

I have Thingworx 6.03 running on Tomcat 8.0.24 with Oracle 1.8.0_51 JDK on Linux - Tomcat silently catastrophically dies when the scheduled ThingWorx DB backup occurs. Has anyone tested the latest versions of Tomcat, Java, and ThingWorx? I followed the installation guidance with these framework versions, and when the TW backup process (Neo4j db file copies) complete in the Tomcat 8 Catalina.out log, the Tomcat process dies, no debug o/p. I can restart Tomcat, and it runs fine until the next Neo4j scheduled DB backup, and then dies silently again. Here's a snippet from the last Catalina.out log content before a crash...

2015-08-10 02:00:00.510+0000 INFO  [org.neo4j]: Copying index/lucene/node/system/_0.frq

2015-08-10 02:00:00.510+0000 INFO  [org.neo4j]: Copied  index/lucene/node/system/_0.frq 3.00 B

2015-08-10 02:00:00.510+0000 INFO  [org.neo4j]: Copying index/lucene-store.db

2015-08-10 02:00:00.510+0000 INFO  [org.neo4j]: Copied  index/lucene-store.db 40.00 B

2015-08-10 02:00:00.510+0000 INFO  [org.neo4j]: Done, copied 107 files

[aleckie]

Additionally, in the Tomcat Catalina log, I also noted that the ThingWorx 6.03 based Neo4j Java class complains that Java 8 is not supported, yet the install guide says it is...

2015-08-12 17:00:00.534+0000 INFO  [org.neo4j]: Copied  index/lucene/node/model/_0.fnm 98.00 B

2015-08-12 17:00:00.534+0000 INFO  [org.neo4j]: Copying index/lucene/node/model/_0.frq

2015-08-12 17:00:00.534+0000 INFO  [org.neo4j]: Copied  index/lucene/node/model/_0.frq 27.31 kB

2015-08-12 17:00:00.534+0000 INFO  [org.neo4j]: Copying index/lucene-store.db

2015-08-12 17:00:00.534+0000 INFO  [org.neo4j]: Copied  index/lucene-store.db 40.00 B

2015-08-12 17:00:00.534+0000 INFO  [org.neo4j]: Done, copied 39 files

2015-08-12 17:00:01.723+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: You are using an unsupported version of the Java runtime. Please use Oracle(R) Java(TM) Runtime Environment 7 or OpenJDK(TM) 7.

2015-08-12 17:00:02.121+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 208ms.

2015-08-12 17:00:02.122+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for 269ms.

Full consistency check

....................  10%

....................  20%

....................  30%

....................  40%

....................  50%

....................  60%

....................  70%

....................  80%

....................  90%

.................... 100%

[aleckie]

Ok, just had this in the log on the last ThingWorx scheduled backup as well (below). My guess is that Neo4j doesn't play nicely with the Java garbage collector in Java 1.8.0_51 even though it is set to the recommended "-XX:+UseConcMarkSweepGC" value...

2015-08-13 02:00:00.629+0000 INFO  [org.neo4j]: Copying index/lucene/node/model/_v.frq

2015-08-13 02:00:00.629+0000 INFO  [org.neo4j]: Copied  index/lucene/node/model/_v.frq 32.02 kB

2015-08-13 02:00:00.629+0000 INFO  [org.neo4j]: Copying index/lucene-store.db

2015-08-13 02:00:00.629+0000 INFO  [org.neo4j]: Copied  index/lucene-store.db 40.00 B

2015-08-13 02:00:00.629+0000 INFO  [org.neo4j]: Done, copied 107 files

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5e3016b000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)

#

# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.

# An error report file with more information is saved as:

# /tmp/hs_err_pid2494.log

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5e3036d000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)

[thread 140042513602304 also had an error]

[aleckie · Accepted solution]

Ok, well, I have tested the installation of these versions of the frameworks and ThingWorx 6.03 on Ubuntu 14.04 LTS server in a VMware VM and in an Amazon EC2 instance, which use the upstart services, and the problem does not seem to occur. Whereas it does on an Amazon AMI Linux instance. There's a lot of possible parameters to investigate, which should probably start with Tomcat running in debug mode, but I'm pretty sure the Java garbage collector is the main culprit in the latter build scenario. Knowing it now works with Ubuntu with these FW versions, and does work with Tomcat 7 and Java 7 in AMI Linux, I'm going to call it at this point, and flag this as 'self-answered'