Thingworx node cannot log to Ignite and Ignite have ConsistentId null

[pStrobl]

Greetings!

I'm installing ThingWorx Platform 9.1 in clustered (active-active) mode and I can't make it work. My Ignite 2.8.0 service is running and also Zookeeper 3.5.7. But when I try to run Tomcat 9.0.41 it loads and after a few seconds it crash. As you can see in the provided log problem is in ConsistentId null. I don't know why ThingWorx cannot connect to Ignite through Zookeeper. All of the instances are running on one server. However it wasn't working even on dedicated servers. The system I'm running is fully upgraded Ubuntu 20.04 and Java is OpenJDK Runtime Environment Corretto-11.0.9.12.1. I also run Zookeeper, Ignite and Tomcat as services. I exported IGNITE_HOME and IGNITE_WORK into /etc/environment for sure. Tcpdump shows no activity after starting tomcat so I assume that Thingworx is not connected even to Zookeeper. I even try to set -J-DIGNITE_JETTY_PORT=9999 and admin.enableServer=false so port 8080 is free for Tomcat but it didnt help. Could you help me with that please? Thank you.

Guides I used:
PTC guides
https://ignite.apache.org/docs/latest/installation/installing-using-zip

---------

Tomcat log

2020-12-10 11:08:29.294+0000 [L: INFO] [O: c.t.c.i.IgniteCacheProvider] [I: ] [U: ] [S: ] [P: twnode0] [T: ForkJoinPool.commonPool-worker-3] Starting Ignite ConsistentId null with configuration: IgniteConfiguration [igniteInstanceName=twx-core-server, pubPoolSize=8, svcPoolSize=null, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=3, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, sqlQryHistSize=1000, dfltQryTimeout=0, igniteHome=null, igniteWorkDir=null, mbeanSrv=null, nodeId=null, marsh=null, marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, soLinger=5, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=null, evtSpi=null, colSpi=null, deploySpi=null, indexingSpi=null, addrRslvr=null, encryptionSpi=null, clientMode=true, rebalanceThreadPoolSize=4, rebalanceTimeout=10000, rebalanceBatchesPrefetchCnt=3, rebalanceThrottle=0, rebalanceBatchSize=524288, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, deadlockTimeout=10000, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=30000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=ConnectorConfiguration [jettyPath=null, host=null, port=11211, noDelay=true, directBuf=false, sndBufSize=32768, rcvBufSize=32768, idleQryCurTimeout=600000, idleQryCurCheckFreq=60000, sndQueueLimit=0, selectorCnt=3, idleTimeout=7000, sslEnabled=false, sslClientAuth=false, sslCtxFactory=null, sslFactory=null, portRange=100, threadPoolSize=8, msgInterceptor=null], odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=REPLICATED, backups=0, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=null, activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, handshakeTimeout=10000, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null, thinCliCfg=ThinClientConfiguration [maxActiveTxPerConn=100]], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=null, commFailureRslvr=null]
2020-12-10 11:08:29.373+0000 [L: ERROR] [O: c.t.s.ThingWorxBootstrapper] [I: ] [U: SuperUser] [S: ] [P: twnode0] [T: main] *** CRITICAL ERROR ON STARTUP: Failed to start CacheProvider com.thingworx.cache.ignite.IgniteCacheProvider
2020-12-10 11:08:29.375+0000 [L: ERROR] [O: c.t.s.ThingWorxBootstrapper] [I: ] [U: SuperUser] [S: ] [P: twnode0] [T: main] *** Web Application STATE is being set to ERROR! ***
2020-12-10 11:08:29.377+0000 [L: INFO] [O: c.t.s.ThingWorxServer] [I: ] [U: SuperUser] [S: ] [P: twnode0] [T: main] >>>>>>> PLATFORM SHUTDOWN START <<<<<<<<<

---------

[slangley · Accepted solution]

Hi @pStrobl.

Please check out this article to see if it helps.  If not, you may have to wait for the 9.1.1 release as there is a fix coming for an incompatibility between OpenJDK and Ignite.  This release is targeted for January.

Regards.

--Sharon

[pStrobl]

Thank you,

I also found this article but unfortunately it didn't work for me... Sorta. I set IGNITE_WORK=/opt/ingite and then Tomcat refused to start. But when I set IGNITE_WORK_DIR="/opt/ignite" then it starts working. I debugged this about for 10 days... But it's working now  and I'm happy with that. Thank you for responding.

[slangley]

Hi @pStrobl.

Can you point out the specifics from the article that did not work for you?  It sounds like it covers your scenario but perhaps I'm missing something.

Regards.

--Sharon

[pStrobl]

Hi @slangley!

Of course I can. Sorry for late response. 

• If Tomcat runs as a service add following line to /etc/systemd/system/tomcat9.service
  • Environment=IGNITE_WORK_DIR=/ignite/work

In my case I need add this line to the service Environment=IGNITE_WORK_DIR="/ignite/work". I don't know why was that but then ThingWorx started.

[slangley]

Hi @pStrobl.

It appears the article was updated with that requirement.  Are you still having any issues?

Note, that the ThingWorx 9.1.1 was delayed but it has now been released and is available for download.

Regards.

--Sharon

[pStrobl]

No, for now I don't have any issues. Thank you for your help

[slangley]

Hi @pStrobl.

Glad to hear you're no longer having any issues.

If you feel this thread was helpful in resolving your problems, please mark the appropriate response as the Accepted Solution for the benefit of others with the same issue.

Regards.

--Sharon