Solved: Thingworx HA Primary server Tomcat stopped(Tomcat ...

vi1 · ‎Mar 19, 2021

Hi,

I have configured Thingworx 8.5.1 with HA Configuration.

Observed primary Tomcat was stopped and pointed to secondary server. I did not get the reason why Tomcat stopped. Also, verified system is not rebooted during that time. I have observed below errors on "tomcat9-stdout" file. Is Tomcat stopped because of below errors? Do I need to verify any other files in Tomcat logs?

2021-03-18 17:35:58 WARN IntrusionDetector:65 - [SECURITY FAILURE Anonymous:null@unknown -> /ExampleApplication/IntrusionDetector] Invalid input: context=HTTP header value: cf-visitor, type(HTTPHeaderValue)=^[a-zA-Z0-9()\-=\*\.\?;,+\/:&_ %$ Â¡-ï¿¿'"]*$, input={"scheme":"https"}
org.owasp.esapi.errors.ValidationException: HTTP header value: cf-visitor: Invalid input. Please conform to regex ^[a-zA-Z0-9()\-=\*\.\?;,+\/:&_ %$ Â¡-ï¿¿'"]*$ with a maximum length of 2000
at org.owasp.esapi.reference.validation.StringValidationRule.checkWhitelist(StringValidationRule.java:144)
at org.owasp.esapi.reference.validation.StringValidationRule.checkWhitelist(StringValidationRule.java:160)
at org.owasp.esapi.reference.validation.StringValidationRule.getValid(StringValidationRule.java:284)
at com.thingworx.security.filter.ESAPICustomValidator.getValidInput(ESAPICustomValidator.java:29)
at com.thingworx.security.filter.ValidatingHttpRequest.getValidInput(ValidatingHttpRequest.java:127)
at com.thingworx.security.filter.ValidatingHttpRequest.getValidHeaderInput(ValidatingHttpRequest.java:143)
at com.thingworx.security.filter.ValidatingHttpRequest.getHeader(ValidatingHttpRequest.java:85)
at com.thingworx.webservices.context.HttpExecutionContext.<init>(HttpExecutionContext.java:227)
at com.thingworx.webservices.BaseService.service(BaseService.java:310)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)

Thanks&Regards,

Latha

dmansfeld · ‎Mar 22, 2021

Hi vi1!

Please check your Zookeeper instances if they are running. You can just run ./zkServer.sh status

View solution in original post

slangley · ‎Mar 19, 2021

Hi @vi1.

Have you taken a look at the ThingWorx logs located at \ThingworxStorage\logs? This would be the best place to start in troubleshooting the issue. If it does appear to be a Tomcat issue, you could also check the tomcat9-stderr log. The warning you found in the stdout log would not have caused the failure.

If you can provide the logs, we'll be happy to take a look. A case may be needed, however.

Regards.

--Sharon

vi1 · ‎Mar 19, 2021

Hello ,

Thank you for response.

I have observed below error on Thingworx Application Logs. Please find the attached "tomcat9-stderr" file.

2021-03-18 17:44:36.345-0700 [L: INFO] [O: c.t.c.s.c.j.Jsr356ServerConnection] [I: ] [U: ] [S: ] [P: ] [T: Thread-1147] Server WebSocket closing, unregistering from Server Endpoint [ws session id: 0, endpoint id: 1, endpoint name: null] CloseReason: code [1001], reason [The web application is stopping]
2021-03-18 17:44:36.345-0700 [L: INFO] [O: c.t.c.c.e.CommunicationEndpoint] [I: ] [U: SuperUser] [S: ] [P: ] [T: Thread-1147] All Connections are closed [endpoint id: 1]
2021-03-18 17:44:36.345-0700 [L: INFO] [O: c.t.c.c.e.CommunicationEndpoint] [I: ] [U: SuperUser] [S: ] [P: ] [T: Thread-1147] Notifying CommunicationEndpoint Container to remove Endpoint [id: 1]
2021-03-18 17:44:36.345-0700 [L: INFO] [O: c.t.c.p.m.DispatchingServerCommunicationModule] [I: ] [U: SuperUser] [S: ] [P: ] [T: Thread-1147] Dynamically de-activing observer [name: *XXXThingName]
2021-03-18 17:44:36.345-0700 [L: INFO] [O: c.t.t.c.RemoteThing] [I: ] [U: SuperUser] [S: ] [P: ] [T: Thread-1147] Communication binding is DE-ACTIVATED.........

When I see system logs in Event Viewer, I got this message "The Apache Tomcat 9.0 Tomcat9 service entered the stopped state at "2021-03-18 5:44:36PM". Before that observed "The IaasVmProvider service entered the stopped state at 5:40:21PM" in Event Viewer log.

Below are "omcat9-stderr" logs..

18-Mar-2021 17:44:35.917 INFO [Thread-1147] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-80"]
18-Mar-2021 17:44:35.960 INFO [Thread-1147] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-jsse-nio-443"]
18-Mar-2021 17:44:35.999 INFO [Thread-1147] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
2021-03-18 17:44:37 Apache Commons Daemon procrun stderr initialized.
18-Mar-2021 17:44:37.915 WARNING [main] org.apache.tomcat.util.net.SSLHostConfig.setProtocols The protocol [TLSv1.1] was added to the list of protocols on the SSLHostConfig named [_default_]. Check if a +/- prefix is missing.
18-Mar-2021 17:44:37.918 WARNING [main] org.apache.tomcat.util.net.SSLHostConfig.setProtocols The protocol [TLSv1.2] was added to the list of protocols on the SSLHostConfig named [_default_]. Check if a +/- prefix is missing.
18-Mar-2021 17:44:38.021 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/9.0.35
18-Mar-2021 17:44:38.021 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: May 5 2020 20:36:20 UTC
18-Mar-2021 17:44:38.021 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.35.0
18-Mar-2021 17:44:38.021 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Windows Server 2012 R2
18-Mar-2021 17:44:38.021 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 6.3
18-Mar-2021 17:44:38.022 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
18-Mar-2021 17:44:38.022 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: C:\Program Files\Java\jre1.8.0_281
18-Mar-2021 17:44:38.022 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_281-b09
18-Mar-2021 17:44:38.022 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation
18-Mar-2021 17:44:38.022 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: C:\Program Files\Apache Software Foundation\Tomcat 9.0
18-Mar-2021 17:44:38.022 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: C:\Program Files\Apache Software Foundation\Tomcat 9.0
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 9.0
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 9.0
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 9.0\temp
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 9.0\conf\logging.properties
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dserver -Dd64
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseG1GC
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfile.encoding=UTF-8
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.library.path=C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\Thingworx\WEB-INF\extensions
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms2g
18-Mar-2021 17:44:38.024 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx16g
18-Mar-2021 17:44:38.024 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: exit
18-Mar-2021 17:44:38.024 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: abort
18-Mar-2021 17:44:38.024 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\Thingworx\WEB-INF\extensions]

Regards,