cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Community Tip - Did you get an answer that solved your problem? Please mark it as an Accepted Solution so others with the same problem can find the answer easily. X

Thingworx HA Primary server Tomcat stopped(Tomcat version 9.0.35)

vi1
15-Moonstone
15-Moonstone

Thingworx HA Primary server Tomcat stopped(Tomcat version 9.0.35)

Hi,

 

I have configured Thingworx 8.5.1 with HA Configuration.

Observed primary Tomcat was stopped and pointed to secondary server. I did not get the reason why Tomcat stopped. Also, verified system is not rebooted during that time. I have observed below errors on "tomcat9-stdout" file. Is Tomcat stopped because of below errors? Do I need to verify any other files in Tomcat logs?

 

  2021-03-18 17:35:58 WARN IntrusionDetector:65 - [SECURITY FAILURE Anonymous:null@unknown -> /ExampleApplication/IntrusionDetector] Invalid input: context=HTTP header value: cf-visitor, type(HTTPHeaderValue)=^[a-zA-Z0-9()\-=\*\.\?;,+\/:&_ %$ ¡-ï¿¿'"]*$, input={"scheme":"https"}
org.owasp.esapi.errors.ValidationException: HTTP header value: cf-visitor: Invalid input. Please conform to regex ^[a-zA-Z0-9()\-=\*\.\?;,+\/:&_ %$ ¡-￿'"]*$ with a maximum length of 2000
at org.owasp.esapi.reference.validation.StringValidationRule.checkWhitelist(StringValidationRule.java:144)
at org.owasp.esapi.reference.validation.StringValidationRule.checkWhitelist(StringValidationRule.java:160)
at org.owasp.esapi.reference.validation.StringValidationRule.getValid(StringValidationRule.java:284)
at com.thingworx.security.filter.ESAPICustomValidator.getValidInput(ESAPICustomValidator.java:29)
at com.thingworx.security.filter.ValidatingHttpRequest.getValidInput(ValidatingHttpRequest.java:127)
at com.thingworx.security.filter.ValidatingHttpRequest.getValidHeaderInput(ValidatingHttpRequest.java:143)
at com.thingworx.security.filter.ValidatingHttpRequest.getHeader(ValidatingHttpRequest.java:85)
at com.thingworx.webservices.context.HttpExecutionContext.<init>(HttpExecutionContext.java:227)
at com.thingworx.webservices.BaseService.service(BaseService.java:310)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)

 

Thanks&Regards,

Latha

1 ACCEPTED SOLUTION

Accepted Solutions
dmansfeld
10-Marble
(To:vi1)

Hi vi1!

 

Please check your Zookeeper instances if they are running. You can just run ./zkServer.sh status

 

 

View solution in original post

5 REPLIES 5
slangley
23-Emerald II
(To:vi1)

Hi @vi1.

 

Have you taken a look at the ThingWorx logs located at \ThingworxStorage\logs?  This would be the best place to start in troubleshooting the issue.  If it does appear to be a Tomcat issue, you could also check the tomcat9-stderr log.  The warning you found in the stdout log would not have caused the failure.

 

If you can provide the logs, we'll be happy to take a look.  A case may be needed, however.

 

Regards.

 

--Sharon

vi1
15-Moonstone
15-Moonstone
(To:slangley)

Hello ,

Thank you for response.

 

I have observed below error on Thingworx Application Logs. Please find the attached "tomcat9-stderr" file.

 

2021-03-18 17:44:36.345-0700 [L: INFO] [O: c.t.c.s.c.j.Jsr356ServerConnection] [I: ] [U: ] [S: ] [P: ] [T: Thread-1147] Server WebSocket closing, unregistering from Server Endpoint [ws session id: 0, endpoint id: 1, endpoint name: null] CloseReason: code [1001], reason [The web application is stopping]
2021-03-18 17:44:36.345-0700 [L: INFO] [O: c.t.c.c.e.CommunicationEndpoint] [I: ] [U: SuperUser] [S: ] [P: ] [T: Thread-1147] All Connections are closed [endpoint id: 1]
2021-03-18 17:44:36.345-0700 [L: INFO] [O: c.t.c.c.e.CommunicationEndpoint] [I: ] [U: SuperUser] [S: ] [P: ] [T: Thread-1147] Notifying CommunicationEndpoint Container to remove Endpoint [id: 1]
2021-03-18 17:44:36.345-0700 [L: INFO] [O: c.t.c.p.m.DispatchingServerCommunicationModule] [I: ] [U: SuperUser] [S: ] [P: ] [T: Thread-1147] Dynamically de-activing observer [name: *XXXThingName]
2021-03-18 17:44:36.345-0700 [L: INFO] [O: c.t.t.c.RemoteThing] [I: ] [U: SuperUser] [S: ] [P: ] [T: Thread-1147] Communication binding is DE-ACTIVATED.........

 

When I see system logs in Event Viewer, I got this message "The Apache Tomcat 9.0 Tomcat9 service entered the stopped state at "2021-03-18 5:44:36PM". Before that observed "The IaasVmProvider service entered the stopped state at 5:40:21PM" in Event Viewer log.

 

Below are "omcat9-stderr" logs..

 

18-Mar-2021 17:44:35.917 INFO [Thread-1147] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-80"]
18-Mar-2021 17:44:35.960 INFO [Thread-1147] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-jsse-nio-443"]
18-Mar-2021 17:44:35.999 INFO [Thread-1147] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
2021-03-18 17:44:37 Apache Commons Daemon procrun stderr initialized.
18-Mar-2021 17:44:37.915 WARNING [main] org.apache.tomcat.util.net.SSLHostConfig.setProtocols The protocol [TLSv1.1] was added to the list of protocols on the SSLHostConfig named [_default_]. Check if a +/- prefix is missing.
18-Mar-2021 17:44:37.918 WARNING [main] org.apache.tomcat.util.net.SSLHostConfig.setProtocols The protocol [TLSv1.2] was added to the list of protocols on the SSLHostConfig named [_default_]. Check if a +/- prefix is missing.
18-Mar-2021 17:44:38.021 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/9.0.35
18-Mar-2021 17:44:38.021 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: May 5 2020 20:36:20 UTC
18-Mar-2021 17:44:38.021 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.35.0
18-Mar-2021 17:44:38.021 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Windows Server 2012 R2
18-Mar-2021 17:44:38.021 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 6.3
18-Mar-2021 17:44:38.022 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
18-Mar-2021 17:44:38.022 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: C:\Program Files\Java\jre1.8.0_281
18-Mar-2021 17:44:38.022 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_281-b09
18-Mar-2021 17:44:38.022 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation
18-Mar-2021 17:44:38.022 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: C:\Program Files\Apache Software Foundation\Tomcat 9.0
18-Mar-2021 17:44:38.022 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: C:\Program Files\Apache Software Foundation\Tomcat 9.0
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 9.0
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 9.0
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 9.0\temp
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 9.0\conf\logging.properties
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dserver -Dd64
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseG1GC
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfile.encoding=UTF-8
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.library.path=C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\Thingworx\WEB-INF\extensions
18-Mar-2021 17:44:38.023 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms2g
18-Mar-2021 17:44:38.024 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx16g
18-Mar-2021 17:44:38.024 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: exit
18-Mar-2021 17:44:38.024 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: abort
18-Mar-2021 17:44:38.024 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\Thingworx\WEB-INF\extensions]

 

Regards,

Latha

vi1
15-Moonstone
15-Moonstone
(To:vi1)

Forgot to attach tomcat log. Please find the attached log file.

dmansfeld
10-Marble
(To:vi1)

Hi vi1!

 

Please check your Zookeeper instances if they are running. You can just run ./zkServer.sh status

 

 

slangley
23-Emerald II
(To:vi1)

Hi @vi1.

 

Have you resolved your issue?  If so, please mark the response that lead you to the solution or provide details of the solution you found.

 

Regards.

 

--Sharon

Top Tags