logging no longer works after tomcat upgrade

[tcoufal]

Hi, we do have a yet another delightful problem with ThingWorx. We have updated Tomcat from 8.0 to 8.5. 

We have changed server config file as instructed and java properties in tomcat config utilities. 

We deployed ThingWorx into the new Tomcat (coppied war file, autodeploy true by default) 

Everything works fine except for the fact that platform is no longer logging anything into the log files. 

No script log, no application log no nothing. 

Anyone any help. 

I am very exhausted to debug such a software.... 

[PaiChung]

Any errors listed in the catalina.log (my guess is that application log is blank)

You've probably already check the permissions on the ThingworxStorage folder and its subfolders?

[tcoufal]

Catalina log (not complete):

20-Nov-2019 22:05:36.532 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
20-Nov-2019 22:05:36.563 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-jsse-nio-443"]
20-Nov-2019 22:05:36.563 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
20-Nov-2019 22:05:36.563 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 110408 ms
20-Nov-2019 22:07:03.115 INFO [https-jsse-nio-443-exec-102] org.apache.tomcat.util.http.parser.Cookie.logInvalidHeader A cookie header was received [255,255; color_1=192,192,192; color_2=128,128,128; color_3=0,0,0; color_4=191,48,48; color_5=249,239,107; color_6=102,204,51; color_7=51,153,255; color_8=27,60,132; color_9=153,102,153; JSESSIONID=EF920AD216E3C090BFD3A07235AF7A9F; PD-HOST=portal.skoda-auto.com; PD-NET=internet] that contained an invalid cookie. That cookie will be ignored.
Note: further occurrences of this error will be logged at DEBUG level.
log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Tomcat runs under the same user as previous version of tomcat so I guess that permission is not an issue. 

Plus - Export, IMport, Repository all works, if permission would be the issue none of that would work as well right? Repository is also under ThingworxStorage. ESAPI file was also created after Tomcat started.  

[slangley]

Hi @tcoufal.

Which version of ThingWorx are you running and which exact version of Tomcat did you upgrade to?  What platform are you running on?  In Composer, do you see logs under Monitoring?

I am happy to open a case on your behalf in order to work with a Support engineer on a solution.

Regards.

--Sharon

[tcoufal]

ThingWorx 7.3.8

Old Tomcat that works 8.0.46

New Tomcat 8.5.47

Java 1.8_144

Windows Server 2012 R2

Tomcat was installed, windows service name was changed (because old Tomcat is still present).

Tomcat8.5w.exe was started, Java options were added as per installation manual for TW 7.3, + added link to java KeyStore and TrustStore.

ThingWorx war file was placed into the Webapp directory. 

Tomcat was started, application deploy successful, Platform running fine, No Log files in ThingworxStorage/logs. 

[slangley]

Hi @

It appears the issue may be the result of upgrading to a version of Tomcat that is not supported for the version of Tomcat you are running.  Here is a link to the system requirements for ThingWorx 7.3.

We strongly recommend that you upgrade to one of the more recent versions.  There have been many security and feature improvements since 7.3.

Regards.

--Sharon

[tcoufal]

Hi Slangly,

Documentation says nothing about unsupported tomcat version or maximum supported version. 

Only says tested version and under the comments that minimum version of 8.0.33 is required. 

Tomcat version 8.5 (branch) is not considered to be a major but rather minor version. 

Apache docu:

This is the first stable release of the 8.5.x branch. Tomcat 8.x users
should now use 8.5.x releases in preference to 8.0.x releases.

Your docu:

In general, you may install later minor versions of the software

Documentation is quite ambiguous.

We are trying (unsuccessfully) to upgrade our Production instance for a while now. 

Bunch of errors happened at the first time (some of them based on inconsistency in the official materials). 

And dont get me started on SQL files for PostgreSQL. 

Are you really using pg_sleep in function exception? I have not seen this for 10 years. 

Copying the whole value_stream table after entry_id type change during update procedure? Again, no mention that it will  cause your DB instance to grow by the same amount...

I could go like this for a quite a while. 

I am working with the platform since 2015 and I am finding these small stuff to be very irritating. 

Please do not take this personally, I have nothing against you and I am glad that you have sort of pointed me at the right direction, but it should not be your job to do it, it should be the documentation's. 

Please PTC make the docu as unambiguous as humanly possible. 

Lot of CS Articles are in direct contradiction to official documentation. 

Tomas 

[slangley]

Hi @tcoufal.

We appreciate your feedback regarding documentation issues and take your comments seriously.  I will be passing your comments along to our tech writers for consideration in future releases.

If my response helped to resolve your issue, marking it as the Accepted Solution would be appreciated, as it will be helpful to others in the community experiencing similar issues.

Regards.

--Sharon

[tcoufal]

Hi slangly, 

Issue have not yet been resolved. We are facing the same issue even after successful upgrade to 8.4.5. 

Support case has been created. I will let you all know.... Thanks

[slangley]

Hi @tcoufal.

Please provide the case number that was opened for this issue.  The only one I found that is similar is already closed indicating the issue was resolved.

Regards.

--Sharon

[tcoufal · Accepted solution]

Hi there,

sorry for delayed answer. It was very hectic at the end of the year. 

So..

After some back and forth it turned out that it was indeed a permission issue after all. 

For some reason Tomcat 8 install it self as (rather "under account of") local system, but 8.5 as local service. Difference is that local service has R/W permissions (that is why Import/Export in/from ThingworxStorage/repository worked, as well validation.properties file has been created after startup), but local system has full access permissions. I dont why that is necessary, but it is. 

We changed under which account Tomcat has to run and now everything works. 

It was not obvious because there were no errors in tomcat log files.

[tcoufal]

Those three last lines in stderr.log are of no help. 

I used old Tomcat, logging works fine, but those three lines are still in the log file (at the end).

log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Tomcat runs as windows service in both cases. ThingworxStorage has full access for SYSTEM.

What is the difference?