cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Thingworx and current Chrome flooding the Application log

atondorf
12-Amethyst

Thingworx and current Chrome flooding the Application log

Hi Everybody,

 

I have an issue with thingworx and the current chrome version. I get a lot or Errors in Application Log:

Error occurred while validating HTTP header: sec-ch-ua. HTTP header value: sec-ch-ua: Invalid input. Please conform to regex ^[0-9a-zA-Z()\-=\*\.\?;,+\/:&_ %$ ¡-￿'"]*$ with a maximum length of 2000 

So i did following steps:

  • Check where Thingworx stores the regex and just changed the order a little bit to [0-9a-zA-Z ...
    So I was able to be sure which Rexeg Thingworx uses for the sec-ch-ua header.
    It is in file ThingworxStorage\esapi\validation.properties:
    Validator.HTTPHeaderValue=^[0-9a-zA-Z()\\-=\\*\\.\\?;,+\\/:&_ %\$ \u00A1-\uFFFF'"]*$
  • Check the content of the sec-ch-us of Chrome. It's: 
    "Chromium";v="86", "\"Not\\A;Brand";v="99", "Google Chrome";v="86"
  • Ok there is an issue with the string of chrome! The backslaches are not allowed by the regex ... 
  • So I tried to adopt the Regex and add support for '\' but even with an all allowing Regex "^.*$" I get Error:
    • "Error occurred while validating HTTP header: sec-ch-ua. Input validation failur"
    • So the error does not display the regex anymore, but still present!

Anyone has an idea?

 

 

 

10 REPLIES 10
slangley
23-Emerald I
(To:atondorf)

Hi @atondorf.

 

Which version of ThingWorx are you running?  Have you recently upgraded the environment?  If so did you remove validation.properties so it gets recreated under the new version?

 

If you've made any modifications to validation.properties, you'll need to save the original in order to merge your changes back in after the file is recreated. 

 

You also may need to double escape the backslash: \\\\\

 

Please let us know if you continue to have issues.

 

Regards.

 

--Sharon

atondorf
12-Amethyst
(To:slangley)

Hi,

 

I have this behaviour on TWX 8.4. We also have a 9.0 instance for testing. But for some reason Chrome does not send the sec-ch-ua header to this instance (Maybe because it's not https).

So i tried it now with several approaches to add the '\' support to the Regex:

- Original RexEx String:

^[a-zA-Z0-9()\\-=\\*\\.\\?;,+\\/:&_ %\$ \u00A1-\uFFFF'"]*$:      Get the error " Please conform to regex ... "

- Modified with '\' Support. I tried it with 3-6 of the '\' in the string. Always with the same result:

^[a-zA-Z0-9()\\-=\\*\\.\\?;,+\\/:&_ %\$ \u00A1-\uFFFF'"\\\]*$    Get the error " Input validation failure"

 

Is it possible to add an special Regex to handle the sec-ch-ua Header?

So e.g.: Validator.HTTPHeaderValue_sec-ch-ua=^.*$

I tried it with this string, but seems to be ignored by system as nothing changed.

 

Greetings

 

Andreas

 

 

slangley
23-Emerald I
(To:slangley)

Hi @atondorf.

 

We found that we do have an internal case for this issue.  It is currently a work in progress but you can review this article for updates.  Currently, the article indicates there is no timeline for a fix but it is actively being worked.

 

I will also be providing updates here for others in the community.

 

Regards.

 

--Sharon

atondorf
12-Amethyst
(To:slangley)

Thanks for the information!
But I am really curious, why I am not able to just set the regex to accept the backslash ...

slangley
23-Emerald I
(To:slangley)

Hi @atondorf.

 

I'm afraid I don't have an answer to that question.

 

Regards.

 

--Sharon

slangley
23-Emerald I
(To:slangley)

Hi @atondorf.

 

We received an update from R&D on the internal case with the following suggestions for solving the regex problem.  You can start by adding/modifying the following options in ThingworxStorage/esapi/validation.properties:

 

Encoder.AllowMultipleEncoding=true
Encoder.AllowMixedEncoding=true


With these options, multiple encoder errors turn into warning messages in the logs.  After turning an error into a warning message you can use logback.xml file (attached) so that the file can either shift the log or disable the class level log.

 

You will also need to add the following to ThingworxStorage/esapi/validation.properties:


ESAPI.Logger=org.owasp.esapi.reference.Log4JLogFactory
Validator.HTTPHeaderValue=<Modify the regex according to the input value from the error log>

 

Example:  Validator.HTTPHeaderValue=^[a-zA-Z0-9()\\-=\\*\\.\\?;,+\\/:&_ %\$\\\\ \u00A1-\uFFFF'"]*$

 

After making any needed updates to the attached logback.xml file, copy it to the ThingworxPlatform folder to disable the Encoder level class so the multiple encoding warning will not appear in the Thingworx and Tomcat logs.  Restart Tomcat.

 

After adding the above configuration any warning or error message will then show that it's clearly a regex validation issue, and you can make further adjustments as needed.

 

Please let us know if this helps to solve your issue.

 

Regards.

 

--Sharon

 

atondorf
12-Amethyst
(To:slangley)

Hi @slangley,

 

thanks for your answer, but I am a little confused with it:

  • To solve my described issue (and I think many other people have the same),
    it should be enough to adopt the regex, to your example regex, which should accept the '\':
    Validator.HTTPHeaderValue=^[a-zA-Z0-9()\\-=\\*\\.\\?;,+\\/:&_ %\$\\\\ \u00A1-\uFFFF'"]*$
  • But Thingworx does not accept this and again gives me the error: "Input validation failure"
    There seems to be something wrong with the regex and validation still fails.

So I checked the other parameters you send me ... and they are all already not defined in "validation.properties" but in "ESAPI.properties":

  • ESAPI.Logger=org.owasp.esapi.reference.Log4JLogFactory (already set here).
    Encoder.AllowMultipleEncoding=false  (you suggested to set it to true
    Encoder.AllowMixedEncoding=false     (you suggested to set it to true)
  • But the comment in the file says:
    # Multiple encoding is when a single encoding format is applied multiple times.
    # Allowing multiple encoding is strongly discouraged.
  • I tried to change these values in ESAPI.properties, but with each start of Tomcat they switch back to false. So I added these two values to the “validation.properties” and set them to true.
  • ==> The Errors in Application Log are gone now! Great!

This is the current situation without any “logback.xml”. And to be honest I do not understand this file at all:

So only with modification to "validation.properties" it's working, but I do not understand how it's working.
What did I really changed with AllowMultipleEncodingAllowMixedEncoding and why is it strongly discouraged?
Is it a fix or is it just and ugly workaround?

 

By the way ... I attached my working validation.properties file, but added the extension "txt".
This forum did not allowed me to upload a textfile with "properties" extension.
Regex validation ...

 

Greetings

 

Andreas

BhushanNehe
14-Alexandrite
(To:atondorf)

Hi @atondorf ,

 

Glad that the errors are gone after you made the changes. The logback.xml file in the Thingworx.platform folder disables the Encoder level class loggers to avoid the multiple encoding warning in the logs. We will publish an article soon with further details and appropriate steps that shall answer most of your other questions. 

 

Regards,

Bhushan

BhushanNehe
14-Alexandrite
(To:atondorf)

Hi @atondorf ,

 

We are still working on a permanent fix for the issue but just wanted check if the logs are still stable with the changes.

 

Regards,

Bhushan

slangley
23-Emerald I
(To:slangley)

Hi @atondorf.

 

After re-reading my response, I can see that it is confusing.  Apologies!

 

I have reached out to a resource that was involved in the issue originally that should be able to provide a better explanation.  If he doesn't respond over the next couple of days, I'll track someone down for a better understanding of what needs to be done.

 

Happy Holidays!

 

--Sharon

Announcements