Unexpected error occurred on server when saving service

[seanccc]

Hi ,

I often get the error "Unexpected error occured on server" when saving service.  I cannot save service anymore as long as the error occurs ,  I have to exit the Edit mode and edit the service again. Even though, the 2nd editing won't assure the save can be successful.  

I don't see any related logs when the error occurs, including Thingworx platform logs and tomcat logs.  It's very boring especially when it happens more than 10 times in one hour sometimes.   

Any suggestion ? 

[zyuan1]

What if you put the log level to Trace level and then reproduce the issue, you may get more detailed information

[BennyB]

Same issue here but no chance to reproduce, it seems to occur randomly.

[Constantine]

Hello,

Just my 2 cents. We used to experience somewhat similar behavior (although I do not recall the exact error message) that was caused by ThingWorx enabling/disabling things. For example, if there were two developers concurrently working on the same instance, and one was modifying a ThingShape, then the second couldn't save his modifications to the ThingTemplate inheriting this ThingShape until the system (re-)enabled/disabled all corresponding Things. The server was full of test data, so those "downtimes" were taking seconds and were quite noticeable. It clearly wasn't the smartest idea to work on a heavily loaded system concurrently, so we solved this issue by changing our development process.

I would suggest enabling TRACE logs for the ApplicationLog, it might give you a better idea of what happens inside the platform when you click Save.

Regards,
Constantine

[seanccc]

@Constantine  ,

Thank you for share the experience,  but I'm the only person who use the Thingworx server . 

I've enabled the Stacking tracking and script stack tracking in the LoggingSubSystem, but still got nothing in the log when the error occurs.  Any more suggestion ? 

Regards,

Sean

[vangne]

@seanccc 

1. Can you confirm the version of ThingWorx?
2. Is this a fresh install or upgrade?
3. Does this error appear for a specific entity or for every entity that you are saving?
4. If you restart Tomcat do you still experience the same error message?

5. Could you please try clearing the cache and closing and reopening the browser if that works?

Thanks,

Vibhuti

[seanccc]

@vangne 

@vangne ,

Please check my reply below: 

1. Can you confirm the version of ThingWorx?

---ThingWorx 8.4.2 + Postgresql 10.x + Tomcat 9.x + Windows Server 2016(VM)
2. Is this a fresh install or upgrade?

--- It's a fresh install
3. Does this error appear for a specific entity or for every entity that you are saving?

--It appears for any entity randomly 
4. If you restart Tomcat do you still experience the same error message?

--I shut down VM every day but sometimes this problem occurs within 5 minutes of turning on the VM.  If click Cancel then try the 2nd time, there is a 10~30% chance that it will happen again.   If press F5 to refresh the page,  it won't occur the 2nd time, but the error may occur in several minutes . 

5. Could you please try clearing the cache and closing and reopening the browser if that works?
--clearing the cache only won't work, reopening the browser works as refresh the page already made it working.

[vangne]

Hi @seanccc

If you were able to resolve your issue, please post it here and mark it as the Accepted Solution for the benefit of others on the community.  If you are still experiencing issues, please do let us know your questions or concerns.

I came across an issue where a similar issue was observed but it involved a load balancer. Please let me know if this is applicable to you. Please take a look at this article for more information.

Also, Can you paste the stack trace logs here, please ?

Thanks,

Vibhuti

[PEHOWE]

@Constantine 

Did you have multiple browser windows open which were editing objects?

I have seen the error. It seemed to be when I had multiple windows open/edit.

[seanccc]

@PEHOWE ,

yes, I usually open multiple tabs in Chrome when developing services. 

[vangne]

@seanccc 

Can you please try developing/saving service in only one open browser/tab? 

Thanks,

Vibhuti

[seanccc]

@vangne ,

I tried saving service in only one Chrome tab, but still got the same error sometimes.  The trace error is also the same. 

[vangne]

@seanccc

Thanks for your previous response.

I would recommend opening a support case with us for further investigation of this issue.

Thanks,

Vibhuti

[seanccc]

Hi everyone,

I found the trace log from ErrorLog.log for the error. but have no idea about the error. 

2019-09-13 11:58:23.815+0800 [L: ERROR] [O: E.c.t.w.Things] [I: ] [U: Administrator] [S: ] [T: http-nio-8080-exec-10] [message: org.json.JSONException: JSONObject["baseType"] not a string.] org.json.JSONException: JSONObject["baseType"] not a string.
at org.json.JSONObject.getString(JSONObject.java:810)
at com.thingworx.metadata.FieldDefinition.fromJSON(FieldDefinition.java:608)
at com.thingworx.metadata.ServiceDefinition.fromJSON(ServiceDefinition.java:331)
at com.thingworx.metadata.collections.ServiceDefinitionCollection.fromJSON(ServiceDefinitionCollection.java:66)
at com.thingworx.implementation.ThingShapeImplementation.fromJSON(ThingShapeImplementation.java:253)
at com.thingworx.things.Thing.fromJSON(Thing.java:9537)
at com.thingworx.system.managers.ThingManager.fromJSON(ThingManager.java:132)
... 50 common frames omitted
Wrapped by: java.util.concurrent.ExecutionException: org.json.JSONException: JSONObject["baseType"] not a string.
at java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
at java.util.concurrent.CompletableFuture.get(Unknown Source)
at com.thingworx.webservices.BaseService.handleUpdate(BaseService.java:996)
at com.thingworx.webservices.BaseService.service(BaseService.java:492)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.thingworx.security.contenttype.ContentTypeFilter.doFilter(ContentTypeFilter.java:138)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.thingworx.security.filter.ValidationFilter.doFilter(ValidationFilter.java:22)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.thingworx.security.authentication.AuthenticationFilter.propagateRequest(AuthenticationFilter.java:541)
at com.thingworx.security.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.java:210)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)
at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:389)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.thingworx.security.filter.ClickjackFilter.doFilter(ClickjackFilter.java:178)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.thingworx.security.filter.ProhibitIfNotLeaderFilter.doFilter(ProhibitIfNotLeaderFilter.java:120)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.thingworx.security.filter.HttpResponseHeadersFilter.doFilter(HttpResponseHeadersFilter.java:173)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)

[aashwait]

I've also seen this issue many times and on many different environments. It can be very frustrating for developers. Has anyone found a solution to this issue?

[wposner-2]

I had a ticket open with support for this issue.  They couldn't figure it out.  The good news is that it seems to be fixed in the 8.5.1 build of TWX.

[Constantine]

Hello @wposner-2,

I can confirm that we still see this error occasionally in 8.5.1.

It happens rarely and thus hard to reproduce. Seems to be related to multiple tabs opened simultaneously, so we just agreed to minimize the number of open windows and it seems to help.

/ Constantine

[BennyB]

We are running 8.4.4 and cannot simply upgrade. Any chances for us other than avoiding multiple tabs?

[slangley]

Hi everyone.

For those experiencing this problem, which browsers are in use?  Has it only been observed under Google Chrome?  Would be helpful to know which versions as well.

Regards.

--Sharon

[jasperlg]

I use chrome, but my colleague uses Firefox and also has this issue.

[BennyB]

Google Chrome 79.0.3945.130 (64 Bit)

[slangley]

Hi @jasperlg.

Please provide the versions as well.

Thanks.

--Sharon

[slangley]

Hi again.

ThingWorx 9.0 which contains the fix for this issue, has been released.  The targeted release date for 8.4.11 and 8.5.7 is July 21.

Regards.

--Sharon

[jasperlg]

We're having the same issue, Very annoying.

[slangley]

Hi everyone.

A new internal ticket has been created on this issue with the feedback you each provided.  We will post an update here once feedback is received from R&D.

Regards.

--Sharon

[slangley]

Hi @Constantine @wposner-2 @seanccc @BennyB @jasperlg.

R&D is reviewing this issue and have requested the following:

• An export of the entity that failed to save.
• After enabling stacktrace and reproducing the issue, a copy of ALL ThingWorx logs (located in \ThingworxStorage\logs)

To enable stacktrace:

1. Browse to Subsystems
2. Select LoggingSubsystem
3. Select Configuration
4. Select "Enable Stack Tracing" and "Enable Script Stack Tracing"
5. Click Save

If anyone is able to recreate the issue in order to provide this information, R&D will then be able to move forward in their investigation for addressing the issue.

Thanks everyone.

--Sharon

[slangley]

Hi @BennyB.

Perhaps I'm missing something, but the original issue that was reported was the inability to save Services.  Your instructions relate to an issue with mashups when using a custom extension.

.

For the current issue, we need the following to be able to continue in our investigation:

• An export of the entity that failed to save.
• After enabling stacktrace and reproducing the issue, a copy of ALL ThingWorx logs (located in \ThingworxStorage\logs)

To enable stacktrace:

1. Browse to Subsystems
2. Select LoggingSubsystem
3. Select Configuration
4. Select "Enable Stack Tracing" and "Enable Script Stack Tracing"
5. Click Save

[BennyB]

I am emphasizing once again, this happens for any kind of entities, Mashups are only the one example I was able to reproduce. Forget about the extension.

I did enable the stacktrace like described.

[slangley]

Hi @BennyB.

This thread is specific to services.  I will spin your issue off to a separate thread and open a separate ticket with R&D for that one, as we cannot lump multiple issues together.

I did, however, receive follow-up from R&D today advising they had been able to recreate the issue regarding the error being received while saving a service.  In that situation, they discovered they had made a typo, and once corrected, they were able to save successfully.  Any thoughts on whether this could be the scenario in your situation?

Regards.

--Sharon

[BennyB]

In my case it was surely not a typo. But since my case is treated separately now I will step out of this one to leave all others a chance to reply.