Need Windchill Server restart every day

[MatteoRussi]

Hi, every day I need to restart my Windchill Server because all queues are stopped.

The error message in log files is:

2018-06-12 00:05:00,033 ERROR [commonScheduleQueue.SchedulingThread] com.ptc.wa.create.WAReportCreationManager Administrator - Report generation could not complete successfully after 5 tries

2018-06-12 00:05:06,210 INFO  [WfPropagationQueue.PollingThread] wt.system.out Administrator - NO All Activities are executed --> GO BACK

2018-06-12 00:05:06,694 WARN  [QueueCheckerThread-28] wt.jmx.notif.queue  - Time=2018-06-12 00:05:04.897 +0200, Name=QueueNotifier, SourceObjectName=com.ptc:wt.subsystem=Monitors,wt.monitorType=Queues,Name=WfPropagationQueue, class=class javax.management.Notification, type=wt.queue.QueueWatcherMBean.exceededWaitingEntriesThreshold, userData=25, message=There are 25 queue entries waiting to be executed in queue WfPropagationQueue, JVM Name=3416@SRVPLMAPP110, Queue Info=[AveEntryExecTimeSeconds=0.048,EntryExecutionTimeThresholdSec=900,ExecutionThreadLoggerLevel=,ExecutionThreadLoggerName=wt.queue.ProcessingQueue.WfPropagationQueue,LoggerLevel=,LoggerName=wt.queue.QueueWatcherImpl.WfPropagationQueue,MaxEntryExecTimeSeconds=528.135,MinEntryExecTimeSeconds=0.0,MinMinutesBetweenNotifications=60,Name=WfPropagationQueue,TotalEntries=26,TotalEntriesExecuted=687835107,TotalExecutionTime=33347301669,TotalThreshold=100,WaitingReadyEntries=25,WaitingReadyThreshold=15]

2018-06-12 00:05:09,587 INFO  [WfPropagationQueue.PollingThread] wt.system.out Administrator - NO All Activities are executed --> GO BACK

2018-06-12 00:05:15,446 INFO  [WfPropagationQueue.PollingThread] wt.system.out Administrator - NO All Activities are executed --> GO BACK

I can't understand what is the problem. Anyone has a solution?

Thanks a lot

[BenLoosli]

Are the servers being reboot automatically every night? I reboot mine at 5:00AM to be sure there is nothing hanging around that could cause any issues.

Do the Windchill processes restart automatically or does someone have to start them each morning?

[MatteoRussi]

No, the server are not being reboot automatically. We must reboot them every morning because the queue are all blocked. As you can see in the error in log file.

Windchill processes starts automatically

[MikeLockwood]

We only reboot once / month, and that is only because of OS patching.  Should not have to reboot very often.  Something fundamentally wrong.

The queue is specifically for "wf propagation" which relates to workflow processes.  Likely have a robot doing something it shouldn't with java code in a workflow template, or possibly in a loop.

Look at this from the message you posted: TotalEntriesExecuted=687835107 (something happening 687 million times)!!

[MatteoRussi]

Oh I haven't seen that detail, good. How I can find where is the error? I know a solution for this problem?

Thanks.

[MikeLockwood]

Clear all queue entries for queues that begin with wf.

In Lifecycle Administration, uncheck "Enabled" for all workflow templates in use (which will of course disrupt the system and users).  Ensure that queue entries do not build up.

One by one enable the workflow templates and start to exercise normal functionality.  One of them will make a huge difference (587 million).

[Invisigoth]

FWIW: I am seeing the "com.ptc.wa.create.WAReportCreationManager Administrator - Report generation could not complete successfully after 5 tries" error after doing a 10.2 M030 CPS-13 to 11.0 M030 CPS10 test upgrade. 

Obviously not a production system with active users, so I am not seeing any of wfpropagation errors.  Am seeing the system trying to send mail about 'queue problems' which could be related. I did not setup the system to exchange data bc this is just a POC at this point to get an idea of time/tasks involved. 

Only two docs that mention this report.  I believe this is because I did not apply the WPA3 patch to the 10.2 source. At least that will be my working proposition when I roll back and re-do this.