Maintenance: Restarting Servers

[Darquise]

Is there a recommendation as to how often the Windchill and database servers should be restarted?

Should they be restarted monthly? Is there an order as to which server should be restarted first?

[ChrisSpartz]

There's no official recommendation from PTC about how often to restart the servers. Theoretically, they shouldn't ever need to be restarted, unless you make a configuration change that requires a restart. Though, in practice, that may not be the case.

As to the order of restarting, the DB needs to be available in order for Windchill to start up. So, the DB will need to be restarted first.

[jessh]

LDAP should be up when Windchill is started as well.

Essentially the back-end datastores should be up first.

[Darquise]

Thank you for your responses...

I find that approx every 25-30 days I get a "Memory usage threshold exceeded" notification and then Windchill slows right down and is unresponsive. so I restart the server, hence why I wanted to setup a monthly scheduled server restart.

How do you fix this error or clear memory usage without restarting the server?

[jessh]

This suggests a slow memory leak. What type of memory threshold was exceeded? Heap or perm gen?

If the threshold was perm gen, then it is possible that the perm gen allocation should simply be increased -- and that it takes 25-30 days simply to make use of enough different features/capabilities to load that many classes. It may, however, be a slow leak, but this sort of issue is very hard to track down. One note in this regard, if using CMS garbage collection, be sure to use the necessary arguments to force this to include perm gen collection (which is unfortunately not the default with CMS garbage collection).

If this is not a perm gen issue, however, then it is most likely a slow memory leak. The only way to get a good idea as to the cause is to grab a heap dump and then analyze this -- or, more likely, have PTC analyze this.

[Darquise]

Here is the notification that I received:

Memory usage threshold exceeded

Class:	class javax.management.Notification
Message:	Memory usage exceeds collection usage threshold
SequenceNumber:	11
Source (Direct):	com.ptc:wt.subsystem=Monitors,wt.monitorType=Memory
Source (Original):	java.lang:type=Memory
TimeStamp:	2014-05-08 19:30:06.340 +0000 [1399577406340]
Type:	java.management.memory.collection.threshold.exceeded
UserData:	count 14 poolName PS Old Gen usage committed 2859139072 init 2859139072 max 2859139072 used 2859139072
JVM Name:	3920@ACM
Threshold Info:	PermGenPercentCollUsage 84.45762062072754 PermGenPercentCollUsageThreshold 95.0 HeapPercentUsage 87.16486411932526 MinAvailableHeapPercThreshold 0.0 CheckIntervalSeconds 30 PermGenPercentUsagePeak 87.18524169921875 PermGenPercentUsage 84.46382713317871 MinAvailableHeapThreshold 0 PermGenPercentUsageThreshold 97.5 OldGenPercentUsage 100.0 GCBaitingEnabled false OldGenPercentCollUsageThreshold 95.0 OldGenPercentUsageThreshold 0.0 HeapPercentUsageThreshold 0.0 OldGenPercentUsagePeak 100.0 NoisyStillExceededNotifications false OldGenPercentCollUsage 100.0
Memory Data:	Verbose true ObjectPendingFinalizationCount 0 NonHeapMemoryUsage committed 404815872 init 195493888 max 469762048 used 404218584 HeapMemoryUsage committed 3812163584 init 4288675840 max 3812163584 used 3322858904
Memory Pool Data:	java.lang:type=MemoryPool,name=Code Cache CollectionUsageThresholdSupported false UsageThresholdExceeded true PeakUsage committed 50331648 init 2555904 max 50331648 used 49951616 Type NON_HEAP Usage committed 50331648 init 2555904 max 50331648 used 49951616 UsageThresholdCount 0 Name Code Cache UsageThreshold 0 UsageThresholdSupported true MemoryManagerNames CodeCacheManager CollectionUsage Valid true java.lang:type=MemoryPool,name=PS Eden Space CollectionUsageThresholdSupported true CollectionUsageThresholdExceeded true CollectionUsageThresholdCount 0 PeakUsage committed 1428094976 init 1072234496 max 1428094976 used 1428094976 Type HEAP Usage committed 476512256 init 1072234496 max 476512256 used 464485688 Name PS Eden Space UsageThresholdSupported false MemoryManagerNames PS MarkSweep PS Scavenge CollectionUsage committed 476512256 init 1072234496 max 476512256 used 458467824 Valid true CollectionUsageThreshold 0 java.lang:type=MemoryPool,name=PS Old Gen CollectionUsageThresholdSupported true CollectionUsageThresholdExceeded true CollectionUsageThresholdCount 14 UsageThresholdExceeded true PeakUsage committed 2859139072 init 2859139072 max 2859139072 used 2859139072 Type HEAP Usage committed 2859139072 init 2859139072 max 2859139072 used 2859139072 UsageThresholdCount 0 Name PS Old Gen UsageThreshold 0 UsageThresholdSupported true MemoryManagerNames PS MarkSweep CollectionUsage committed 2859139072 init 2859139072 max 2859139072 used 2859139072 Valid true CollectionUsageThreshold 2716182118 java.lang:type=MemoryPool,name=PS Perm Gen CollectionUsageThresholdSupported true CollectionUsageThresholdExceeded false CollectionUsageThresholdCount 0 UsageThresholdExceeded false PeakUsage committed 369360896 init 192937984 max 419430400 used 365681408 Type NON_HEAP Usage committed 354484224 init 192937984 max 419430400 used 354273576 UsageThresholdCount 0 Name PS Perm Gen UsageThreshold 408944640 UsageThresholdSupported true MemoryManagerNames PS MarkSweep CollectionUsage committed 354484224 init 192937984 max 419430400 used 354240936 Valid true CollectionUsageThreshold 398458880 java.lang:type=MemoryPool,name=PS Survivor Space CollectionUsageThresholdSupported true CollectionUsageThresholdExceeded true CollectionUsageThresholdCount 0 PeakUsage committed 476512256 init 178651136 max 476512256 used 476491120 Type HEAP Usage committed 476512256 init 178651136 max 476512256 used 0 Name PS Survivor Space UsageThresholdSupported false MemoryManagerNames PS MarkSweep PS Scavenge CollectionUsage committed 476512256 init 178651136 max 476512256 used 0 Valid true CollectionUsageThreshold 0

[jessh]

So with 100% of old generation used after garbage collection and 87.16% of overall heap being used, that's not looking particularly good. Was the system idle at the time? If so, then that's even worse.

As for what's filling all that memory, really only a heap dump can tell in detail. A heap histogram might give some clues, though.