Request for help on Monitoring Windchill performace issues

[jeffruff]

We are currently running Windchill 9.1 M050 on windows server that has 8 cores, 72 gigs of ram and are seeing intermittent performance issues. At periods during the day actions like folder expansion goes from 8 seconds to minutes. Looking at the performance meter on the server shows nothing to support that degradation. Running Jconsole on the serverManager [I cannot see any of other java processes found in Task Manager]shows the occassional 100% Heap utilization for Memory Pool OS Survivor Space. I am looking for help in identifying what additional tools we should be using and if there are documents to assist. Also any experiences other users may have worked through would be of great help.

Thanks in advance

[AL_ANDERSON]

Jeff,

If the problem is specific to one user looking at folders, but other users
on using that same method server do not experience any similar slow down,
then I would recommend having your DBA look for long-running queries
during the episodes of slow performance. Often, you can find an
inefficient query, missing or incorrect indexes, or even a problem where
an oracle "plan" for a certain type of query periodically encouters an
unexpected query of that type causing it to run for long periods of time.
We have found examples of each type of problem. Our local DBA is able to
find these long running queries pretty easily. If it is an indexing
problem, our DBA can usually fix it. For other problems, we usually need
PTC technical support or a performance tuning consultant from PTC Global
Services. My guess is that it is a database index issue based on what was
in your original post, but you will need to get your DBA involved to see
if the database is where the problem lies.

If the problem is not specific to one user, but rather general for all
operations on a common method server for some period of time, especially
if you find that the method server auto-restarts during that time period,
then you likely have a system properties and/or hardware problem requiring
modifications to your memory and garbage collection settings. We needed a
PTC performance tuning consultant to get to a point where we were
reasonably happy with most of the system's performance. Nevertheless, we
still periodically encounter cases where garbage collection on an
individual method server will cause a general slow down for all users on
that method server until it restarts. We have an on-going Tech Support
case with PTC for this issue. Two weeks ago, we made some additional
system properties changes, and since then we have not yet seen that same
problem, but we are not yet confident that the problem is completely
solved.

Al Anderson






Jeffry Ruff <jeffry.ruff@lineagepower.com>
10/11/2011 02:57 PM
Please respond to
Jeffry Ruff <jeffry.ruff@lineagepower.com>


To
-
cc

Subject
[solutions] - Request for help on Monitoring Windchill performace issues




Caterpillar: Confidential Green Retain Until: 11/10/2011



We are currently running Windchill 9.1 M050 on windows server that has 8
cores, 72 gigs of ram and are seeing intermittent performance issues. At
periods during the day actions like folder expansion goes from 8 seconds
to minutes. Looking at the performance meter on the server shows nothing
to support that degradation. Running Jconsole on the serverManager [I
cannot see any of other java processes found in Task Manager]shows the
occassional 100% Heap utilization for Memory Pool OS Survivor Space. I am
looking for help in identifying what additional tools we should be using
and if there are documents to assist. Also any experiences other users
may have worked through would be of great help.

Thanks in advance

Site Links: View post online View mailing list online Send new post
via email Unsubscribe from this mailing list Manage your subscription
Use of this email content is governed by the terms of service at:

[sdrzewiczewski]

To monitor your methodservers with jconsole (assuming your running as windows services) you need to use a different command. I don't have it handy but you can probably search the PTC knowledgebase and find it there too. If you can't email me and i'll send it tomorrow.

Steve D.

Sent from my iPhone

On Oct 11, 2011, at 5:57 PM, Jeffry Ruff <jeffry.ruff@lineagepower.com> wrote

> We are currently running Windchill 9.1 M050 on windows server that has 8 cores, 72 gigs of ram and are seeing intermittent performance issues. At periods during the day actions like folder expansion goes from 8 seconds to minutes. Looking at the performance meter on the server shows nothing to support that degradation. Running Jconsole on the serverManager [I cannot see any of other java processes found in Task Manager]shows the occassional 100% Heap utilization for Memory Pool OS Survivor Space. I am looking for help in identifying what additional tools we should be using and if there are documents to assist. Also any experiences other users may have worked through would be of great help.
>
>
>
> Thanks in advance
>
>
> Site Links: View post online View mailing list online Send new post via email Unsubscribe from this mailing list Manage your subscription
>
> Use of this email content is governed by the terms of service at:
>

[MikeLockwood]

Agree w/Al. Very likely database contention.
Need to create and dedicate a tablespace to each of the largest, most-active tables, possibly each with multiple data files.

[sdrzewiczewski]

Do you only have a single server with both the application and database running on it? If so I would suggest splitting that into two servers.

Steve D.

[patrick.chin1]

We have almost the same setup as you.

An excellent tool is http://www.appdynamics.com. App dynamics would have save $100k's of consulting to find the point of origin. These are my rules of setting up your system to run with the best performance:

1. Installed Windchill on RedHat 5.4 x86_64bit


2. One methodserver per 20-30 users. Since you have 8 cores, you can have 3 to 5.


3. If you have CAD applications, you require CAD workers:
   
   
   
   • 1 worker per 10 CAD users. If you have 60 ProE users, you should have 6 to 7 CAD workers. You can put all 6 CAD workers on a 2X quad core or 1 hex core system with 48 GIGS of ram. You can ask your PTC sales rep for more non-user ProE licences.
   
   
   • with CAD workers, the WVS querying and queuing process takes a heavy burden on both a methodserver and background method server. You will need 2 additional background methodservers for WVS1 and WVS2 (see TPI 130600 https://www.ptc.com/appserver/cs/view/solution.jsp?n=130600)
   
   


4. Each methodserver, foreground or background gets a min memory of 768 and max of 3968.
   
   
   
   • <property name="wt.method.maxHeap" overridable="true" targetfile="codebase/wt.properties"&lt;br"/> value="3968"/>
     <property name="wt.method.minHeap" overridable="true" targetfile="codebase/wt.properties"&lt;br"/> value="768"/>
   
   


5. Optional, but I did it, for oracle:
   
   
   
   • Used Oracle 11G on RedHat 5.4
   
   
   • I changed the Oracle block sizes to 32K which requires a new database and a import and mods to the Windchill blocks:
     
     
     
     • <property name="wt.generation.sql.tinyBlobSize" overridable="true"&lt;br"/> targetFile="codebase/user.properties"
       value="64k"/>
       <property name="wt.generation.sql.smallBlobSize" overridable="true"&lt;br"/> targetFile="codebase/user.properties"
       value="64k"/>
     
     
     • you have to run this in your windchill shell:
       
       
       
       • JavaGen.sh registry false false true false false
       
       
     
     
   
   
   • I used a 2X quad core with 48 GIGs of ram
   
   
   • allocated 40% of ram to Memory Size when creating my database (SGA and PGA)
   
   
   • Made sure that there is at least 50% of free space in each tablespace for growth and peak conditions.
   
   

Our previous HP-UX itaniums were running at 100% like yours with one methodserver getting pegged. We are now running intel 2Xquad 3.0 GIGHZs core with 48 GIGs of ram.I would go with the more cores per socket than just the 4X 2 dual cores. Found out it was due to the querying of the CAD files for the worker.Though the WVS is should be all done in the background, the singe worker itself is pulling files from Windchill which was causing issues. The single worker was backlogged of tons of object to be published in the queue. More workers flushes out the queue and doesn't cause a performance hit on the methodservers if there is more than 1 or 2.

Both our windchill and database servers never runs more than 25% CPU utilization on average.

Good luck,

Patrick