cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

General tuning question

Aquamarine

General tuning question

We have a real fast system on PDMLink 8.0 with plenty of memory. There
are 3 foreground MS running and one background MS. We are still
adjusting tuning but I have observed something that puzzles me. We
occasionally see once methodserver run out of memory and get into a full
GC loop. See log below. Once this occurs, performance drops to nothing,
users call in, etc., etc. I would think that the other two methodserver
would pick up the slack while one of them is hung up. This does not
appear to be the case as they are idle while this is occurring. I do
observe that normally, that there is activity on the other MS at the
same time, just not when one of them is hung up.



My concern is that if I just give it more memory, it will just take
longer to get to the Full GC issue and when it does, the time to do a
Full GC will be even longer. Notice it was taking 50 seconds each time.
I almost wish the MS would just die on its own and restart. Any
thoughts on this?



<property name="wt.manager.cmd.MethodServer.java.command"&lt;br"/>overridable="true" targetFile="codebase/wt.properties"
value="$(wt.java.cmd.quoted) $(wt.manager.cmd.common.java.args)
-Djava.protocol.handler.pkgs=HTTPClient
-DHTTPClient.disableKeepAlives=true -DHTTPClient.dontChunkRequests=true
-Xms1024m -Xmx2048m $(wt.manager.cmd.MethodServer.platform.java.args)
-XX:+DisableExplicitGC -XX:SurvivorRatio=32 -XX:-UseParallelGC
-Xloggc:$(wt.logs.dir)$(dir.sep)ms_gc.log -verbose:gc
wt.method.MethodServerMain"/>







51494.979: [Full GC 2076607K->1949236K(2076608K), 46.1364942 secs]

51545.018: [Full GC 2076607K->1994339K(2076608K), 46.3318416 secs]

51594.079: [Full GC 2076608K->2018604K(2076608K), 46.5626202 secs]

51644.007: [Full GC 2076607K->2043226K(2076608K), 57.8493474 secs]

51703.710: [Full GC 2076607K->2047904K(2076608K), 47.3376738 secs]

51753.463: [Full GC 2076606K->2057851K(2076608K), 47.5200578 secs]

51803.119: [Full GC 2076604K->2063739K(2076608K), 46.9235827 secs]

51851.433: [Full GC 2076607K->2067256K(2076608K), 58.5157997 secs]

51911.229: [Full GC 2076607K->2069884K(2076608K), 45.3832626 secs]

51957.736: [Full GC 2076605K->2071433K(2076608K), 45.9500268 secs]

52004.382: [Full GC 2076607K->2073742K(2076608K), 48.4937066 secs]

8 REPLIES 8

General tuning question


We ran into more or less the same situation only when a MethodServer runs out of memory it reboots itself. The snag is that even though the other method servers are processing, users are hung up until the MethodServer that is rebooting is fully up. What I have found is that you have to increase the number of method servers so that the load is spread across as many as the system can tolerate without existing memory and saturating the CPU.


Our memory setting for each MethodServer is :

xconfmanager -s wt.manager.cmd.MethodServer.java.command="$(wt.java.cmd.quoted) $(wt.manager.cmd.common.java.args) -Djava.protocol.handler.pkgs\=HTTPClient -DHTTPClient.disableKeepAlives\=true -DHTTPClient.dontChunkRequests\=true -Xms1024m -Xmx1024m -XXSmiley TongueermSize\=128m -XX:MaxPermSize\=128m $(wt.manager.cmd.MethodServer.platform.java.args) wt.method.MethodServerMain" -t codebase/wt.properties


I thought I read that the JRE cannot handle more than 1024 memory allocated to it but I could be wrong.



"Villanueva, Antonio" <->


"Villanueva, Antonio" <->
08/09/2007 09:15 AMPlease respond to
"Villanueva, Antonio" <->
To
-
cc

Subject
[solutions] - General tuning question
We have a real fast system on PDMLink 8.0 with plenty of memory. There are 3 foreground MS running and one background MS. We are still adjusting tuning but I have observed something that puzzles me. We occasionally see once methodserver run out of memory and get into a full GC loop. See log below. Once this occurs, performance drops to nothing, users call in, etc., etc. I would think that the other two methodserver would pick up the slack while one of them is hung up. This does not appear to be the case as they are idle while this is occurring. I do observe that normally, that there is activity on the other MS at the same time, just not when one of them is hung up.

My concern is that if I just give it more memory, it will just take longer to get to the Full GC issue and when it does, the time to do a Full GC will be even longer. Notice it was taking 50 seconds each time. I almost wish the MS would just die on its own and restart. Any thoughts on this?

<property name="wt.manager.cmd.MethodServer.java.command" overridable="true" targetfile="codebase/wt.properties" value="$(wt.java.cmd.quoted)" $(wt.manager.cmd.common.java.args)=" -djava.protocol.handler.pkgs="HTTPClient" -dhttpclient.disablekeepalives="true" -dhttpclient.dontchunkrequests="true" -xms1024m=" -xmx2048m=" $(wt.manager.cmd.methodserver.platform.java.args)=" -xx:+disableexplicitgc=" -xx:survivorratio="32" -xx:-useparallelgc=" -xloggc:$(wt.logs.dir)$(dir.sep)ms_gc.log=" -verbose:gc=" wt.method.methodservermain&quot;="/>



51494.979: [Full GC 2076607K->1949236K(2076608K), 46.1364942 secs]
51545.018: [Full GC 2076607K->1994339K(2076608K), 46.3318416 secs]
51594.079: [Full GC 2076608K->2018604K(2076608K), 46.5626202 secs]
51644.007: [Full GC 2076607K->2043226K(2076608K), 57.8493474 secs]
51703.710: [Full GC 2076607K->2047904K(2076608K), 47.3376738 secs]
51753.463: [Full GC 2076606K->2057851K(2076608K), 47.5200578 secs]
51803.119: [Full GC 2076604K->2063739K(2076608K), 46.9235827 secs]
51851.433: [Full GC 2076607K->2067256K(2076608K), 58.5157997 secs]
51911.229: [Full GC 2076607K->2069884K(2076608K), 45.3832626 secs]
51957.736: [Full GC 2076605K->2071433K(2076608K), 45.9500268 secs]
52004.382: [Full GC 2076607K->2073742K(2076608K), 48.4937066 secs]

General tuning question

These 4 settings are key to making sure those 3 MS's are balancing
properly also. You can have 50 but if they aren't balanced, it's all
moot.



<property name="wt.method.rmi.maxSockets" overridable="true"&lt;br"/>targetFile="codebase/wt.properties" value="500"/>

<property name="wt.manager.rmi.maxSockets" overridable="true"&lt;br"/>targetFile="codebase/wt.properties" value="100"/>

<property name="wt.method.loadbalance.activeContext" overridable="true"&lt;br"/>targetFile="codebase/wt.properties" value="5"/>

<property name="wt.method.loadbalance.maxRedirects" overridable="true"&lt;br"/>targetFile="codebase/wt.properties" value="4"/>



The loadbalance ones are a bit touchy in that you have to do some
testing for your specific method server setup. I "think" I used these
for 2 method servers.











Cheers,





Roger Willson

Global CAD Manager



SRAM Corporation

1333 North Kingsbury

Chicago, IL 60622

312-664-8800 x1358

-


General tuning question

32-bit JVMs can generally handle more than a 1GB heap.

How much more is dependent on the OS and its configuration.

--
Jess Holle

Ryan Hoffmann wrote:
>
> We ran into more or less the same situation only when a MethodServer
> runs out of memory it reboots itself. The snag is that even though the
> other method servers are processing, users are hung up until the
> MethodServer that is rebooting is fully up. What I have found is that
> you have to increase the number of method servers so that the load is
> spread across as many as the system can tolerate without existing
> memory and saturating the CPU.
>
>
> Our memory setting for each MethodServer is :
>
> xconfmanager -s
> wt.manager.cmd.MethodServer.java.command="$(wt.java.cmd.quoted)
> $(wt.manager.cmd.common.java.args)
> -Djava.protocol.handler.pkgs\=HTTPClient
> -DHTTPClient.disableKeepAlives\=true
> -DHTTPClient.dontChunkRequests\=true -Xms1024m -Xmx1024m
> -XXSmiley TongueermSize\=128m -XX:MaxPermSize\=128m
> $(wt.manager.cmd.MethodServer.platform.java.args)
> wt.method.MethodServerMain" -t codebase/wt.properties
>
>
> I thought I read that the JRE cannot handle more than 1024 memory
> allocated to it but I could be wrong.
>
>
>
> Inactive hide details for "Villanueva, Antonio"
> <->"Villanueva, Antonio"
> <->
>
>
> *"Villanueva, Antonio"
> <->*
>
> 08/09/2007 09:15 AM
> Please respond to
> "Villanueva, Antonio"
> <->
>
>
>
> To
>
> -
>
> cc
>
>
> Subject
>
> [solutions] - General tuning question
>
>
>
>
> We have a real fast system on PDMLink 8.0 with plenty of memory. There
> are 3 foreground MS running and one background MS. We are still
> adjusting tuning but I have observed something that puzzles me. We
> occasionally see once methodserver run out of memory and get into a
> full GC loop. See log below. Once this occurs, performance drops to
> nothing, users call in, etc., etc. I would think that the other two
> methodserver would pick up the slack while one of them is hung up.
> This does not appear to be the case as they are idle while this is
> occurring. I do observe that normally, that there is activity on the
> other MS at the same time, just not when one of them is hung up.
>
> My concern is that if I just give it more memory, it will just take
> longer to get to the Full GC issue and when it does, the time to do a
> Full GC will be even longer. Notice it was taking 50 seconds each
> time. I almost wish the MS would just die on its own and restart. Any
> thoughts on this?
>
> <property name="wt.manager.cmd.MethodServer.java.command" <br="/>> overridable="true" targetFile="codebase/wt.properties"
> value="$(wt.java.cmd.quoted) $(wt.manager.cmd.common.java.args)
> -Djava.protocol.handler.pkgs=HTTPClient
> -DHTTPClient.disableKeepAlives=true
> -DHTTPClient.dontChunkRequests=true -Xms1024m -Xmx2048m
> $(wt.manager.cmd.MethodServer.platform.java.args)
> -XX:+DisableExplicitGC -XXSmiley FrustratedurvivorRatio=32 -XX:-UseParallelGC
> -Xloggc:$(wt.logs.dir)$(dir.sep)ms_gc.log -verbose:gc
> wt.method.MethodServerMain"/>
>
>
>
> 51494.979: [Full GC 2076607K->1949236K(2076608K), 46.1364942 secs]
> 51545.018: [Full GC 2076607K->1994339K(2076608K), 46.3318416 secs]
> 51594.079: [Full GC 2076608K->2018604K(2076608K), 46.5626202 secs]
> 51644.007: [Full GC 2076607K->2043226K(2076608K), 57.8493474 secs]
> 51703.710: [Full GC 2076607K->2047904K(2076608K), 47.3376738 secs]
> 51753.463: [Full GC 2076606K->2057851K(2076608K), 47.5200578 secs]
> 51803.119: [Full GC 2076604K->2063739K(2076608K), 46.9235827 secs]
> 51851.433: [Full GC 2076607K->2067256K(2076608K), 58.5157997 secs]
> 51911.229: [Full GC 2076607K->2069884K(2076608K), 45.3832626 secs]
> 51957.736: [Full GC 2076605K->2071433K(2076608K), 45.9500268 secs]
> 52004.382: [Full GC 2076607K->2073742K(2076608K), 48.4937066 secs]
>

General tuning question

Antonio,



There are so many gc options that its tough to know exactly which
combination to use, but we are having success with the following:



-Djava.protocol.handler.pkgs\=HTTPClient
-DHTTPClient.disableKeepAlives\=true
-DHTTPClient.dontChunkRequests\=true -DLD_PRELOAD\=/usr/lib/libumem.so
-Xms3400m -Xmx3400m -XX\:NewSize\=1500m -XX\:MaxNewSize\=1500m
-XX\:SurvivorRatio\=4 -XX\:TargetSurvivorRatio\=90
-XX\:MaxTenuringThreshold\=31 -XX\:PermSize\=96m -XX\:MaxPermSize\=96m
-XX\:+UseParNewGC -XX\:ParallelGCThreads\=2 -XX\:+UseAdaptiveSizePolicy
-XX\:+UseTLAB -XX\:+ResizeTLAB -XX\:+DisableExplicitGC -XX\:+UseMPSS
-Xloggc:/software/ptc/wc80/Windchill/codebase/gclogs/ms_gc{0}.log



Notice the use of the gc model: UseParNewGC



These settings are being used on a solaris system with 48G of memory and
using 5 methodservers.



The settings below are important as well; maxRedirects need not be >
then the number of MethodServers.




General tuning question

Antonio,

This probably isn't causing your problem, but it's a good rule of
thumb to set -Xms=-Xmx. This avoids full GCs while the heap resizes.

Also, another rule of thumb is to set maxRedirects=(#MSs - 1). The
theory behind this is that in the fraction of a second it takes to
redirect the request through the first two MSs, the first one will
still be using all of its activeContexts.

Please share a little about your architecture? OS, RAM, and CPU info
would be helpful to make further suggestions.

BP

> These 4 settings are key to making sure those 3 MS's are balancing
> properly also. You can have 50 but if they aren't balanced, it's all
> moot.
>
>
>
> <property name="wt.method.rmi.maxSockets" overridable="true"&lt;br"/>> targetFile="codebase/wt.properties" value="500"/>
>
> <property name="wt.manager.rmi.maxSockets" overridable="true"&lt;br"/>> targetFile="codebase/wt.properties" value="100"/>
>
> <property name="wt.method.loadbalance.activeContext" overridable="true"&lt;br"/>> targetFile="codebase/wt.properties" value="5"/>
>
> <property name="wt.method.loadbalance.maxRedirects" overridable="true"&lt;br"/>> targetFile="codebase/wt.properties" value="4"/>
>
>
>
> The loadbalance ones are a bit touchy in that you have to do some
> testing for your specific method server setup. I "think" I used these
> for 2 method servers.
>
>
>
>
>
>
>
>
>
>
>
> Cheers,
>
>
>
>
>
> Roger Willson
>
> Global CAD Manager
>
>
>
> SRAM Corporation
>
> 1333 North Kingsbury
>
> Chicago, IL 60622
>
> 312-664-8800 x1358
>
> -
>
>
>

Follow up: General tuning question

Thank you all for the many responses. This is a summary message plus
some follow up questions.

It appears that most people have -Xms and -Xmx set to the same value to
avoid FullGC when the heap expands. I always tried to avoid this since
minor GCs would be shorted with a smaller heap. But when it looked at
it, it was the difference between 500 ms and 1000 ms every 10-20 minutes
or so. I think I am at 1-2% GC.



Some of you commented on the -XX:MaxPermSize. We have that at 256MB and
we will see if that was the cause of our issues but I suspect not. I
think adding -XX:+PrintGCDetails will help because it now tells us what
area the GC is affecting (Eden vs. Tenured).



The maximum amount of RAM for the process is not an issue since we are
on UNIX 64bit. We can go higher than 1GB and we have plenty of reserve.



Many of your called out a setting of -XX:+UseParNewGC. My assumption is
that this is a Java 1.5 setting to specify a new parallel GC method. I
do not think this is available in Java 1.4.2 which is what we are using.
Sorry, I did not mention that. Anyone know the difference with this new
collector?


RE: General tuning question

Antonio,
To avoid issues with individual methodservers running out of memory in our experience it is essential to set the following parameters;

wt.pom.paging.snapshotQueryLimit

wt.pom.queryLimit

We use 5000 as the limit for both settings above. We are running 7.0 foundation, but I expect the same things still apply with PDMLink.

Without these set any user can accidentally crash a methodserver by simply hitting enter on a search screen and causing the methodserver to try and return the entire dataset. We had issues similar to you describe before we set these. Worse is that in extreme cases the servermanager assumes a dead methodserver, and starts a new one. If this process repeats a few times eventually the operating system starts to run out of memory.

A couple of things to note are that;

You do need to set the query limit high enough to allow normal user activity, and this includes browsing contents etc. So the limit must be higher than any typical user return say from the contents of a sub folder.

The error message returned to users when the limit is exceeded is sometimes not clear. This depends on the screen being used and just requires a little education.

There might be valid business reasons why you might temporarily require a very high, or no limit to run some wide queries. Adjusting this requires a restart of the methodserver, so any such activity will require some co-ordination.

Hope this is useful,
-----
Lewis



Command Line Setting ACL's Within PDMLink

Good Day All!

Does PDMLink provide a command line option for setting ACL's within
PDMLink by making use of a XML file? If so where can I find more
information and an example file(s).

Thanks


Jim Van Dragt
Information Technology
Herman Miller Inc.
616.654.5285 - Office
616.836.8394 - Cell

Announcements
LiveWorx Call For Papers Happening Now!