CAD worker slowness troubleshooting

[Dobi]

I'm trying to troubleshoot CAD worker performance to figure out why registering a server and downloading a simple (small), single part takes minutes. 

We've done bandwidth tests between the CAD worker VM and Windchill and can move a ton of data within seconds. But when the CAD worker is publishing (either Creo or NX), that performance is non-existent. 

For example: an average time to register server 1+ minute. 

This seems to happen for every single job that goes through on Creo. Why?

A single part (cube and cylinder for a demo) takes nearly 3 minutes to download.

While some recent settings seem to have bypassed NX jobs needing to re-register the server every job, even a washer takes 2+ minutes to download through the WGM. But if that same file is passed between the worker and server, it's near instant. What would cause such a drastic slowdown when the file is downloaded through WGM? 

[TomU]

It's probably worth opening a case with tech support.  I believe there is logging that can be turned on to monitor the entire process and then you can send them the logs for analysis.  It might also be worth monitoring the communication with something like Wireshark.

[TomU · Accepted solution]

https://www.ptc.com/en/support/article/cs292241

https://www.ptc.com/en/support/article/cs140107

[Dobi]

You, sir, are a treasure. 

I did some more tests and logging today and found that with the WGM Logging tool (opening a session to add to workspace the same file I would send through the publisher) no logs are captured for the publisher-initiated session. When I open WGM and add to workspace the same file it takes seconds instead of minutes. So the issue is buried somewhere in how the workers themselves are set up. At least it's pointing me in the right direction. 

[jlecoz]

To segregate network performance impact you could install the Creo worker on the Windchill server (test server).

Also, seems that server registration takes lots of time, this can be investigated just by running interactively a regular Creo connected session or NX with WGM session on the CAD worker environment. Any performance issue you get when doing that impacts the CAD worker performance and investigating is easier. For example, if Creo starts for a conversion and if it takes a lot of time to get a license you will notice it. You will also have the ability to tune the Creo configuration before working on, the Windchill visualization services configuration.

[Dobi]

I did some tests today with NX and WGM (while logging) and found that if I open WGM on the worker and add to workspace a file I would send through the publisher, the download performance is what I'd expect (7 seconds for ~100mb file set). However, when I send the same file through the publisher and only a few minutes apart, it took 5 minutes to add that same file set to the worker-initiated session. 

So to your point, it seems as though the behavior of the worker is different than the behavior of the stand-alone NX/WGM or Creo on that machine. 

@TomU 's pointers to articles are a great starting point so I'll exhaust those and hopefully get some time with tech support to get it sorted out. 

[MikeLockwood]

Please do post the eventual solution once discovered - all of us have struggled with this type of thing at times - maddening.

Sometimes it hangs or takes a long time on the thumbnail, but seems that what you are seeing is not that.

Sounds like getting a license is not the issue from running it manually on the CAD worker server, but it may be if the user in auth.properties is different that the user utilized when running manually.

Assuming the CAD worker is on a separate server, do you experience same after rebooting that server?

Do you experience same after deleting the cache folder and allowing it to get re-created?

Note: When launching CAD, are there any popup windows (e.g. Solidworks links to the community page by default) that require a human to do something until it times out and goes ahead?  The checkmarks, etc. to say "don't show again" are by user and may not apply to the user called by auth.properties.

[Dobi]

Maddening is right. 

You bring up a couple of good points to look at still though. 

The CAD workers are on separate servers, yes and I did log in as a different user than what's in auth.properties. Will track that down. 

Reboots don't seem to change the behavior and neither does starting with a clean cache. I do have a good bit of cache tidying in the XXXworker.bat scripts to have the starts be as clean as can be so I don't think that's the problem. 

There aren't popups that hang up NX. Really, once the file is downloaded the rest of the process is smooth (ish... some of our files are very large both in terms of BOM count and file size so the conversions sometime take a bit but are less than half of the publish duration). But again, this is with my user account. 

I guess to be fair to the post, I asked for troubleshooting ideas and in that sense of the ask Tom's direction was the bump I needed. I'll post any additional findings here.