Windchill Content Replication

RahulK · ‎Sep 17, 2009

All,

We are planning content replication for Windchill PDMLink data. This
replication is primarily for non cad content (PDFs, postscripts, docs) .
Currently all the content data is vaulted in the US and we need to
replicate this data at India site (which is part of our WAN). I have
gone through PTC's documentation on replication and would like to hear
more about replication from people who have implemented it.

If you have done data replication at your site, can you share your
knowledge on:

1. How the setup was done. (per PTC sys admin guide or any
variation)
2. How to determine server sizing for replication server. Are there
any guidelines?
3. How stable is the solution ?
4. Any other input.

We are on PDMLink 8.0 M050.

Thank you,

Rahul Kharkar

Manager - PLM

Novellus Systems Inc

achukwuka · ‎Sep 17, 2009

Hi Rahul,

We have implemented replication at three sites, France, Inida and a location here in the US. The procedures I used is called local replication which entails initially replicating the files locally here in the US, then copy the files to USB drive, and then ship the USB drive to the appropriate location. Once the USB arrived there, I then copied the contents to teh replica server maintaining the same folder structure. Additionally, I then went into the site configuration UI and updated the Site name from the local replica server to the one in India. This procedure is supported by PTC GSO. I used this procedure as we had over 1.5 TB of data to replica and I did not want to do it over the WAN. It worked out very well for us and all the site are functioning correctly today. Interestingly, each of this site has a myriad of severs and OS platforms, (France - IBM AIX, India Windows, Coffeyville.KS SUN Solaris). The replica needs only minimum amount of resource as it only holds files and all processing are done on the master, the only thing is that you will need huge amount of space depending on the amount of files you will be replicating. We are also on PDMLink 8.0 M050.

HTH,
Regards,

Alexius C. Chukwuka
IT Analyst, PDP Systems
John Deere Power Systems
Product Engineering Center
*Voice: 319-292-8575
*Mobile: 319-429-5336
*FaxFax:319-292-6282
*E-Mail: -

CONFIDENTIALITY. This electronic mail and any files transmitted with it may contain information proprietary to Deere & Company, or one of its subsidiaries or affiliates, and are intended solely for the use of the individual or entity to whom they are addressed, shall be maintained in confidence and not disclosed to third parties without the written consent of the sender. If you are not the intended recipient or the person responsible for delivering the electronic mail to the intended recipient, be advised that you have received this electronic mail in error and that any use, dissemination, forwarding, printing, or copying of this electronic mail is strictly prohibited. If you have received this electronic mail in error, please immediately notify the sender by return mail.

RandyJones · ‎Sep 17, 2009

In Reply to Stephen Drzewiczewski:

One annoyance is that you can't delete unreferenced files from the
replica vaults. I think it's comin I. 9.1 or X-20, but as of 9 it's
not directly possible.

In 9.1 (M020) this is possible.

Site | Utilities | External Storage Administrator | Vault Configuration

Select the replica vault in question and then "Object | Remove unreferenced files"

RandyJones · ‎Sep 17, 2009

In Reply to Alexius Chukwuka:

Hi Rahul,

We have implemented replication at three sites, France, Inida and a location here in the US. The procedures I used is called local replication which entails initially replicating the files locally here in the US, then copy the files to USB drive, and then ship the USB drive to the appropriate location. Once the USB arrived there, I then copied the contents to teh replica server maintaining the same folder structure. Additionally, I then went into the site configuration UI and updated the Site name from the local replica server to the one in India. This procedure is supported by PTC GSO. I used this procedure as we

Exactly what did you update in order to accomplish this local replication. There are several places in which a hostname is entered to define hosts, urls, and mounts. There is the following:

Site | Utilities | File Server Administrator | Site Administration - in which you enter a hostname as part of a url

Site | Utilities | File Server Administrator | Vault Configuration - in which you define hosts/vaults/root folders/mounts by:

- enter hostname(s) for site(s)

- choose a hostname for a vault

- select a hostname and enter a path for root folders.

- and probably something else I have overlooked here...

had over 1.5 TB of data to replica and I did not want to do it over the WAN. It worked out very well for us and all the site are functioning correctly today. Interestingly, each of this site has a myriad of severs and OS platforms, (France - IBM AIX, India Windows, Coffeyville.KS SUN Solaris). The replica needs only minimum amount of resource as it only holds files and all processing are done on the master, the only thing is that you will need huge amount of space depending on the amount of files you will be replicating. We are also on PDMLink 8.0 M050.

HTH,
Regards,

Alexius C. Chukwuka
IT Analyst, PDP Systems
John Deere Power Systems
Product Engineering Center
*Voice: 319-292-8575
*Mobile: 319-429-5336
*FaxFax:319-292-6282
*E-Mail: -

CONFIDENTIALITY. This electronic mail and any files transmitted with it may contain information proprietary to Deere & Company, or one of its subsidiaries or affiliates, and are intended solely for the use of the individual or entity to whom they are addressed, shall be maintained in confidence and not disclosed to third parties without the written consent of the sender. If you are not the intended recipient or the person responsible for delivering the electronic mail to the intended recipient, be advised that you have received this electronic mail in error and that any use, dissemination, forwarding, printing, or copying of this electronic mail is strictly prohibited. If you have received this electronic mail in error, please immediately notify the sender by return mail.

Thanks

achukwuka · ‎Sep 17, 2009

Hi Randy,

The mount needs to be repointed to in the UI to whatever it will be at the remote site. Also, I updated the URL in the Site Administrator UI to point to the remote replica server in India, initially it was pointing to the local replica server.

So in PDMLink 8.0 M050 you to go

Site | Utilities | Replication Administartor | Site Administrator

Once the site management UI comes up, just select the local replica site and click update, then change the <server-name> portion of the URL to that of the remote replica server. Once this is done, then update your mounts if they are different from the one on the local replica server. This can be done by going to

Site | Utilities | Replication Administartor | Vault Configuration

Once the vault configuration UI comes up, select the local replica site and update the hosts and folder mounts. When you are done perform Broadcast conmfiguration" to ensure communication is going through between the master and the replica.

From your note below, it seems that you are using 9.xxx and I have not done this in any version of 9.xx yet only in 8.0 M050. The procedure might be the same for 9.x, but I have not investigated that yet.

Hope this helps.

Alexius C. Chukwuka
IT Analyst, PDP Systems
John Deere Power Systems
Product Engineering Center
*Voice: 319-292-8575
*Mobile: 319-429-5336
*FaxFax:319-292-6282
*E-Mail: -

CONFIDENTIALITY. This electronic mail and any files transmitted with it may contain information proprietary to Deere & Company, or one of its subsidiaries or affiliates, and are intended solely for the use of the individual or entity to whom they are addressed, shall be maintained in confidence and not disclosed to third parties without the written consent of the sender. If you are not the intended recipient or the person responsible for delivering the electronic mail to the intended recipient, be advised that you have received this electronic mail in error and that any use, dissemination, forwarding, printing, or copying of this electronic mail is strictly prohibited. If you have received this electronic mail in error, please immediately notify the sender by return mail.

RandyJones · ‎Sep 18, 2009

I don't know if it is appropriate to keep adding on to this thread since Rahul was the original poster however the questions I have are directly related to Rahul's inquiry.

In Windchill 9.1 M020 I have setup a master site and several remote sites. I am attempting to do the following:

1. initial local replication

2. haul the locally replicated disk drive to remote site

3. copy files to remote server

4. change hostname and paths as necessary in site administrator and vault administrator

I am currently at step 1 with the initial local replication. The (local) replication job has been running for 12hrs now and has only managed to replicate 130000 files (42.4GB) so far. To me this is extremely paltry performance as compared to using something like rsync. Using rsync I would expect to replicate this file system in a couple of hours over the lan. I have 2 questions that hopefully someone can shed some light:

1. The obvious question: Has anybody figured out how to speed up the initial local replication? Any magic windchill properties to set?

2. Not so obvious question: How to I tell how many items are remaining to be replicated for a particular replication job? If viewing the replication jobs (Site | Utilities | File Server Administrator | Replication Schedule Administrator | View Replication Jobs) you get a handy screen that includes the columns titled "Start Time", "Items Completed", and "Total Items". However after refreshing this screen a few times one sees that the "Total Items" column is not really a "Total Number of Items that will be replicated when this job is finished". The "Total Items" that is displayed is always a few more than the "Items Completed" and is a constantly increasing number as the job progresses. So as far as I can tell the "Total Items" column is a useless piece of information.

One item I would like to caution other "future initial replicators" on is this: When replicating Windchill is doing this:

1. copies the files to WC_INSTALL/tmp/wrmfTransport on remote site

2. copies the files from WC_INSTALL/tmp/wrmfTransport to the file vault

3. does NOT delete the file copied in step 1

What this means is that you are not only using space in your file vault you are also using space (large amounts) in the file system that WC_INSTALL/tmp/wrmfTransport resides in. So make sure you have enough space or have a cronjob or script running that deletes files in WC_INSTALL/tmp/wrmfTransport that are older than 1hr for example. This leads to 2 observations:

- why doesn't Windchill delete the files in WC_INSTALL/tmp/wrmfTransport

- or better yet... why not put the files in the intended file vault in the first place instead of under the tmp directory

Thanks

--

achukwuka · ‎Sep 18, 2009

Hi Randy,

your first question..

1. The obvious question: Has anybody figured out how to speed up the initial local replication? Any magic windchill properties to set?

There is a property called "wt.fv.revaultQuerySize" In my environment this is currently set to 10000. This means it grab a chunk of 10000 at a time. This might help you a bit,
not sure if there are drawbacks to setting it higher. PTC recommended that I set this when I complained about performance.

For your second question, I am sure how you tell the total number of objects to ge replicated. I am sure there must be sql script to tell that. Just as you said the number you see in the UI is not the total number of files to be replicated. It just a partial number indicating teh number of files in the replication queue waiting to get moved to the vault on the replica. Also windchill does not delete the files during replication it merely does a copy and paste. You will need to run the "Removed Unreferenced files" to clean things up, additionally it might be worth it to also do a database re-org especially after a revault process.

Hope this helps,

Regards,

Alexius C. Chukwuka
IT Analyst, PDP Systems
John Deere Power Systems
Product Engineering Center
*Voice: 319-292-8575
*Mobile: 319-429-5336
*FaxFax:319-292-6282
*E-Mail: -

CONFIDENTIALITY. This electronic mail and any files transmitted with it may contain information proprietary to Deere & Company, or one of its subsidiaries or affiliates, and are intended solely for the use of the individual or entity to whom they are addressed, shall be maintained in confidence and not disclosed to third parties without the written consent of the sender. If you are not the intended recipient or the person responsible for delivering the electronic mail to the intended recipient, be advised that you have received this electronic mail in error and that any use, dissemination, forwarding, printing, or copying of this electronic mail is strictly prohibited. If you have received this electronic mail in error, please immediately notify the sender by return mail.

----------

wneuman · ‎Sep 18, 2009

All - not directly related to the question at hand, but you might want
to visit the Windchill Install and Config Resource Page
<">https://www.ptc.com/appserver/cs/view/kdb.jsp?im_dbkey=90970> and
navigate down to the tech brief on WAN Accelerators
<">http://www.ptc.com/view?im_dbkey=92390> . They can definitely have a
significant positive impact on performance, although an initial sync is
probably not going to get as much benefit.

Best Regards,

Bill Neuman

Director, Windchill Infrastructure and Integrations

+1-781-370-5565 (Office)

+1-508-507-0745 (Mobile)

+1-781-707-0470 (Fax)

RandyJones · ‎Sep 30, 2009

I am looking at all options in order to improve the performance of the initial replication. As describe on pages 181 and 182 of the WCSysAdminGuide the following 2 properties can be set:

wt.fv.disableReplContentCopy=true
wt.fv.logReplItemInTempFile=true

With these 2 properties set to true Windchill won't actually copy any files during a replication but instead write a log file that (sort of) contains what would have been copied. This is similar to migration when setting migration properties doFileTransfer=false and migration.filetransfer.mapFileHome=/some/directory.

Sample output from a Windchill replication log:

494321_/export/wc_vaults/master/corporate_master_Folder_1_/export/wc_vaults/replica_local/tipton_replica_Folder_1
494320_/export/wc_vaults/master/corporate_master_Folder_1_/export/wc_vaults/replica_local/tipton_replica_Folder_1
494319_/export/wc_vaults/master/corporate_master_Folder_1_/export/wc_vaults/replica_local/tipton_replica_Folder_1
494325_/export/wc_vaults/master/corporate_master_Folder_1_/export/wc_vaults/replica_local/tipton_replica_Folder_1
494324_/export/wc_vaults/master/corporate_master_Folder_1_/export/wc_vaults/replica_local/tipton_replica_Folder_1

As you can see there are 3 "fields" in this file: an "id", the master folder, and the replica folder. What we are missing in order to manually copy files (ie like migration) is:

a. the name of the file to copy from the master folder

b. the name of the file to create in the replica folder

I have figured out that the "id" is the streamid field in the fvitem table. However after examining the fvitem table it is not apparent to me on how to get the missing 2 pieces of information. The fvitem table is this:

SQL> describe fvitem;

Name Null? Type

----------------------------------------- -------- ----------------------------

CLASSNAMEKEYA4 VARCHAR2(600)

 IDA3A4 NUMBER

QUARANTINED NUMBER(1)

REVAULTSESSIONID NUMBER

STREAMID NUMBER

CREATESTAMPA2 DATE

MARKFORDELETEA2 NOT NULL NUMBER

MODIFYSTAMPA2 DATE

CLASSNAMEA2A2 VARCHAR2(600)

IDA2A2 NOT NULL NUMBER

UPDATECOUNTA2 NUMBER

UPDATESTAMPA2 DATE

UNIQUESEQUENCENUMBER NUMBER


SQL>

Does anybody have any idea on a query to run that would resolve the data in the fvitem table into a "file that needs copied name" and a "file that needs created name"?

If I had these 2 missing pieces of information it would be easy to script the copying of the files for initial replication.

Thanks

--