ildataloader FileTransfer

Robert-Altman · ‎Sep 06, 2012

So my test migration from 3.4 to 10.0 has been going as well as can be
expected.

At the point of running "ildataloader FileTransfer 1" and get a constant
flow of error messages for what I'll assume is every file in the vault.

I used settings for "PreCopy" with the location of c:\temp\vault

06/09/2012 19:53:50 WARN loadercontroller.FileLoader
- [vrm003.arthrex.local: 1131]: FileTransfer failed for DOCID{341845}
with ERRID{401}.

06/09/2012 19:53:50 WARN loadercontroller.FileLoader
- [vrm003.arthrex.local: 1131]: WARNING !!! Cannot
rename{c:\temp\vault/c/vault/160491.drw},
to{c:/vault/folder001_Folder_1/0000000001444d}. Continue to next file.

06/09/2012 19:53:50 WARN loadercontroller.FileLoader
- [vrm003.arthrex.local: 1131]: FileTransfer failed for DOCID{290484}
with ERRID{401}.

06/09/2012 19:53:50 WARN loadercontroller.FileLoader
- [vrm003.arthrex.local: 1131]: WARNING !!! Cannot
rename{c:\temp\vault/c/vault/159497.drw},
to{c:/vault/folder001_Folder_1/0000000001430b}. Continue to next file.

06/09/2012 19:53:50 WARN loadercontroller.FileLoader
- [vrm003.arthrex.local: 1131]: FileTransfer failed for DOCID{164607}
with ERRID{401}.

06/09/2012 19:53:50 WARN loadercontroller.FileLoader
- [vrm003.arthrex.local: 1131]: WARNING !!! Cannot
rename{c:\temp\vault/c/vault/159991.drw},
to{c:/vault/folder001_Folder_1/00000000014427}. Continue to next file.

06/09/2012 19:53:50 WARN loadercontroller.FileLoader
- [vrm003.arthrex.local: 1131]: FileTransfer failed for DOCID{290481}
with ERRID{401}.

06/09/2012 19:53:50 WARN loadercontroller.FileLoader
- [vrm003.arthrex.local: 1131]: WARNING !!! Cannot
rename{c:\temp\vault/c/vault/159965.prt},
to{c:/vault/folder001_Folder_1/0000000001441f}. Continue to next file.

06/09/2012 19:53:50 WARN loadercontroller.FileLoader
- [vrm003.arthrex.local: 1131]: FileTransfer failed for DOCID{343852}
with ERRID{401}.

06/09/2012 19:53:50 WARN loadercontroller.FileLoader
- [vrm003.arthrex.local: 1131]: WARNING !!! Cannot
rename{c:\temp\vault/c/vault/159699.prt},
to{c:/vault/folder001_Folder_1/0000000001437b}. Continue to next file.

06/09/2012 19:53:50 WARN loadercontroller.FileLoader
- [vrm003.arthrex.local: 1131]: FileTransfer failed for DOCID{343850}
with ERRID{401}.

06/09/2012 19:53:50 WARN loadercontroller.FileLoader
- [vrm003.arthrex.local: 1131]: WARNING !!! Cannot
rename{c:\temp\vault/c/vault/159588.asm},
to{c:/vault/folder001_Folder_1/000000000142ff}. Continue to next file.

06/09/2012 19:53:50 WARN loadercontroller.FileLoader
- [vrm003.arthrex.local: 1131]: FileTransfer failed for DOCID{599626}
with ERRID{401}.

Would anyone have any thought where I may have configured something
incorrectly?

Regards,

RandyJones · ‎Sep 06, 2012

On 09/06/12 15:12, Robert Altman wrote:
>
> So my test migration from 3.4 to 10.0 has been going as well as can be expected.
>
> At the point of running "ildataloader FileTransfer 1" and get a constant flow of error messages for what I'll assume is every file in the vault.
>
> I used settings for "PreCopy" with the location of c:\temp\vault
>
> 06/09/2012 19:53:50 WARN loadercontroller.FileLoader - [vrm003.arthrex.local: 1131]: FileTransfer failed for DOCID{341845} with ERRID{401}.
>
> 06/09/2012 19:53:50 WARN loadercontroller.FileLoader - [vrm003.arthrex.local: 1131]: WARNING !!! Cannot rename{c:\temp\vault/c/vault/160491.drw}, to{c:/vault/folder001_Folder_1/0000000001444d}. Continue to next file.
>
> 06/09/2012 19:53:50 WARN loadercontroller.FileLoader - [vrm003.arthrex.local: 1131]: FileTransfer failed for DOCID{290484} with ERRID{401}.
>
> 06/09/2012 19:53:50 WARN loadercontroller.FileLoader - [vrm003.arthrex.local: 1131]: WARNING !!! Cannot rename{c:\temp\vault/c/vault/159497.drw}, to{c:/vault/folder001_Folder_1/0000000001430b}. Continue to next file.
>
> 06/09/2012 19:53:50 WARN loadercontroller.FileLoader - [vrm003.arthrex.local: 1131]: FileTransfer failed for DOCID{164607} with ERRID{401}.
>
> 06/09/2012 19:53:50 WARN loadercontroller.FileLoader - [vrm003.arthrex.local: 1131]: WARNING !!! Cannot rename{c:\temp\vault/c/vault/159991.drw}, to{c:/vault/folder001_Folder_1/00000000014427}. Continue to next file.
>
> 06/09/2012 19:53:50 WARN loadercontroller.FileLoader - [vrm003.arthrex.local: 1131]: FileTransfer failed for DOCID{290481} with ERRID{401}.
>
> 06/09/2012 19:53:50 WARN loadercontroller.FileLoader - [vrm003.arthrex.local: 1131]: WARNING !!! Cannot rename{c:\temp\vault/c/vault/159965.prt}, to{c:/vault/folder001_Folder_1/0000000001441f}. Continue to next file.
>
> 06/09/2012 19:53:50 WARN loadercontroller.FileLoader - [vrm003.arthrex.local: 1131]: FileTransfer failed for DOCID{343852} with ERRID{401}.
>
> 06/09/2012 19:53:50 WARN loadercontroller.FileLoader - [vrm003.arthrex.local: 1131]: WARNING !!! Cannot rename{c:\temp\vault/c/vault/159699.prt}, to{c:/vault/folder001_Folder_1/0000000001437b}. Continue to next file.
>
> 06/09/2012 19:53:50 WARN loadercontroller.FileLoader - [vrm003.arthrex.local: 1131]: FileTransfer failed for DOCID{343850} with ERRID{401}.
>
> 06/09/2012 19:53:50 WARN loadercontroller.FileLoader - [vrm003.arthrex.local: 1131]: WARNING !!! Cannot rename{c:\temp\vault/c/vault/159588.asm}, to{c:/vault/folder001_Folder_1/000000000142ff}. Continue to next file.
>
> 06/09/2012 19:53:50 WARN loadercontroller.FileLoader - [vrm003.arthrex.local: 1131]: FileTransfer failed for DOCID{599626} with ERRID{401}.
>
> Would anyone have any thought where I may have configured something incorrectly?
>

I would recommend you use copy, set mapFileHome/fileTransferOption/withMapFile/doFileTransfer, and then copy the files with a script. The "ildataloader FileTransfer" stage will run much faster when using copy than precopy. This is what I did:

migration.filetransfer.mapFileHome=/opt/ptc/ILMigration/mapfilehome
fileTransferOption=Copy
withMapFile=true
doFileTransfer=false

Doing it this way the migrator will create a text file that tells you what file to copy where and it is trivial to create a script to do the copying.

The contents of the file mapping file looks like this:
# This is a csv file mapping the Intralink (Source) file location to PDMLink External Storage (destination) location
# Format of the map file:
#<pro intralink=" file=" server=" host=" name=">, <port>, <pro intralink=" vault=" directory=">, <pro intralink=" file=" name=">, <pdmlink fileserver=" host=">, <pdmlink vault=" directory=">, <pdmlink file=" name=">
server3,7777,/export/il_vault1/assaria/master1,3175.prt,wc.greatplainsmfg.com,/opt/ptc/Windchill/vaults/master/corporate_master_Folder_1,000000000000fd
server3,7777,/export/il_vault1/assaria/master1,3140.prt,wc.greatplainsmfg.com,/opt/ptc/Windchill/vaults/master/corporate_master_Folder_1,000000000000fc
server3,7777,/export/il_vault1/assaria/master1,3141.prt,wc.greatplainsmfg.com,/opt/ptc/Windchill/vaults/master/corporate_master_Folder_1,000000000000fb
server3,7777,/export/il_vault1/assaria/master1,3176.drw,wc.greatplainsmfg.com,/opt/ptc/Windchill/vaults/master/corporate_master_Folder_1,000000000000fa
server3,7777,/export/il_vault1/assaria/master1,3139.drw,wc.greatplainsmfg.com,/opt/ptc/Windchill/vaults/master/corporate_master_Folder_1,000000000000fe
server3,7777,/export/il_vault1/assaria/master1,3239.drw,wc.greatplainsmfg.com,/opt/ptc/Windchill/vaults/master/corporate_master_Folder_1,00000000000100
server3,7777,/export/il_vault1/assaria/master1,3242.prt,wc.greatplainsmfg.com,/opt/ptc/Windchill/vaults/master/corporate_master_Folder_1,00000000000102
.
.
.

I wrote a sh script that creates 2 other scripts. One to check to see if the file to be copied actually existed and one to actually do the copying. The contents of this script which I called create_copy_script is here:
==========================================================================================
#!/bin/sh
#rsj 08/25/2009
#quicky hack to make copy scripts to copy files from Intralink vaults to WC Vaults.
#also makes check_files script to check the existence of the files
#This makes the assumption that the target host is always the same as the
#source host

echo '#!/bin/sh' > copy_files
grep -v '^#' fileMap_wc_greatplainsmfg_com.csv | awk -F, '{printf("cp -p /il_vaults/%s%s/%s %s/%s\n", $1, $3, $4, $6, $7)}' >> copy_files
chmod a+x copy_files

echo '#!/bin/sh' > check_files
grep -v '^#' fileMap_wc_greatplainsmfg_com.csv | awk -F, '{printf("if [ ! -f \"/il_vaults/%s%s/%s\" ]; then echo /il_vaults/%s%s/%s not found; fi\n", $1, $3, $4, $1, $3, $4)}' >> check_files
chmod a+x check_files
==========================================================================================

This created the check_files and the copy_files scripts which are basically one line for each file.

First few lines of the check_files script:
==========================================================================================
#!/bin/sh
if [ ! -f "/il_vaults/server3/export/il_vault1/assaria/master1/3175.prt" ]; then echo /il_vaults/server3/export/il_vault1/assaria/master1/3175.prt not found; fi
if [ ! -f "/il_vaults/server3/export/il_vault1/assaria/master1/3140.prt" ]; then echo /il_vaults/server3/export/il_vault1/assaria/master1/3140.prt not found; fi
if [ ! -f "/il_vaults/server3/export/il_vault1/assaria/master1/3141.prt" ]; then echo /il_vaults/server3/export/il_vault1/assaria/master1/3141.prt not found; fi
if [ ! -f "/il_vaults/server3/export/il_vault1/assaria/master1/3176.drw" ]; then echo /il_vaults/server3/export/il_vault1/assaria/master1/3176.drw not found; fi
if [ ! -f "/il_vaults/server3/export/il_vault1/assaria/master1/3139.drw" ]; then echo /il_vaults/server3/export/il_vault1/assaria/master1/3139.drw not found; fi
if [ ! -f "/il_vaults/server3/export/il_vault1/assaria/master1/3239.drw" ]; then echo /il_vaults/server3/export/il_vault1/assaria/master1/3239.drw not found; fi
if [ ! -f "/il_vaults/server3/export/il_vault1/assaria/master1/3242.prt" ]; then echo /il_vaults/server3/export/il_vault1/assaria/master1/3242.prt not found; fi
if [ ! -f "/il_vaults/server3/export/il_vault1/assaria/master1/3245.asm" ]; then echo /il_vaults/server3/export/il_vault1/assaria/master1/3245.asm not found; fi
if [ ! -f "/il_vaults/server3/export/il_vault1/assaria/master1/3243.prt" ]; then echo /il_vaults/server3/export/il_vault1/assaria/master1/3243.prt not found; fi
if [ ! -f "/il_vaults/server3/export/il_vault1/assaria/master1/3840.drw" ]; then echo /il_vaults/server3/export/il_vault1/assaria/master1/3840.drw not found; fi
if [ ! -f "/il_vaults/server3/export/il_vault1/assaria/master1/3837.drw" ]; then echo /il_vaults/server3/export/il_vault1/assaria/master1/3837.drw not found; fi
if [ ! -f "/il_vaults/server3/export/il_vault1/assaria/master1/3258.drw" ]; then echo /il_vaults/server3/export/il_vault1/assaria/master1/3258.drw not found; fi
if [ ! -f "/il_vaults/server3/export/il_vault1/assaria/master1/3852.prt" ]; then echo /il_vaults/server3/export/il_vault1/assaria/master1/3852.prt not found; fi
if [ ! -f "/il_vaults/server3/export/il_vault1/assaria/master1/3856.prt" ]; then echo /il_vaults/server3/export/il_vault1/assaria/master1/3856.prt not found; fi
.
.
.
==========================================================================================

First few lines of the copy_files script:
==========================================================================================
#!/bin/sh
cp -p /il_vaults/server3/export/il_vault1/assaria/master1/3175.prt /opt/ptc/Windchill/vaults/master/corporate_master_Folder_1/000000000000fd
cp -p /il_vaults/server3/export/il_vault1/assaria/master1/3140.prt /opt/ptc/Windchill/vaults/master/corporate_master_Folder_1/000000000000fc
cp -p /il_vaults/server3/export/il_vault1/assaria/master1/3141.prt /opt/ptc/Windchill/vaults/master/corporate_master_Folder_1/000000000000fb
cp -p /il_vaults/server3/export/il_vault1/assaria/master1/3176.drw /opt/ptc/Windchill/vaults/master/corporate_master_Folder_1/000000000000fa
cp -p /il_vaults/server3/export/il_vault1/assaria/master1/3139.drw /opt/ptc/Windchill/vaults/master/corporate_master_Folder_1/000000000000fe
cp -p /il_vaults/server3/export/il_vault1/assaria/master1/3239.drw /opt/ptc/Windchill/vaults/master/corporate_master_Folder_1/00000000000100
cp -p /il_vaults/server3/export/il_vault1/assaria/master1/3242.prt /opt/ptc/Windchill/vaults/master/corporate_master_Folder_1/00000000000102
cp -p /il_vaults/server3/export/il_vault1/assaria/master1/3245.asm /opt/ptc/Windchill/vaults/master/corporate_master_Folder_1/00000000000101
cp -p /il_vaults/server3/export/il_vault1/assaria/master1/3243.prt /opt/ptc/Windchill/vaults/master/corporate_master_Folder_1/000000000000ff
cp -p /il_vaults/server3/export/il_vault1/assaria/master1/3840.drw /opt/ptc/Windchill/vaults/master/corporate_master_Folder_1/00000000000109
cp -p /il_vaults/server3/export/il_vault1/assaria/master1/3837.drw /opt/ptc/Windchill/vaults/master/corporate_master_Folder_1/00000000000108
cp -p /il_vaults/server3/export/il_vault1/assaria/master1/3258.drw /opt/ptc/Windchill/vaults/master/corporate_master_Folder_1/00000000000107
cp -p /il_vaults/server3/export/il_vault1/assaria/master1/3852.prt /opt/ptc/Windchill/vaults/master/corporate_master_Folder_1/00000000000111
cp -p /il_vaults/server3/export/il_vault1/assaria/master1/3856.prt /opt/ptc/Windchill/vaults/master/corporate_master_Folder_1/00000000000110
.
.
.
==========================================================================================

I had 569196 files to copy. The check_files script took 2min 9sec to run and the copy_files script took 4hrs and 18min to run. However while the copy_files script is still running you can continue on with the rest of the migration ie ildataloader.sh MigrationStatus -report

> Regards,
>
> Description: cid:012445618@15012007-1FE4**
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> *Robert Altman***
>
> Manager, CAD Engineering & Tech Support
>
> Tel: (239) 643-5553 ext: 1300
>
>
>
> 1370 Creekside Blvd
>
> Toll-Free: (800) 933-7001
>
>
>
> Naples, FL 34108
>
> Fax: 239-598-5539
>
>
>
> E-Mail: - <">mailto:->
>
> This e-mail and any files transmitted with it are the property of Arthrex, Inc. and/or its affiliates, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to
> believe that you have received this message in error, please notify the sender at 239-643-5553 and delete this message immediately from your computer. Any other use, retention, dissemination forwarding, printing or copying of this e-mail is strictly prohibited. Please note that any views or
> opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Finally, while Arthrex uses virus protection, the recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any
> damage caused by any virus transmitted by this email.
> -----End Original Message-----

--
------------------------------------------------------------------------
Randy Jones
Systems Administrator
Great Plains Mfg., Inc.
1525 E North St
PO Box 5060
Salina, KS USA 67401
email: -
Phone: 785-823-3276
Fax: 785-667-2695
------------------------------------------------------------------------

dips52 · ‎Oct 03, 2012

Hi Rob,

I can't really give much input on the error messages you're getting however I do have a question, is there a specific reason why you used the pre-copy method instead of copy?

I have always installed the intralink NFS Fileserver on the PDMLink machine and used the copy option during migration and not had any issues.

Maybe it might be worth vising this option in the future (or if not too much work doing the test migration agian and trying this method?)

-=D