1.7 million files in defaultcachevault, possible solution

[ptc-2210193]

Let me begin by saying that I have searched for this particular topic, and came up empty handed. I hope someone else has at least thought of this as an option, but as far as I know, we are the first to put into practice. Ok, so, this is my first post here (of many, I hope). I'm a new Windchill Admin in the Midwest (Quad Cities, IA/IL), and I was presented with a problem that we had from the initial setup of our system. I 'inherited' the system from a sys admin who retired in December 2008, and I ran the system as best I could while I was going to school full time until I graduated in May 2009. We that in November we had 1.4million files in DefaultCacheVault. The way the system was set up, as I understand it, we did not anticipate having so many files in one place. Once it was out of control at 1.2million or so, we were in deep. I ended up adding more folders in June 2009, setting DefaultCacheVault to read only, and limiting files to 50k per folder. We looked into revaulting the system, and tried it on our original test system. It failed by starting then stopping after moving about 10,000 files Right now we are looking at running purges, but those have been unable to run on our new test environment, so we proceeded on a different path...My boss had suggested "Figuring out the tables, and updating the pointers to the file...it can't be that hard it IS a relational database". I won't go into details on this just yet, but I believe that we have done this. So we've done that (on our test system). A colleague of mine figured out how to return the hex file number when inputting a document, and from that I backed into how to get file lists and copy them out, update the pointers and finally update the database. I have written a program (Java) that does this in 50,000 file increments, first querying the DB, then making a list and reading from it, then copying the files out to their new location, finally updating the database. Note I did not say that we are removing the files yet from their old location. Instead after the entire move and update was done, I renamed the folder to ensure that nothing was still pointing to it. I am now looking for ways to test this. I believe that I can pass WinDU tests without regard to where the files are because it doesn't check that. I am able to check files in and out, revise them, bump the revision, change the iteration state, etc. Soon I will be having users connect to the test system so that they can do the kinds of things they normally do. If this is successful on our test server (which so far we have moved all 1.7million files elsewhere and it seems to work), then we will do this on production. If we do this on production then we will also keep the original folder with 1.7m files in it for a certain period of time, as it would only take an SQL update and commit to change the pointers back to their original location (after renaming the folder to defaultcachevault).Does anybody have any thoughts, tips, or ideas that might help out how to test this solution? I will be unable to provide the code for now, but if this works there MAY be software available to do this for almost anywhere (I'm working with my company to make it free, with no warranty, support...nothing, but hey...it would be FREE!) and trying to make it have a decent front-end that would work for most places.Steve Jones

[wpalm_99]

Steve,
I've done pretty much the same process you describe at a few customers. If
you are using Windows, I also recommend changing a couple of OS settings
that almost make this a non-issue.

fsutil behavior set disable8dot3 1
fsutil behavior set disablelastaccess 1

You can see the current values with the query option for both...
fsutil behavior query disable8dot3
fsutil behavior query disablelastaccess

The reason I say "almost" is that browsing and listing this folder on
Windows or Unix would have a severe performance impact while the OS tries to
display the million+ files. Because of this potential, I recommend using
file thresholds to control this even with these fsutil settings in place.

Also, I've commented on this before on the solutions list. You can probably
search for "disable8dot3" on ptc/user to find the old, more detailed, post.

[ptc-2210193]

Just to clarify for all: We are on PDMLink/ProjectLinkWC8.0 M040. We will eventually be going to 9.1.

Steve

[ptc-2210193]

To update all, I have recieved confirmation from PTCthat what we are planning on doing with our production enviroment with the modification of tables plus copy of files to a new location will work. I am currently sorting through the code as one final check, but this DOES indeed work on our test environment. I will post back when I have confirmation.

Special thanks toone user who sent me an e-mail letting me know that what we had planned is very similar to what PTC did for them. He gave me his call number and this put me in touch with someone at PTC who was able to confirm what we had planned. It was indeed the exact same procedure this user had done, though myprogram is more automated.

Thanks,

Steve Jones

[ptc-2210193]

To update everybody:
I've completed this on our production environment. It isn't very elegant code, but it works.

I am working on a guide using the software that I have written to do this, and working with the company I work for to make the software free and public. Right now I am working with a 'beta tester' of sorts, someone who has helped us, and now we are helping them. I will also be working on a presentation to describe how we did what we did and hope to present it at a regional group (or larger) conference.

This isn't for everybody, least of all the faint of heart. However, if you are like we were (and couldn't 'revault')

Also, I appreciate the feedback on changing the 8.3 file names, and I might have just done that, but we don't have an exact replica test system and I was unsure of the possible outcomes.

So, stay tuned....

Steve Jones