Not all purgable items are purged

[DarrenStorey]

As our vault size grows, and our backups take longer and longer, we have decided to give purge a try. So far it’s not going that well. The problem can be replicated by creating a purge collection rule, as follows

• Single product (all the data is isolated in the product)
• All folders
• All object types
• No restriction on versions to keep

When we run a purge job using this rule, some items are purged. Yes, we use baselines and promotions so I’m not expecting just the latest to remain, but I’m really surprised by how much stuff remains. In many cases no version of objects are purged at all. I find it strange for such a purge to results in <10% data removal. So strange that we ran another purge job using the same collection rule (i.e. on the same product data that had already been purged). Additional items were purged, which were not removed on the previous job. We can repeat this process many times, and on each subsequent job more items are purged, albeit a reducing number each time.

Surely all the items that are available to purge (not part of baselines) should get purged on the first attempt. I would not expect any subsequent purge jobs using the same collection to find any additional items to delete? I know you cannot delete a part or CAD doc with build rules in isolation, but if you hit the affected part and CAD Doc at the same time then it’s OK.

Can anybody comment on this?

[GaryMansell]

I am a colleague of Darren's and can add a bit more detail:

My suspicion is that this might be caused by the property wt.team.teamChunkSize (or the chunk size site level preference) which is set to 1000 on our System (as directed by the Windchill Archive/Purge Guide) - I suspect that it tries to purge items in groups of 1000 and if there are referential items outside of this group of 1000, then the related item in the chunk is not allowed to be purged. When you then run the same Purge rule again, there are differing items in each chunk and some more items are able to be purged each time. The trouble is that I have run the same purge rule 8 times now and it is still finding data to purge each time!!

I would not expect this behaviour, though - my understanding of purge is that you should only need to run it once and it should purge everything as per the purge rule specification (but in chunks).

Unfortunately setting wt.team.teamChunkSize back to -1 (unlimited) is not possible as then the purges fail because they hit the search query limit - even small purges seem to fall foul of this and I have boosted it from the recommended 350,000 to 1,000,000 to see if I can get a purge to complete.

Does anyone have an explanation for what we are seeing? Is it expected behaviour, or is something wrong here? How do we go about sensibly being able to purge data from our system (with Chunk Size set) without running a purge on a product context many many times?

Rgds

Gary

[jrozanc]

We've started our own Purge journey for the same reasons as your company. I am also seeing inconsistent results when running a purge with the same criteria on the same context between systems (DEV and QA). I feel your theory holds weight. I'm going to continue run passes on a selected context until the results end up being 0 after a few purges. That would add more weight behind this theory.