WC 13.0.2.0 Solr tmp folder filling up after 12-16

[mwaite]

Version: Windchill 13.0

Use Case: Bulk indexing millions of files runs the drive out of disk space

Description:

We are in the process of upgrading to WC 13.0.2.0. We are in the post upgrade stage and we bulk index all of our files. We have 2M+ files and it takes a few weeks to index them all.

Everything was running fine until 12-16. On 12-16 we rebooted the server for Windows patching . The Solr tmp folder has been filling up with files since then.

Our path looks like this:

E:\ptc\IndexSearchServer\SolrServer\solr\server\tmp\jetty-0_0_0_0-8085-webapp-_solr-any-12436573060378850937

I

t seems like Solr creates a new tmp/jetty-0_0_0_0-8085-webapp-_solr-any-XXXXXXXX folder every time it starts.  Sometimes it deletes the old jetty folders and sometimes it doesn't.

We find the current jetty folder is filling up with files as they get indexed.  The Solr Log shows an error saying it could not delete the file.

I found Article - CS433080 which describes the problem but offers no solution.  It was created on 12-15-2024. 

I opened a Support Request with PTC and they told me to delete the files manually.

That might be OK on a test server where downtime is acceptable, but not in production.

Has anyone else noticed the problem?
Any solutions?

[avillanueva]

This seems like a similar problem to this:

https://stackoverflow.com/questions/56947094/jetty-running-as-windows-service-doesnt-clean-up-temp-file-after-shutdown

The fact that PTC is having R&D look at it is a good sign. Hopefully they come back with a resolution. This sounds like tmp file so there should not be a big deal in deleting them after restarts. From the link above, it seems like jetty is unpacking itself so on restarts, it starts all over again.

[mwaite]

The tmp/jetty-0_0_0_0-8085-webapp-_solr-any-XXXXXXXX folder contains copies of the files being indexed.  Solr copies them to its tmp folder, indexes them and then tries to delete them.  The problem is that it can't delete them.  This is the error:

2024-12-19 19:46:24.751 WARN (qtp709133385-8449) [c: s: r: x: t:WVTWCT01.gd-ots.NET-221812] o.a.s.s.SolrRequestParsers Errors deleting multipart tmp files => java.io.IOException: Could Not Delete File
at org.eclipse.jetty.server.MultiPartFormInputStream$MultiPart.delete(MultiPartFormInputStream.java:331)
java.io.IOException: Could Not Delete File

[avillanueva]

Notes indicated this might be exclusively a Windows problem. I am curious where SOLR on RHEL stores temp files. I cannot find anything that jumps out so I do not think its an issue for us though I am on the version supplied with WC 12.

[mwaite]

We are running 12.0.2 in production, and it was not affected.  Only our 13.0 systems are affected.

[rhart]

I hadn't noticed it, but I started bulk index after seeing your post.

The tmp file is growing with 'Caused by: java.io.IOException: Could Not Delete File' in solr\server\log

Windchill 13.0.2.0

Solr 9.5.0

Server jetty-10.0.19

[mwaite]

PTC sent me a note telling me that more customers are reporting the issue.  The support engineer said R&D is looking into it.  

[RandyJones]

Might be caused by SOLR_SECURITY_MANAGER_ENABLED environment variable.

We are running on Linux here and install solr under /opt/ptc/Windchill/Solr. This makes the default install of the data directory:

/opt/ptc/Windchill/Solr/SolrServer/solr/data

We always change that to:

/opt/ptc/Windchill/solrdata

• Which is a separate zfs filesystem
• keeps the data separate from the install
• different backup rules can be applied
• also very handy now with: https://www.ptc.com/en/support/article/CS355758

Before Windchill 12.1.2.5 this was never an issue. When we were installing the cps update 12.1.2.5 (from 12.1.2.4) the Solr installer was failing. It turns out that solr was refusing to create files outside of the /opt/ptc/Windchill/Solr/SolrServer/solr directory.

Similar issue google link here:

https://stackoverflow.com/questions/76169675/how-to-set-solr-index-data-path-version-9-2-0

The "solution" or workaround was to set SOLR_SECURITY_MANAGER_ENABLED=false and then run the psi installer:

export SOLR_SECURITY_MANAGER_ENABLED=false

This allows the installer to run but then attempting to run solr after words failed because of the same issue. So the "solution" or work around for that is to edit  Solr/SolrServer/solr/bin/solr.in.sh and add a line:

SOLR_SECURITY_MANAGER_ENABLED=false

We reported this issue to PTC thinking they might fix the installer to support the data directory outside of the Solr/SolrServer/solr parent.

So as a workaround you might try setting SOLR_SECURITY_MANAGER_ENABLED=false in:

Solr/SolrServer/solr/bin/solr.in.cmd (for Windows)