Rebuilding solr index

[rhart]

A recent CPS included a major solr server version upgrade 8.x.x to 9.x.x, so it’s recommended to rebuild the index but this took days in dev/test and the index search capability is out until it’s finished. Are there any ways to avoid the outage, perhaps run the two in parallel until the rebuild is done then handover, set the old one to read only somehow? Any suggestions 

[avillanueva]

Do you think things are tuned properly? I have been able to reindex over night but I've been told my system is on smaller end. What indexing rates are you seeing on Dev/Test and do you think Prod will be comparable?  Perhaps by comparing stats, field can determine if you may need to add resources. 

For comparison, Index Admin shows I have 900K objects indexed, 50K being documents with content.

[jbailey]

I just updated CPS and I see around 1M objects indexed in roughly 8 hours... @rhart  -When you go into the bulk index tool, and verify indexing status on your current build of SOLR... How many objects?

[mmeadows-3]

Just curious if anyone else has SOLR tuning tips.  This is what I do post-install (on Windows), standalone SOLR install.

:: Set Java Heap to 8 GB RAM for improved performance
:: Per CS272602 and CS244688
:: https://www.ptc.com/en/support/article/CS244800
:: File: SolrServer\bin\set_env.bat
REM SET SOLR_HEAP=2048m
SET SOLR_HEAP=8096m

:: Java Tuning
:: Per CS288920, CS151487, and CS203455
:: File: SolrServer\solr\bin\solr.in.cmd
set GC_TUNE=%GC_TUNE% -XX:ParallelGCThreads=2
set GC_TUNE=%GC_TUNE% -XX:ConcGCThreads=2
set GC_TUNE=%GC_TUNE% -XX:+DisableExplicitGC

[rhart]

I increased the HEAP and restarted the bulk index, should know the effect of that soon. I looked at the GC tuning but my solr server doesn't have this solr.in.cmd file mentioned in the articles for Windchill 11.0, these are already tuned in wt.properties but I can't tell if solr has used them.

I have some xconf to run and monitor the solr in the background separately, no idea if it helps or hinders the performance

<Property name="wt.manager.monitor.services" overridable="true"
targetFile="codebase/wt.properties"
value="MethodServer BackgroundMethodServer BackgroundWVS BackgroundSOLR"/>



<Property name="wt.manager.monitor.start.BackgroundSOLR" overridable="true"
targetFile="codebase/wt.properties"
value="1"/>



<!-- Start Background SOLR Settings -->  
   <Property name="wt.manager.cmd.BackgroundSOLR" overridable="true"
             targetFile="codebase/wt.properties"
             value="$(wt.manager.cmd.BackgroundMethodServer)"/>
   <Property name="wt.manager.cmd.BackgroundSOLRLauncher" overridable="true"
             targetFile="codebase/wt.properties"
             value="$(wt.manager.cmd.BackgroundMethodServerLauncher)"/>
   <Property name="wt.manager.cmd.BackgroundSOLR.singleton.enforce" overridable="true"
             targetFile="codebase/wt.properties"
             value="true"/>
   <Property name="wt.manager.cmd.BackgroundSOLR.param.4" overridable="true"
             targetFile="codebase/wt.properties"
             value="$(wt.method.bg.maxHeap)"/>
   <Property name="wt.manager.cmd.BackgroundSOLR.param.5" overridable="true"
             targetFile="codebase/wt.properties"
             value="NoQueues"/>
   <Property name="wt.manager.monitor.start.BackgroundSOLR" overridable="true"
             targetFile="codebase/wt.properties"
             value="1"/>
<!-- End BackgroundSOLR Settings -->  

[rleir]

@rhart 

Tell me if I am overdoing this.

The tips in Lucid's article are mostly things that PTC can build into Windchill, but you may be able to gain from reading these tips.

https://support.lucidworks.com/hc/en-us/articles/18186491585047-Apache-Solr-Indexing-Performance-Guide

Clearly you are better off re-indexing over the weekend when the machine is mostly idle.

Solr likes to have lots of memory, so check whether you are close to using swap. That would slow you down to a crawl.

It is possible to rebuild an index on a separate server, then install the new index into your production machine but this gets complex.

[rhart]

I don't know if the company is OK posting about that so I won't, anyway there's a lot more objects with content to index but they could be smaller/bigger content files, maybe if there was a way to see the count of tokens in the index that would be comparable