Skip to main content
avillanueva
23-Emerald I
23-Emerald I
April 11, 2025
Solved

SOLR STILL not starting on RHEL 9.2 as a service

  • April 11, 2025
  • 1 reply
  • 715 views

I've gone done a serious rabbit hole but failing to see my error. I am assuming it must be me. I have two instances and both experience same behavior. I can start and stop SOLR normally as my non-root user.  But when I configure my solr.service (which is identical to my previous installs), it fails to see that solr is running, exits and stops the process. I can verify that SOLR does indeed start but the part of the solr script  (called from Index_Search_Server.sh) where it checks that SOLR is running, I think is not detecting it. It times out after 180 seconds and stops. 

Here is what I observed:

  • Confirmed that SOLR starts from java process, PID file and logs.
  • Noticed "lsof -v" running while it was waiting for SOLR to start and consuming high CPU. Disappears when it bails after the time out. 
  • I identified the part of the solr script where it tries to verify SOLR is running. (rabbit hole). I can see that it checks the lsof version (where I see "lsof -v", and then runs it to verify to get the PID of the process running at port 8085. I can validate that these commands return normally when I ran them while waiting for startup script to timeout. 

avillanueva_0-1744378818292.png

I zeroed in on lsof since that is what I saw running and a while loop in the script. I gather that Index_Search_Server is calling solr but not getting a successful return. I even dug into the github for solr trying to see when this script change (here it is if you are curious https://github.com/apache/solr/commit/3ed17c2737d )

 

Here is the section where its stuck

 # no lsof on cygwin though
 if lsof -v 2>&1 | grep -q revision; then
 echo -n "Waiting up to $SOLR_START_WAIT seconds to see Solr running on port $SOLR_PORT"
 # Launch in a subshell to show the spinner
 (loops=0
 while true
 do
 running=$(lsof -t -PniTCP:$SOLR_PORT -sTCP:LISTEN || :)
 if [ -z "${running:-}" ]; then
 slept=$((loops * 2))
 if [ $slept -lt $SOLR_START_WAIT ]; then
 sleep 2
 loops=$((loops+1))
 else
 echo -e "Still not seeing Solr listening on $SOLR_PORT after $SOLR_START_WAIT seconds!"
 tail -30 "$SOLR_LOGS_DIR/solr.log"
 exit # subshell!
 fi
 else
 SOLR_PID=$(ps auxww | grep start\.jar | awk "/\-Djetty\.port=$SOLR_PORT/"' {print $2}' | sort -r)
 echo -e "\nStarted Solr server on port $SOLR_PORT (pid=$SOLR_PID). Happy searching!\n"
 exit # subshell!
 fi
 done) &
 spinner $!
 else
 echo -e "NOTE: Please install lsof as this script needs it to determine if Solr is listening on port $SOLR_PORT."
 sleep 10
 SOLR_PID=$(ps auxww | grep start\.jar | awk "/\-Djetty\.port=$SOLR_PORT/"' {print $2}' | sort -r)
 echo -e "\nStarted Solr server on port $SOLR_PORT (pid=$SOLR_PID). Happy searching!\n"
 return;
 fi
 fi

what's tough is I do not see any of these echo statements anywhere when you start as service so not sure what it saying or seeing. This is top after I start solr. You can see java process which is solr and lsof of script trying to determine its running.

avillanueva_1-1744379788865.png

Here is info on that process showing it was running with -v option. Odd since that runs normally and 

avillanueva_2-1744379860387.png

This runs normally and I get output:

avillanueva_3-1744379973999.png

Totally stumped here.

 

 

Best answer by avillanueva

So returning to this open issue. AI did not give me the answer but gave me something to try. Would love to know why this seemed to work. I changed the solr.service file to change "Type" from "forking" to "simple". I gave it a whirl and it seems stable. I still do not know why solr was timing out trying to see that it was running but this seemed to be a possible work around. 

1 reply

20-Turquoise
April 11, 2025

Is it possible for you to post the contents of your solr.service file?

avillanueva
23-Emerald I
23-Emerald I
April 11, 2025
[Unit]
Description=Windchill Solr Index Search
After=network.target

[Service]
Type=forking
RemainAfterExit=no
PIDFile=<mypath>/SolrServer/solr/bin/solr-8085.pid
ExecStart=<mypath>/SolrServer/bin/Index_Search_Server.sh start
ExecReload=<mypath>/SolrServer/bin/Index_Search_Server.sh stop
ExecReload=<mypath>/SolrServer/bin/Index_Search_Server.sh start
ExecStop=<mypath>/SolrServer/bin/Index_Search_Server.sh stop
User=<username>
Group=<group>
PrivateTmp=True
LimitNOFILE=infinity

[Install]
WantedBy=multi-user.target

I removed some personal details. Thanks for reply.

avillanueva
23-Emerald I
avillanueva23-Emerald IAuthorAnswer
23-Emerald I
January 20, 2026

So returning to this open issue. AI did not give me the answer but gave me something to try. Would love to know why this seemed to work. I changed the solr.service file to change "Type" from "forking" to "simple". I gave it a whirl and it seems stable. I still do not know why solr was timing out trying to see that it was running but this seemed to be a possible work around.