SOLR STILL not starting on RHEL 9.2 as a service
I've gone done a serious rabbit hole but failing to see my error. I am assuming it must be me. I have two instances and both experience same behavior. I can start and stop SOLR normally as my non-root user. But when I configure my solr.service (which is identical to my previous installs), it fails to see that solr is running, exits and stops the process. I can verify that SOLR does indeed start but the part of the solr script (called from Index_Search_Server.sh) where it checks that SOLR is running, I think is not detecting it. It times out after 180 seconds and stops.
Here is what I observed:
- Confirmed that SOLR starts from java process, PID file and logs.
- Noticed "lsof -v" running while it was waiting for SOLR to start and consuming high CPU. Disappears when it bails after the time out.
- I identified the part of the solr script where it tries to verify SOLR is running. (rabbit hole). I can see that it checks the lsof version (where I see "lsof -v", and then runs it to verify to get the PID of the process running at port 8085. I can validate that these commands return normally when I ran them while waiting for startup script to timeout.

I zeroed in on lsof since that is what I saw running and a while loop in the script. I gather that Index_Search_Server is calling solr but not getting a successful return. I even dug into the github for solr trying to see when this script change (here it is if you are curious https://github.com/apache/solr/commit/3ed17c2737d )
Here is the section where its stuck
# no lsof on cygwin though
if lsof -v 2>&1 | grep -q revision; then
echo -n "Waiting up to $SOLR_START_WAIT seconds to see Solr running on port $SOLR_PORT"
# Launch in a subshell to show the spinner
(loops=0
while true
do
running=$(lsof -t -PniTCP:$SOLR_PORT -sTCP:LISTEN || :)
if [ -z "${running:-}" ]; then
slept=$((loops * 2))
if [ $slept -lt $SOLR_START_WAIT ]; then
sleep 2
loops=$((loops+1))
else
echo -e "Still not seeing Solr listening on $SOLR_PORT after $SOLR_START_WAIT seconds!"
tail -30 "$SOLR_LOGS_DIR/solr.log"
exit # subshell!
fi
else
SOLR_PID=$(ps auxww | grep start\.jar | awk "/\-Djetty\.port=$SOLR_PORT/"' {print $2}' | sort -r)
echo -e "\nStarted Solr server on port $SOLR_PORT (pid=$SOLR_PID). Happy searching!\n"
exit # subshell!
fi
done) &
spinner $!
else
echo -e "NOTE: Please install lsof as this script needs it to determine if Solr is listening on port $SOLR_PORT."
sleep 10
SOLR_PID=$(ps auxww | grep start\.jar | awk "/\-Djetty\.port=$SOLR_PORT/"' {print $2}' | sort -r)
echo -e "\nStarted Solr server on port $SOLR_PORT (pid=$SOLR_PID). Happy searching!\n"
return;
fi
fi
what's tough is I do not see any of these echo statements anywhere when you start as service so not sure what it saying or seeing. This is top after I start solr. You can see java process which is solr and lsof of script trying to determine its running.

Here is info on that process showing it was running with -v option. Odd since that runs normally and

This runs normally and I get output:

Totally stumped here.

