SOLR STILL not starting on RHEL 9.2 as a service

[avillanueva]

I've gone done a serious rabbit hole but failing to see my error. I am assuming it must be me. I have two instances and both experience same behavior. I can start and stop SOLR normally as my non-root user.  But when I configure my solr.service (which is identical to my previous installs), it fails to see that solr is running, exits and stops the process. I can verify that SOLR does indeed start but the part of the solr script  (called from Index_Search_Server.sh) where it checks that SOLR is running, I think is not detecting it. It times out after 180 seconds and stops. 

Here is what I observed:

• Confirmed that SOLR starts from java process, PID file and logs.
• Noticed "lsof -v" running while it was waiting for SOLR to start and consuming high CPU. Disappears when it bails after the time out. 
• I identified the part of the solr script where it tries to verify SOLR is running. (rabbit hole). I can see that it checks the lsof version (where I see "lsof -v", and then runs it to verify to get the PID of the process running at port 8085. I can validate that these commands return normally when I ran them while waiting for startup script to timeout. 

I zeroed in on lsof since that is what I saw running and a while loop in the script. I gather that Index_Search_Server is calling solr but not getting a successful return. I even dug into the github for solr trying to see when this script change (here it is if you are curious https://github.com/apache/solr/commit/3ed17c2737d )

Here is the section where its stuck

    # no lsof on cygwin though
    if lsof -v 2>&1 | grep -q revision; then
      echo -n "Waiting up to $SOLR_START_WAIT seconds to see Solr running on port $SOLR_PORT"
      # Launch in a subshell to show the spinner
      (loops=0
      while true
      do
        running=$(lsof -t -PniTCP:$SOLR_PORT -sTCP:LISTEN || :)
        if [ -z "${running:-}" ]; then
          slept=$((loops * 2))
          if [ $slept -lt $SOLR_START_WAIT ]; then
            sleep 2
            loops=$((loops+1))
          else
            echo -e "Still not seeing Solr listening on $SOLR_PORT after $SOLR_START_WAIT seconds!"
            tail -30 "$SOLR_LOGS_DIR/solr.log"
            exit # subshell!
          fi
        else
          SOLR_PID=$(ps auxww | grep start\.jar | awk "/\-Djetty\.port=$SOLR_PORT/"' {print $2}' | sort -r)
          echo -e "\nStarted Solr server on port $SOLR_PORT (pid=$SOLR_PID). Happy searching!\n"
          exit # subshell!
        fi
      done) &
      spinner $!
    else
      echo -e "NOTE: Please install lsof as this script needs it to determine if Solr is listening on port $SOLR_PORT."
      sleep 10
      SOLR_PID=$(ps auxww | grep start\.jar | awk "/\-Djetty\.port=$SOLR_PORT/"' {print $2}' | sort -r)
      echo -e "\nStarted Solr server on port $SOLR_PORT (pid=$SOLR_PID). Happy searching!\n"
      return;
    fi
  fi

what's tough is I do not see any of these echo statements anywhere when you start as service so not sure what it saying or seeing. This is top after I start solr. You can see java process which is solr and lsof of script trying to determine its running.

Here is info on that process showing it was running with -v option. Odd since that runs normally and 

This runs normally and I get output:

Totally stumped here.

[RandyJones]

Is it possible for you to post the contents of your solr.service file?

[avillanueva]

[Unit]
Description=Windchill Solr Index Search
After=network.target

[Service]
Type=forking
RemainAfterExit=no
PIDFile=<mypath>/SolrServer/solr/bin/solr-8085.pid
ExecStart=<mypath>/SolrServer/bin/Index_Search_Server.sh start
ExecReload=<mypath>/SolrServer/bin/Index_Search_Server.sh stop
ExecReload=<mypath>/SolrServer/bin/Index_Search_Server.sh start
ExecStop=<mypath>/SolrServer/bin/Index_Search_Server.sh stop
User=<username>
Group=<group>
PrivateTmp=True
LimitNOFILE=infinity

[Install]
WantedBy=multi-user.target

I removed some personal details. Thanks for reply.