Skip to main content
avillanueva
23-Emerald I
avillanueva23-Emerald I
23-Emerald I
March 4, 2025
Solved

SOLR not starting as a service (systemctl) on RHEL 9

  • March 4, 2025
  • 1 reply
  • 1913 views

New installation of SOLR 13.0.2.4 here. Its in and I have it running on RHEL 9, on a NFS mounted folder like I always have done.  System is running SELinux in enforcing mode. I copied my solr.service from my other systems (since I used the same paths) but its not working and cannot figure out way (but it sources sooo familiar). I am able to use my install user and start/stop it perfectly, no issues. 

When I try to run this using systemctl, no joy. I need another pair of eyes on this. 

Service definition:

[Unit]
Description=Windchill Solr Index Search
After=network.target

[Service]
Type=forking
RemainAfterExit=no
PIDFile=<installpath>/SolrServer/solr/bin/solr-<port>.pid
ExecStart=<installpath>/SolrServer/bin/Index_Search_Server.sh
ExecReload=<installpath>/SolrServer/bin/Index_Search_Server.sh stop
ExecReload=<installpath>/SolrServer/bin/Index_Search_Server.sh start
ExecStop=<installpath>/SolrServer/bin/Index_Search_Server.sh stop
User=<installuser>
Group=<installuser>
PrivateTmp=True
LimitNOFILE=infinity

[Install]
WantedBy=multi-user.target

and here is what I get. Note there is no denials in audit.logs which is typical with SELinux. I tried audit2why but that is not it. All rights are 775. 

Mar 04 17:08:04 <hostname> systemd[1]: Starting Windchill Solr Index Search...
░░ Subject: A start job for unit solr.service has begun execution
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ A start job for unit solr.service has begun execution.
░░
░░ The job identifier is 2090.
Mar 04 17:08:04 <hostname> systemd[2277]: solr.service: Failed to execute <installpath>/SolrServer/bin/Index_Search_Server.sh: Exec format error
Mar 04 17:08:04 <hostname> systemd[2277]: solr.service: Failed at step EXEC spawning <installpath>/SolrServer/bin/Index_Search_Server.sh: Exec format error
░░ Subject: Process <installpath>/SolrServer/bin/Index_Search_Server.sh could not be executed
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ The process <installpath>/SolrServer/bin/Index_Search_Server.sh could not be executed and failed.
░░
░░ The error number returned by this process is ERRNO.
Mar 04 17:08:04 <hostname> systemd[1]: solr.service: Control process exited, code=exited, status=203/EXEC
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ An ExecStart= process belonging to unit solr.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 203.
Mar 04 17:08:04 <hostname> systemd[1]: solr.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ The unit solr.service has entered the 'failed' state with result 'exit-code'.
Mar 04 17:08:04 <hostname> systemd[1]: Failed to start Windchill Solr Index Search.
░░ Subject: A start job for unit solr.service has failed
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ A start job for unit solr.service has finished with a failure.
░░
░░ The job identifier is 2090 and the job result is failed.

I tried running in permissive mode but that had no impact. I cannot see a path issue. Online says to look at shebangs but never had to touch those before. I ran chmod recursive to update rights. Anything else I can try or test. 

Best answer by avillanueva

So it was a simple issue that was cause the error "Exec format error". All articles were pointing to check shebang. And it was there but I noticed that there was a line return at the start of the Index_Search_Server.sh file. I did not edit this file so it must have been there from PSI install. Removing this line return got past the issue. So marking this as solution but damn, what a PITA. 

 

Next issue is that (as root) server is timing out when starting. Here is verbose startup output:

░░ The job identifier is 2664.
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: BASEDIR= /<installpath>/SolrServer/bin
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Starting Solr Server outside installer
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: start
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: verbose
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Home= /<installpath>/SolrServer
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_High_Availability= false
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Data_Dir= /<installpath>/SolrServer/solr/data
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Hostname= <hostname>
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Port= 8085
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Zookeeper_Hosts=
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Solr_Java_Home= /<installpath>/Java
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Solr_Heap= 2048m
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Path= /<installpath>/SolrServer/solr/bin
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: start
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: BASISTECH_ROOT= /<installpath>/SolrServer/solr-basis
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: INDEX_SEARCH_EXTRACTORS= /<installpath>/SolrServer/config/solrserver.properties
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: SOLR_JUTE_MAXBUFFER= 3145728
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: start high false
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2602]: Java 17 detected. Enabled workaround for SOLR-16463
Mar 05 16:22:41 <hostname> systemd[1]: solr.service: start operation timed out. Terminating.
Mar 05 16:22:42 <hostname> systemd[1]: solr.service: Failed with result 'timeout'.

linking to the workaround that was noted: https://issues.apache.org/jira/browse/SOLR-16463

Dev server so both are planned to be installed on same server. 

I noticed that lsof spikes to 100% CPU when this timeout occurs but ends after SOLR gives up starting and times out. I might open a new thread on this one.

1 reply

avillanueva
23-Emerald I
avillanueva23-Emerald IAuthorAnswer
23-Emerald I
March 5, 2025

So it was a simple issue that was cause the error "Exec format error". All articles were pointing to check shebang. And it was there but I noticed that there was a line return at the start of the Index_Search_Server.sh file. I did not edit this file so it must have been there from PSI install. Removing this line return got past the issue. So marking this as solution but damn, what a PITA. 

 

Next issue is that (as root) server is timing out when starting. Here is verbose startup output:

░░ The job identifier is 2664.
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: BASEDIR= /<installpath>/SolrServer/bin
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Starting Solr Server outside installer
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: start
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: verbose
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Home= /<installpath>/SolrServer
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_High_Availability= false
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Data_Dir= /<installpath>/SolrServer/solr/data
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Hostname= <hostname>
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Port= 8085
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Zookeeper_Hosts=
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Solr_Java_Home= /<installpath>/Java
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Solr_Heap= 2048m
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Path= /<installpath>/SolrServer/solr/bin
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: start
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: BASISTECH_ROOT= /<installpath>/SolrServer/solr-basis
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: INDEX_SEARCH_EXTRACTORS= /<installpath>/SolrServer/config/solrserver.properties
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: SOLR_JUTE_MAXBUFFER= 3145728
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: start high false
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2602]: Java 17 detected. Enabled workaround for SOLR-16463
Mar 05 16:22:41 <hostname> systemd[1]: solr.service: start operation timed out. Terminating.
Mar 05 16:22:42 <hostname> systemd[1]: solr.service: Failed with result 'timeout'.

linking to the workaround that was noted: https://issues.apache.org/jira/browse/SOLR-16463

Dev server so both are planned to be installed on same server. 

I noticed that lsof spikes to 100% CPU when this timeout occurs but ends after SOLR gives up starting and times out. I might open a new thread on this one.

    avillanueva
    23-Emerald I
    avillanueva23-Emerald IAuthor
    23-Emerald I
    March 6, 2025

    My last note on lsof. The solr script that gets eventually called uses lsof to check if the server did indeed get started. As root, using systemctl to start as my install user (not root), it is not working. Cannot seem to id fault or see why its not starting. This is why I saw lsof running since it was looping to check if SOLR started. It eventually times out. This part is normal. Moving on for now.

    Terms & ConditionsCookie settingsAccessibility statement