cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Community Tip - When posting, your subject should be specific and summarize your question. Here are some additional tips on asking a great question. X

Translate the entire conversation x

SOLR not starting as a service (systemctl) on RHEL 9

avillanueva
23-Emerald I

SOLR not starting as a service (systemctl) on RHEL 9

New installation of SOLR 13.0.2.4 here. Its in and I have it running on RHEL 9, on a NFS mounted folder like I always have done.  System is running SELinux in enforcing mode. I copied my solr.service from my other systems (since I used the same paths) but its not working and cannot figure out way (but it sources sooo familiar). I am able to use my install user and start/stop it perfectly, no issues. 

When I try to run this using systemctl, no joy. I need another pair of eyes on this. 

Service definition:

[Unit]
Description=Windchill Solr Index Search
After=network.target

[Service]
Type=forking
RemainAfterExit=no
PIDFile=<installpath>/SolrServer/solr/bin/solr-<port>.pid
ExecStart=<installpath>/SolrServer/bin/Index_Search_Server.sh
ExecReload=<installpath>/SolrServer/bin/Index_Search_Server.sh stop
ExecReload=<installpath>/SolrServer/bin/Index_Search_Server.sh start
ExecStop=<installpath>/SolrServer/bin/Index_Search_Server.sh stop
User=<installuser>
Group=<installuser>
PrivateTmp=True
LimitNOFILE=infinity

[Install]
WantedBy=multi-user.target

and here is what I get. Note there is no denials in audit.logs which is typical with SELinux. I tried audit2why but that is not it. All rights are 775. 

Mar 04 17:08:04 <hostname> systemd[1]: Starting Windchill Solr Index Search...
░░ Subject: A start job for unit solr.service has begun execution
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ A start job for unit solr.service has begun execution.
░░
░░ The job identifier is 2090.
Mar 04 17:08:04 <hostname> systemd[2277]: solr.service: Failed to execute <installpath>/SolrServer/bin/Index_Search_Server.sh: Exec format error
Mar 04 17:08:04 <hostname> systemd[2277]: solr.service: Failed at step EXEC spawning <installpath>/SolrServer/bin/Index_Search_Server.sh: Exec format error
░░ Subject: Process <installpath>/SolrServer/bin/Index_Search_Server.sh could not be executed
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ The process <installpath>/SolrServer/bin/Index_Search_Server.sh could not be executed and failed.
░░
░░ The error number returned by this process is ERRNO.
Mar 04 17:08:04 <hostname> systemd[1]: solr.service: Control process exited, code=exited, status=203/EXEC
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ An ExecStart= process belonging to unit solr.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 203.
Mar 04 17:08:04 <hostname> systemd[1]: solr.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ The unit solr.service has entered the 'failed' state with result 'exit-code'.
Mar 04 17:08:04 <hostname> systemd[1]: Failed to start Windchill Solr Index Search.
░░ Subject: A start job for unit solr.service has failed
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ A start job for unit solr.service has finished with a failure.
░░
░░ The job identifier is 2090 and the job result is failed.

I tried running in permissive mode but that had no impact. I cannot see a path issue. Online says to look at shebangs but never had to touch those before. I ran chmod recursive to update rights. Anything else I can try or test. 

ACCEPTED SOLUTION

Accepted Solutions

So it was a simple issue that was cause the error "Exec format error". All articles were pointing to check shebang. And it was there but I noticed that there was a line return at the start of the Index_Search_Server.sh file. I did not edit this file so it must have been there from PSI install. Removing this line return got past the issue. So marking this as solution but damn, what a PITA. 

 

Next issue is that (as root) server is timing out when starting. Here is verbose startup output:

░░ The job identifier is 2664.
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: BASEDIR= /<installpath>/SolrServer/bin
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Starting Solr Server outside installer
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: start
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: verbose
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Home= /<installpath>/SolrServer
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_High_Availability= false
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Data_Dir= /<installpath>/SolrServer/solr/data
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Hostname= <hostname>
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Port= 8085
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Zookeeper_Hosts=
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Solr_Java_Home= /<installpath>/Java
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Solr_Heap= 2048m
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Path= /<installpath>/SolrServer/solr/bin
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: start
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: BASISTECH_ROOT= /<installpath>/SolrServer/solr-basis
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: INDEX_SEARCH_EXTRACTORS= /<installpath>/SolrServer/config/solrserver.properties
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: SOLR_JUTE_MAXBUFFER= 3145728
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: start high false
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2602]: Java 17 detected. Enabled workaround for SOLR-16463
Mar 05 16:22:41 <hostname> systemd[1]: solr.service: start operation timed out. Terminating.
Mar 05 16:22:42 <hostname> systemd[1]: solr.service: Failed with result 'timeout'.

linking to the workaround that was noted: https://issues.apache.org/jira/browse/SOLR-16463

Dev server so both are planned to be installed on same server. 

I noticed that lsof spikes to 100% CPU when this timeout occurs but ends after SOLR gives up starting and times out. I might open a new thread on this one.

View solution in original post

11 REPLIES 11

So it was a simple issue that was cause the error "Exec format error". All articles were pointing to check shebang. And it was there but I noticed that there was a line return at the start of the Index_Search_Server.sh file. I did not edit this file so it must have been there from PSI install. Removing this line return got past the issue. So marking this as solution but damn, what a PITA. 

 

Next issue is that (as root) server is timing out when starting. Here is verbose startup output:

░░ The job identifier is 2664.
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: BASEDIR= /<installpath>/SolrServer/bin
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Starting Solr Server outside installer
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: start
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: verbose
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Home= /<installpath>/SolrServer
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_High_Availability= false
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Data_Dir= /<installpath>/SolrServer/solr/data
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Hostname= <hostname>
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Port= 8085
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Zookeeper_Hosts=
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Solr_Java_Home= /<installpath>/Java
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Solr_Heap= 2048m
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: Index_Search_Solr_Path= /<installpath>/SolrServer/solr/bin
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: start
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: BASISTECH_ROOT= /<installpath>/SolrServer/solr-basis
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: INDEX_SEARCH_EXTRACTORS= /<installpath>/SolrServer/config/solrserver.properties
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: SOLR_JUTE_MAXBUFFER= 3145728
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2537]: start high false
Mar 05 16:21:11 <hostname> Index_Search_Server.sh[2602]: Java 17 detected. Enabled workaround for SOLR-16463
Mar 05 16:22:41 <hostname> systemd[1]: solr.service: start operation timed out. Terminating.
Mar 05 16:22:42 <hostname> systemd[1]: solr.service: Failed with result 'timeout'.

linking to the workaround that was noted: https://issues.apache.org/jira/browse/SOLR-16463

Dev server so both are planned to be installed on same server. 

I noticed that lsof spikes to 100% CPU when this timeout occurs but ends after SOLR gives up starting and times out. I might open a new thread on this one.

My last note on lsof. The solr script that gets eventually called uses lsof to check if the server did indeed get started. As root, using systemctl to start as my install user (not root), it is not working. Cannot seem to id fault or see why its not starting. This is why I saw lsof running since it was looping to check if SOLR started. It eventually times out. This part is normal. Moving on for now.


@avillanueva wrote:

So it was a simple issue that was cause the error "Exec format error". All articles were pointing to check shebang. And it was there but I noticed that there was a line return at the start of the Index_Search_Server.sh file. I did not edit this file so it must have been there from PSI install. Removing this line return got past the issue. So marking this as solution but **bleep**, what a PITA. 

 


I can confirm the same issue on a test 13.0.2.2 system I just cps updated to 13..0.2.4. The 13.0.2.4 Index_Search_Server.sh has an empty line as the first line before the shebang. This is not the case with the 13.0.2.2 system. Have you opened a case with PTC on this?

 

No but I should. Thank you.

I just opened a case with PTC.

C17725412 in case you want to reference it if you open a case.

I did too.

I just cps updated a test system from 13.0.2.4 to 13.0.2.5 and this issue no longer exists.

The first line of the installed Index_Search_Server.sh is a shebang as expected.

I just installed a test 13.1.1.0 system and it has this issue.

Which issue? Was it the shebang issue in the startup script or not starting as root? I still am stumped on the last one.


@avillanueva wrote:

Which issue? Was it the shebang issue in the startup script or not starting as root? I still am stumped on the last one.


Should have been more specific...

The shebang issue in the startup script is back for 13.1.1.0.

avillanueva_0-1751042585100.png

 

Announcements

Top Tags