Not able to start postgre sql

[sanjanabalaji]

I am not able to start my postgre sql. when I attempt to start, it stops immediately. Also, my "ThingworxPostgresqlStorage" folder is taking 198 Gb  space out of 250 Gb. Because of this my "C drive" goes full. I tried removing some files from this folder and attempted to start "postgresql serviece". But, again "ThingworxPostgresqlStorage" folder is taking lots of space. I am using Thingworx version 8.4. What is the solution for this ?

[raluca_edu]

Hi,

We need more info on Thingworx side:

- exact version: 8.4.x

- it is a new installation?

- attach ThingworxStorage/logs, Tomcat logs, Postgres logs.

Thanks,

Raluca Edu

[sanjanabalaji]

It is not a new installation.

Since composer is not opening, can you tell me how to check the version of thingworx?

Attaching the current logs of Tomcat

[Jimwang]

@sanjanabalaji,

If the database Postgresql could not be started, Thingworx could not be started also.

The below log files will be useful to understand the issue:

1) Postgresql's log

2) Tomcat's catalina.<TimeStamp>.log.

3) Thingworx's log flies under <ThingworxStorage>/logs. It will be helpful to generate fresh log files after backup and delete the old log files and restart Thingworx.

Tomcat's localhost and manager log files are not useful for this kind of case.

[sanjanabalaji]

Attaching postgre logs

[raluca_edu]

Hi,

Found this error: "ERROR: could not extend file "pg_tblspc/16396/PG_9.4_201409291/16397/16874.191": No space left on device"

This means you don't have enough space on disk, so please free some space and start again Thingworx.

Hope it helps,

Raluca Edu

[sanjanabalaji]

I increased the space but postgre service did not run, then I found that postgre sql dependency service was stopped. So, I started the dependency service. Now my postgre sql service is running, but my tomcat is not starting. It is starting and then stopping. I am attaching the tomcat logs here. 

[sanjanabalaji]

remaining files are empty.

[Jimwang]

Per Tomcat's log, it seems that Tomcat was restarted frequently without any errors. It seems to me that Tomcat was running in Windows service. Please correct me if I am wrong.

No any errors indicated that Tomcat was restarted frequently. Are there any errors in Window's Application event log?

If you stop Tomcat, remove the Tomcat cache under <Tomcat>/work/catalina, then restart it and see if it could be started.

[sanjanabalaji]

yes, Tomcat is running in Windows. And in event viewer I found this,

under  <Tomcat>/work/catalina, these many folders are there which cache i have to clear.

 

 

[Jimwang]

If you click the Detail tab in the event log, do you see any messages? or does it provide any hints?

For Tomcat cache, you can delete all of the information under <Tomcat>\work\Catalina\localhost.

You can back up the folder before you delete them.

[sanjanabalaji]

I got this in details

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

- <System>

  <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />

  <EventID Qualifiers="49152">7031</EventID>

  <Version>0</Version>

  <Level>2</Level>

  <Task>0</Task>

  <Opcode>0</Opcode>

  <Keywords>0x8080000000000000</Keywords>

  <TimeCreated SystemTime="2020-03-11T07:00:11.932060700Z" />

  <EventRecordID>13541</EventRecordID>

  <Correlation />

  <Execution ProcessID="700" ThreadID="5800" />

  <Channel>System</Channel>

  <Computer>DESKTOP-CP9UMFG</Computer>

  <Security />

  </System>

- <EventData>

  <Data Name="param1">Apache Tomcat 8.5 Tomcat8</Data>

  <Data Name="param2">1</Data>

  <Data Name="param3">60000</Data>

  <Data Name="param4">1</Data>

  <Data Name="param5">Restart the service</Data>

  <Binary>54006F006D0063006100740038000000</Binary>

  </EventData>

  </Event>

[sanjanabalaji]

Attaching Thingworx storage/logs here,

[Jimwang]

No useful information about why Windows terminate Tomcat service.

I would suggest:

1) Update Tomcat's service to Manual instead of the automatic

2) Reboot the machine

3)Then restart Tomcat Service

[Jimwang]

It appears Thingworx log files were actually generated.

Postgresql reports the below error, it says the file "pg_tblspc/16396/PG_9.4_201409291/16397/16707" couldn't be found. You can check if the file is corrupted or doesn't exist under <PostgreSQL>\data folder:

2020-03-11 14:58:11 IST ERROR: could not open file "pg_tblspc/16396/PG_9.4_201409291/16397/16707": No such file or directory
2020-03-11 14:58:11 IST STATEMENT: INSERT INTO system_ownership (platform) VALUES ($1) RETURNING id
2020-03-11 14:58:12 IST LOG: could not receive data from client: An existing connection was forcibly closed by the remote host.

You can try:

1) Stop Tomcat

2) Restart Postgresql

3) Launch PG Admin or pgsql, check if you can query the table "system_ownership"  of Thingworx

4) check if there are any errows in Postgresql log file

5) If everything is fine, restart Tomcat

I would also suggest to open a ticket to PTC technical support to get support about this.

[sanjanabalaji]

I restarted the system and postgre sql service is running. but my tomcat is starting->Running->stopped. this is happening continuously even after i changed it to Manual from Automatic and restarted the pc.

[Jimwang]

Although Postgresql is running, but I am afraid that some data is corrupted or lost per Postgresql log file, thus ThingWorx could not query or operate on the data properly, then shutdown the Tomcat.

So firstly we need to make sure Postgresql is working fine and No data lost/corrupted.

To do that,

1. stop Tomcat firstly

2. Restart Postgresql, check the log and see if there are any errors/warnings.

3. Using Pgsql or PG Admin to check if the table "system_ownership" exists or not.

4. If Postgresql is fine, restart Tomcat.

4. Check Postgresql log file and Thingworx log file and see what the error message is.

[sanjanabalaji]

I cleared the cache files in <Tomcat>\work\Catalina\localhost but tomcat is not starting. Also my postgre sql service is also not starting now. I am attaching here the current log files of tomcat and postgre sql.

[sanjanabalaji]

Instead of increasing the disk space. Can I move the "ThingworxPostgreSqlStorage" folder to another drive. will it affect anything?

[Jimwang]

Refer to the below articles:

https://www.ptc.com/en/support/article/CS273702 and https://www.ptc.com/en/support/article/CS299054

Important Note: It is VERY IMPORTANT to verify the operations on the test system before deploying it on Production system!

[sanjanabalaji]

Attaching Thingworx storage/logs

[Jimwang]

@sanjanabalaji ,

Per the below message in postgresql-2020-03-10_141648.log, it seems that PostGresql could be started successfully now:

        2020-03-10 14:21:26 IST LOG: database system is ready to accept connections

Next Steps:

1) Stop ThingWorx,

2) Backup and delete all the log files under <ThingworxStorage>/logs

3)  Restart Thingworx.

If ThingWorx still couldn't be started, please provide all the new log files under <ThingworxStorage>/logs