ThingWorx Navigate is now Windchill Navigate Learn More

Translate the entire conversation x

Best Approach to Move Large Data from ThingWorx DataTable to SQL Server

MA8731174
15-Moonstone

Best Approach to Move Large Data from ThingWorx DataTable to SQL Server

Hello ThingWorx Community,

 

I am currently working with a ThingWorx DataTable that contains approximately 150,000 entries. Since handling large datasets in ThingWorx DataTables is not optimal, I have implemented SQL Server as a more efficient storage solution.

 

Now, I need to migrate this large dataset from ThingWorx DataTable to SQL Server, but I am facing challenges in doing so efficiently. Here are the key concerns:

 

  1. Row-by-Row Insert: When I try to insert data row by row, I run into execution time issues.
  2. Bulk Insert Using SQL File: I attempted to create a large SQL insert file, but some data entries are missing or cause issues during execution.
  3. Performance & Best Practices: What is the most optimal and scalable way to migrate this large dataset to SQL Server?

Has anyone successfully handled a similar data migration in ThingWorx? I would appreciate any insights, best practices, or recommendations for moving large datasets from ThingWorx DataTables to SQL Server efficiently.

 

Looking forward to your suggestions!

Thanks in advance!

ACCEPTED SOLUTION

Accepted Solutions
wcui
16-Pearl
(To:MA8731174)

Hi @MA8731174 

 

Javascript service has a default execution timeout of 30 seconds. It is configured with platform-setting.json. Changing this value needs tomcat restart.

https://support.ptc.com/help/thingworx/platform/r9.7/en/index.html#page/ThingWorx/Help/Composer/System/Logs/ScriptErrorLog.html

 

However, you can set the serivce "Async" option to allow long-time execution.

https://support.ptc.com/help/thingworx/platform/r9.7/en/index.html#page/ThingWorx/Help/Composer/Things/ThingServices.html

 

It might be a good idea to loop the data of the large DataTable to create Insert SQL statement into a file and finally run those statements on DB side.

 

View solution in original post

8 REPLIES 8
MA8731174
15-Moonstone
(To:MA8731174)

 

I have been working on this migration and wanted to share my progress so far.

When using a for loop to read from the ThingWorx DataTable and insert records into SQL Server, I encountered the following error:

 

 
Error executing service. Message ::Execution of Script terminated after : 41 seconds. Timeout configured for 30 seconds. - See Script Error Log for more details.
 

I have already increased the SQL query timeout in the Service Info settings to 7200 seconds, but the issue persists. Based on my observations, I suspect this might be related to the Apache server timeout settings rather than ThingWorx itself.

Has anyone faced a similar issue before? Could this be an Apache timeout limitation, and if so, is there a way to adjust it? Or should I consider breaking down the data migration into smaller batches to prevent timeouts? But i cannot break it into smaller batches as i have 150K entries in datatable. it will be very time consuming...

 

Any insights or suggestions would be greatly appreciated!

 

Thanks in advance!

Hi @MA8731174 ,

 

After upgrading Postgres DB Schema to the latest version, I recently encountered this error. This is due to Database User Permission issues.

 

Datatable data gets stored in a Table in the Database itself. I suggest writing SQL Script or Python code to move the data from one Table (where the data of the Datatable exists) to a new table in the same database.

 

Thanks,

Shashi Preetham,
Email: psp316r@outlook.com,
Mobile: +91 8099838001.
wcui
16-Pearl
(To:MA8731174)

Hi @MA8731174 

 

Javascript service has a default execution timeout of 30 seconds. It is configured with platform-setting.json. Changing this value needs tomcat restart.

https://support.ptc.com/help/thingworx/platform/r9.7/en/index.html#page/ThingWorx/Help/Composer/System/Logs/ScriptErrorLog.html

 

However, you can set the serivce "Async" option to allow long-time execution.

https://support.ptc.com/help/thingworx/platform/r9.7/en/index.html#page/ThingWorx/Help/Composer/Things/ThingServices.html

 

It might be a good idea to loop the data of the large DataTable to create Insert SQL statement into a file and finally run those statements on DB side.

 

MA8731174
15-Moonstone
(To:wcui)

Thank you @wcui  for your response! Your suggestion helped a lot.

 

Now, I’m facing another challenge: when generating an SQL file with more than 1000 rows, SQL Server returns an error stating that the number of row value expressions in the INSERT statement exceeds the maximum allowed limit of 1000. This limitation is preventing me from executing bulk inserts efficiently.

Would you have any recommendations on how to handle this? Should I split the INSERT statements into batches of 1000 rows, or would using BULK INSERT be a more efficient approach? And how can i split it in SQL Server? Any recommendation

 

I’ve also attached a screenshot of the error for reference. Looking forward to your insights!

 

MA8731174_0-1742553743518.png

 

wcui
16-Pearl
(To:MA8731174)

Hi @MA8731174 

Glad to know it helped.

I'm not very proficient in SQL insert script, depending on the error content, the following link may be able to help you.

https://dba.stackexchange.com/questions/82921/the-number-of-row-value-expressions-in-the-insert-statement-exceeds-the-maximum

 

MA8731174
15-Moonstone
(To:wcui)

Thanks for your reply. In the meantime i have one more query. Would you please share your insights about it

 

I am currently working on a ThingWorx Database Thing service and came across a best practice recommendation stating that [[ ]] should not be used in SQL queries. Instead, values should be passed as input parameters. However, I am facing difficulties implementing this correctly.

 

I referred to this ThingWorx documentation:
https://support.ptc.com/help/thingworx/platform/r9.7/en/index.html?_gl=1*br4gd2*_gcl_au*OTY2NDE4Mjg3LjE3Mzk5NDI0MjM.*_ga*MjAwNjg4ODQ2MS4xNzM5OTQyNDIy*_ga_7NMP2MSYPM*MTc0MjgxODk5NS4zOS4xLjE3NDI4MTkwMjIuMzMuMC4w#page/ThingWorx/Help/Composer/Things/ThingServices.htmlThingWorx Service Best Practices

My SQL Query in ThingWorx Service

Previously, I was using [[ ]] like this:

 

sql
 
SELECT * FROM dbo.QualityDataTable WHERE lineName = [[lineName]] AND
FREETEXT((serialNumber, userIdentification, jsonData), <<filterParameter>>)
ORDER BY qualityCheckTimestamp DESC;

 

 

This approach worked fine, but based on the documentation, I wanted to switch to named parameters instead.


Approach 1: Using Named Parameters in JavaScript

I attempted to pass the parameters like this in my JavaScript service:

 

javascript
 
let params = { filterParameter: "'" + searchinput + "'",
lineName: lineName };
let dataBaseQuery = Things["V2.MYSQLDatabaseController"].SQLQuery_GetEntryBySearchInput(params);

 

 

However, this approach did not work, and I’m unsure whether I should keep the manual quotes around searchinput for FREETEXT().


Approach 2: Using ? Placeholders

To follow best practices, I modified my SQL query to use ? placeholders:

 

sql
 
SELECT * FROM dbo.QualityDataTable WHERE lineName = ? AND
FREETEXT((serialNumber, userIdentification, jsonData), ?)
ORDER BY qualityCheckTimestamp DESC;

 

 

And updated my JavaScript code to:

 

javascript
 
let params = { values: [lineName, searchinput] // Passing parameters as an array };
let dataBaseQuery = Things["V2.MYSQLDatabaseController"].SQLQuery_GetEntryBySearchInput(params);
 

However, this also did not work, and I suspect it may be due to how parameters are bound in the query.


Questions:

  1. How should I properly bind named parameters (lineName, filterParameter) in a ThingWorx Database Thing query?

  2. Do I need to manually add quotes around searchinput for FREETEXT() to work correctly?

  3. Does ThingWorx support ? placeholders for SQL queries in Database Thing, or should I stick with named parameters?

wcui
16-Pearl
(To:MA8731174)

Hi @MA8731174 

Can you please open a new post for the SQL queries questions?

It is more efficient to manage 1 issue with 1 post.

Thanks for the cooperation in advance.

 

Wenjing

MA8731174
15-Moonstone
(To:wcui)

Thank you for your cooperation. I have accepted an ANSWER of the question i orginally raised and  The question which i have raised now in the last is well explained in this below post. 

 

https://community.ptc.com/t5/ThingWorx-Developers/SQL-passing-part-of-query-as-input-parameter/m-p/1006131#M69355

Announcements




Top Tags