Hello ThingWorx Community,
I am currently working with a ThingWorx DataTable that contains approximately 150,000 entries. Since handling large datasets in ThingWorx DataTables is not optimal, I have implemented SQL Server as a more efficient storage solution.
Now, I need to migrate this large dataset from ThingWorx DataTable to SQL Server, but I am facing challenges in doing so efficiently. Here are the key concerns:
Has anyone successfully handled a similar data migration in ThingWorx? I would appreciate any insights, best practices, or recommendations for moving large datasets from ThingWorx DataTables to SQL Server efficiently.
Looking forward to your suggestions!
Thanks in advance!
Solved! Go to Solution.
Hi @MA8731174
Javascript service has a default execution timeout of 30 seconds. It is configured with platform-setting.json. Changing this value needs tomcat restart.
However, you can set the serivce "Async" option to allow long-time execution.
It might be a good idea to loop the data of the large DataTable to create Insert SQL statement into a file and finally run those statements on DB side.
I have been working on this migration and wanted to share my progress so far.
When using a for loop to read from the ThingWorx DataTable and insert records into SQL Server, I encountered the following error:
I have already increased the SQL query timeout in the Service Info settings to 7200 seconds, but the issue persists. Based on my observations, I suspect this might be related to the Apache server timeout settings rather than ThingWorx itself.
Has anyone faced a similar issue before? Could this be an Apache timeout limitation, and if so, is there a way to adjust it? Or should I consider breaking down the data migration into smaller batches to prevent timeouts? But i cannot break it into smaller batches as i have 150K entries in datatable. it will be very time consuming...
Any insights or suggestions would be greatly appreciated!
Thanks in advance!
Hi @MA8731174 ,
After upgrading Postgres DB Schema to the latest version, I recently encountered this error. This is due to Database User Permission issues.
Datatable data gets stored in a Table in the Database itself. I suggest writing SQL Script or Python code to move the data from one Table (where the data of the Datatable exists) to a new table in the same database.
Thanks,
Hi @MA8731174
Javascript service has a default execution timeout of 30 seconds. It is configured with platform-setting.json. Changing this value needs tomcat restart.
However, you can set the serivce "Async" option to allow long-time execution.
It might be a good idea to loop the data of the large DataTable to create Insert SQL statement into a file and finally run those statements on DB side.
Thank you @wcui for your response! Your suggestion helped a lot.
Now, I’m facing another challenge: when generating an SQL file with more than 1000 rows, SQL Server returns an error stating that the number of row value expressions in the INSERT statement exceeds the maximum allowed limit of 1000. This limitation is preventing me from executing bulk inserts efficiently.
Would you have any recommendations on how to handle this? Should I split the INSERT statements into batches of 1000 rows, or would using BULK INSERT be a more efficient approach? And how can i split it in SQL Server? Any recommendation
I’ve also attached a screenshot of the error for reference. Looking forward to your insights!
Hi @MA8731174
Glad to know it helped.
I'm not very proficient in SQL insert script, depending on the error content, the following link may be able to help you.
Thanks for your reply. In the meantime i have one more query. Would you please share your insights about it
I am currently working on a ThingWorx Database Thing service and came across a best practice recommendation stating that [[ ]] should not be used in SQL queries. Instead, values should be passed as input parameters. However, I am facing difficulties implementing this correctly.
I referred to this ThingWorx documentation:
https://support.ptc.com/help/thingworx/platform/r9.7/en/index.html?_gl=1*br4gd2*_gcl_au*OTY2NDE4Mjg3LjE3Mzk5NDI0MjM.*_ga*MjAwNjg4ODQ2MS4xNzM5OTQyNDIy*_ga_7NMP2MSYPM*MTc0MjgxODk5NS4zOS4xLjE3NDI4MTkwMjIuMzMuMC4w#page/ThingWorx/Help/Composer/Things/ThingServices.htmlThingWorx Service Best Practices
Previously, I was using [[ ]] like this:
This approach worked fine, but based on the documentation, I wanted to switch to named parameters instead.
I attempted to pass the parameters like this in my JavaScript service:
However, this approach did not work, and I’m unsure whether I should keep the manual quotes around searchinput for FREETEXT().
To follow best practices, I modified my SQL query to use ? placeholders:
And updated my JavaScript code to:
However, this also did not work, and I suspect it may be due to how parameters are bound in the query.
How should I properly bind named parameters (lineName, filterParameter) in a ThingWorx Database Thing query?
Do I need to manually add quotes around searchinput for FREETEXT() to work correctly?
Does ThingWorx support ? placeholders for SQL queries in Database Thing, or should I stick with named parameters?
Hi @MA8731174
Can you please open a new post for the SQL queries questions?
It is more efficient to manage 1 issue with 1 post.
Thanks for the cooperation in advance.
Wenjing
Thank you for your cooperation. I have accepted an ANSWER of the question i orginally raised and The question which i have raised now in the last is well explained in this below post.