Best Practice for Large Data Migration – Script Timeout Issues

Hi ThingWorx Community,

I’m working on a large data migration project in ThingWorx and would love your advice or best practices!

🔍 Context:

• I have around 100 DataTables, each with a significant number of rows.

• For each DataTable, I read all entries, transform the data (sometimes with nested loops, adding extra logic), and then either:

  • Build a large InfoTable result,

     

    This logic works fine for small datasets, but with real data volumes, I’m consistently hitting this error:

     

    Execution of Script terminated after: 30 seconds. Timeout configured for 30 seconds.

    🔄 I understand the ThingWorx ScriptTimeout is there for a reason, but:

    • I’ve seen loops that just insert thousands of rows run much longer without hitting this.

    • Now that I’m reading + transforming + updating, I’m hitting the limit quickly.

    💡 Questions:

    1. How do others handle large migrations like this without hitting the script timeout?

    2. Is there a proven batching or chunking pattern that works well inside ThingWorx services?

    3. Should I switch to a Timer/Scheduler Thing approach? Any examples or pitfalls?

    4. Any other general tips for making heavy transformations scalable and safe?

    I’d really appreciate any real-world lessons or suggestions.

     

    Thanks in advance for your help!
    Best regards,