Best Practice for Handling Large-Scale File Storage in ThingWorx – Local vs. File Share

[MA8731174]

Hi everyone,

We're currently facing a performance challenge in our ThingWorx implementation related to saving a large volume of image files (potentially millions over time).

We’ve observed that writing files directly to a file share (network storage) from ThingWorx is quite slow, whereas saving them to the local repository (on the same server where the ThingWorx ApplicationServer/Experience Service runs) is significantly faster.

To address this, our development team has proposed the following approach:

• Save incoming files temporarily to the local ThingWorx repository (for fast write operations).

• Then, in the background, run a scheduled service (e.g., every 15 or 30 minutes) that:

  • Moves these files to the file share for long-term storage.

  • Deletes them from the local repository after successful transfer.

We’d like to ask the community:

• Is this a recommended or sustainable strategy for handling large-scale file storage in ThingWorx?

• Are there risks or performance concerns (e.g., file locking, memory issues, scaling problems) we should be aware of with this kind of background file migration?

• Any alternatives or best practices others have used for similar use cases?

Thanks in advance for your insights!

[GuillaumeL]

Hello,

I don't have so much experience with such volumetry but I will try to give you some inputs waiting other people to help you !

• When you talk about "a file share (network storage)". Is it a ThingWorx local repository that has been mounted afterward as a share folder ? If not, you can have a try to see if it improves performance.
• In ThingWorx 10.0, you will have a feature that will increase performance but just for read purpose on client side. This is ETag (https://support.ptc.com/help/thingworx/platform/r10.0/en/index.html#page/ThingWorx/Help/Composer/System/Subsystems/ETagSupportForFileRepository.html). It will not help you for your question but still good to know.

Regards,

Guillaume

[MA8731174]

Hello Guillaume,

Thank you very much for your input and for taking the time to share these ideas!

Regarding your first point: Yes, we are indeed using a connector (like a symbolic link) to store the images — essentially similar to a desktop shortcut that points to a shared storage. So we already have that setup in place.

For your second point: We are currently working with ThingWorx 9.4, so unfortunately we don’t have the ETag feature yet — but it’s good to know this is coming in ThingWorx 10.0.

Thanks again for your suggestions!

[abandal]

Hi @MA8731174,

 
I wanted to follow up with you on your post to see if your question has been answered.

if you have more to share on your issue, please let the Community know so that we can continue to support.

Thanks,
Abhi