Skip to main content
M
MA873117416-Pearl
16-Pearl
October 22, 2024
Solved

File Repository has 1 million photos

  • October 22, 2024
  • 2 replies
  • 2119 views

Hello Community,

  I have file repository which is running with network link. I have uploader widget and i upload the 10 pictures each pic is ok 100kb but uploader only upload randomly 3 4 pics in one go and other would not get upload and failed. 

When i upload the pictures on development server it works completely fine and upto 10 pictures can upload in one go with selection of upload widget.

 

Do you guys think that is there any solution which can help me here? I want to just upload the pictures and for that my service creates path first for the folder creation in repository and then pictures get uploaded there with the upload widget on mashup. 

 

Any solutions are welcome here. 

 

@Rocko if you would like to share your insights, it would be great!

 

Thank you!

Best answer by Rocko

We can't guess the problem, there could be many things causing it and you didn't give an error message.

It could be anything, maybe it's trying to save in parallel but you can have only a fixed number of connections or file handles.  Try to find the error message and drill down from there. You can also try to simulate multifile upload in a service, without the widget to isolate the issue. You could also try to "buffer" the upload in a local repo and move to the network repo later on.

2 replies

C
Constantine18-Opal
18-Opal
October 22, 2024

Hello,

 

You can split it into folders, similar to HashMap buckets. In your mashup with file upload widgets, use an Expression to generate a random integer from 1 to 1000. Then use this integer as the folder name where you upload your files. This way you'd have 1000 folders, each with approx. 1000 files.

 

Regards,

Constantine

    Rocko
    Rocko19-Tanzanite
    19-Tanzanite
    October 23, 2024

    As @Constantine suggested, do not have 1mil files in one folder. You could create folders per random number or a structure like year/month/day, in any case you would then also need a database (or a logic) where you store the correct path to find it back. (Note with 1mil rows you shouldn't use a Datatable)

    On the issue that it uploads 10 files in one go on dev but not on prod, check the logs for the error and try to find the source. As it works on the one system, it should work on the other unless some configuration is different between the systems, e.g. how you connect to the system, encryption, timeouts, latency, network, etc.

      M
      MA873117416-PearlAuthor
      16-Pearl
      October 23, 2024

      @Constantine @Rocko  Thanks both of you for your feedback.

       

      Actually at the moment we have folder for every picture which can be taken and saved. It is working same as @Rocko mentioned with datetime folder so here is the structure serialNumber/Timestamp/stepnumber --> 222/12345678/1 , 222/12345678/2 , 222/12345678/3 

       

      so as you can see it saves the pictures like this so the 1,2,3 they are also name of folders and in them we have image. thats how we are doing since few years... now i made a mashup where there is one field called serialNumber and user gives serialnumber there and press next and on press next i create a path and the path would be for eg  222/12345678/4 and in this folder name 4 i upload around 10 pictures but it just could not upload all the pictures if i choose together 10 pics so may be 6 or 7 would be upload and others not. One important thing we are using network with repository and with this connector we are sending pictuers to our repository to save them. Its not just straight thingworx repository.

       

      As you can see picture below of my mashup.. What do you think what can be the reason? do you think its because of network performance that it cannot handle it? because if i upload one by one then everything works fine but if i choose together 9 or 10 pictures then NOT. 

       

       

      MA8731174_0-1729669778650.png

       

      Rocko
      Rocko19-TanzaniteAnswer
      19-Tanzanite
      October 23, 2024

      We can't guess the problem, there could be many things causing it and you didn't give an error message.

      It could be anything, maybe it's trying to save in parallel but you can have only a fixed number of connections or file handles.  Try to find the error message and drill down from there. You can also try to simulate multifile upload in a service, without the widget to isolate the issue. You could also try to "buffer" the upload in a local repo and move to the network repo later on.

        Terms & ConditionsCookie settingsAccessibility statement