File Repository has 1 million photos

[MA8731174]

Hello Community,

  I have file repository which is running with network link. I have uploader widget and i upload the 10 pictures each pic is ok 100kb but uploader only upload randomly 3 4 pics in one go and other would not get upload and failed. 

When i upload the pictures on development server it works completely fine and upto 10 pictures can upload in one go with selection of upload widget.

Do you guys think that is there any solution which can help me here? I want to just upload the pictures and for that my service creates path first for the folder creation in repository and then pictures get uploaded there with the upload widget on mashup. 

Any solutions are welcome here. 

@Rocko if you would like to share your insights, it would be great!

Thank you!

[Constantine]

Hello,

You can split it into folders, similar to HashMap buckets. In your mashup with file upload widgets, use an Expression to generate a random integer from 1 to 1000. Then use this integer as the folder name where you upload your files. This way you'd have 1000 folders, each with approx. 1000 files.

Regards,

Constantine

[Rocko]

As @Constantine suggested, do not have 1mil files in one folder. You could create folders per random number or a structure like year/month/day, in any case you would then also need a database (or a logic) where you store the correct path to find it back. (Note with 1mil rows you shouldn't use a Datatable)

On the issue that it uploads 10 files in one go on dev but not on prod, check the logs for the error and try to find the source. As it works on the one system, it should work on the other unless some configuration is different between the systems, e.g. how you connect to the system, encryption, timeouts, latency, network, etc.

[MA8731174]

@Constantine @Rocko  Thanks both of you for your feedback.

Actually at the moment we have folder for every picture which can be taken and saved. It is working same as @Rocko mentioned with datetime folder so here is the structure serialNumber/Timestamp/stepnumber --> 222/12345678/1 , 222/12345678/2 , 222/12345678/3 

so as you can see it saves the pictures like this so the 1,2,3 they are also name of folders and in them we have image. thats how we are doing since few years... now i made a mashup where there is one field called serialNumber and user gives serialnumber there and press next and on press next i create a path and the path would be for eg  222/12345678/4 and in this folder name 4 i upload around 10 pictures but it just could not upload all the pictures if i choose together 10 pics so may be 6 or 7 would be upload and others not. One important thing we are using network with repository and with this connector we are sending pictuers to our repository to save them. Its not just straight thingworx repository.

As you can see picture below of my mashup.. What do you think what can be the reason? do you think its because of network performance that it cannot handle it? because if i upload one by one then everything works fine but if i choose together 9 or 10 pictures then NOT. 

[Rocko · Accepted solution]

We can't guess the problem, there could be many things causing it and you didn't give an error message.

It could be anything, maybe it's trying to save in parallel but you can have only a fixed number of connections or file handles.  Try to find the error message and drill down from there. You can also try to simulate multifile upload in a service, without the widget to isolate the issue. You could also try to "buffer" the upload in a local repo and move to the network repo later on.

[Constantine]

I'm not sure if it uploads each file individually, or it's just a single large multipart request. You can check it in your browser's DevTools > Network tab. If it's a single large request, then the reason would likely be one of the following:

1. Upload timeout -- in this case your requests will be failing after the same time, e.g. 300 seconds,
2. File size limitations -- in this case your requests will be failing once they reached the same size, e.g. 100 MB.

You should be able to detect both of those in the same DevTools > Network tab.

If it turns out that each file is uploaded as an individual HTTP request, then you need to start checking logs, as @Rocko suggests.

[MA8731174]

Thanks @Rocko and @Constantine .  It works!

As you guys gave your insights .....So, I have created a service with which i upload the files first to local repository in thingworx, which is working completely fine and fast. Then i take these pictures one by one using saveImage service and save the pictures from there to the network connector repository. 

My service is as below...

let repository = "ExportRepository";


let pictureListPaths = Things[repository].GetFileListingWithLinks({path:path});
let binaryData;
let i=0;

pictureListPaths.rows.forEach(row=>{

    try {     
      binaryData =  Things[repository].LoadBinary({path:row.path});
        
    } catch(e){
        
    logger.info(" There is an error in LoadBinary Service   "+e);   
    }
    
    
    try {
      Things["export.FileRepository.Connector"].SaveImage({
                    path: path + "/" + (i + 1) + ".png",
                    content: binaryData
                });
    } catch(e){
    logger.info(" There is an error in SaveImage Service   "+e);
    }
    
    i++;
});

let result = pictureListPaths;

Now i have one question the 

uploadComplete event of the fileuploaderwidget gets executed for all the files uploaded so if i choose 10 pictures then it executes 10 times and i cannot use this event to execute the service which takes the pictures from local repo to connector repo.. I mean its not a big issue now but still wondering that how can i get the event call only for one time when all the selected files are uploaded... at the time i am using instandupload ..any idea about it? I can also do that with additional new button on mashup that can be visible when upload complete and then user can press it to upload pics from local repo to network repo but i do not like this one additional click much....

[Rocko]

You can have a Timer regularly checking if there are documents in that repository, and if so, take the last 10 or so and move them over.

Use a flag/property to check if the previous timer is still running (might happen if there are large files which take longer).

Don't set Timer frequency too high. You will find more information on how to use timers and schedulers properly here in the community.

[Constantine]

You can still call it on each upload, but instead of i = 0; get the max value by checking existing files in your connector repo

// Before doing that make sure that path exists
const existing = Things['export.FileRepository.Connector'].GetFileListing({ path: path }).rows.toArray().map(f => Number(f.name.split('.')[0]));
let i = Math.max.apply(null, existing);

As a nice bonus you can make it work even with parallel uploads, if you move this snippet into the loop:

// Before calling this, make sure that path exists

const source = Things["ExportRepository"];
const target = Things["export.FileRepository.Connector"];

source.GetFileListingWithLinks({ path:path }).rows.forEach(row => {
    try {
        const existing = target.GetFileListing({ path: path }).rows.toArray().map(f => Number(f.name.split('.')[0]));
        const next = Math.max.apply(null, existing) + 1;
        target.SaveImage({
            path: path + "/" + next + ".png",
            content: source.LoadBinary({ path: row.path })
        });
    } catch(e) {
        // ...
    }
});

[MA8731174]

So, I got issue again even though i was doing well copying images from repo and then  I have disable instanduploading from widget and use upload button and now i have no issues at all. Surpisingly the uploadComplete event is also working well now and it get executed only when all the selected pictures would be uploaded. 

Thank you both of you!