A design problem of FileTransferSubsystem

[htran-21]

Hi,

I have tried to call Copy({ async: true, queueable: true, ... }) 13 times, and there were ONLY 10 files copied. I have figured out that there is a setting called maxOfflineQueuePerThingSize. That's fine. The problem is, GetQueuedTransferJobCountByThing() does not return a number of jobs queued when the Copy above is called => I would be unable to check when I should not queue more Copy tasks.

My Bests,

Hung Tran

[slangley]

Hi @htran-21.

Which version of ThingWorx are you running?

Regards.

--Sharon

[htran-21]

Hi Sharon,

It is 7.4.19-b262

My Bests,

Hung Tran

[PaiChung]

Do the other services like GetQueuedTransferJobs or GetQueuedTransferJobCount give you different counts?

vs. the by Thing?

These services are available in the FileTransfer Subsystem

[htran-21]

Hi PaiChung,

I tried them all, I guess they work based on data in file_transfer_job and file_transfer_job_offline_queue table. At the moment Copy() is called, they returned no result. It might be that they did not run in the same transaction, therefore they didn't see incoming rows inserted.

    $logger.debug(IPU.isDebug, JSON.stringify(fileTransferJob))
    let ds = FileTransferSubsystem.Copy(fileTransferJob)

    const tid = ds.rows[0].transferId
    $logger.debug(IPU.isDebug, ' IsTransferJobActive:' + FileTransferSubsystem.IsTransferJobActive({tid: tid}) 
                 	     + ' IsTransferJobQueued:' + FileTransferSubsystem.IsTransferJobQueued({tid: tid})
                  	     + ' GetQueuedTransferJobCountByThing: ' + FileTransferSubsystem.GetQueuedTransferJobCountByThing ({ thingName: IPUName }))

    return ds  

A log shows that such methods did not recognize a new job queued.

My Bests,

Hung Tran

[slangley]

Hi @htran-21.

We would need to have more information on the files being transferred.  For example, is the size exceeding the limit?  The errors should appear in the logs on the edge where you are executing the copy. 

There are settings on the edge side and the FileTransfer Subsystem on the platform side to tune for your needs.  If you can provide more information regarding your method, we can point you to some articles for tuning.

Regards.

--Sharon

[htran-21]

Hi Sharon,

There is no error noticed, those files are transferred well.That's just a problem related to GetQueuedTransferJobCountByThing(), it does not return a number of jobs queued at the moment when Copy is called. A test code is simple, just need to create a test service with script content as following

...
fileTransferJob = { async: true, queueable: true, ... }
IPUName
...

FileTransferSubsystem.Copy(fileTransferJob) // file 1
tid = ds.rows[0].transferId
logger.debug(' IsTransferJobActive:' + FileTransferSubsystem.IsTransferJobActive({tid: tid}) 
           + ' IsTransferJobQueued:' + FileTransferSubsystem.IsTransferJobQueued({tid: tid})
           + ' GetQueuedTransferJobCountByThing: ' + FileTransferSubsystem.GetQueuedTransferJobCountByThing ({ thingName: IPUName }))

// OUTPUT: IsTransferJobActive: false IsTransferJobQueued: false GetQueuedTransferJobCountByThing: 0

FileTransferSubsystem.Copy(fileTransferJob) // file 2
tid = ds.rows[0].transferId
logger.debug(' IsTransferJobActive:' + FileTransferSubsystem.IsTransferJobActive({tid: tid}) + ' IsTransferJobQueued:' + FileTransferSubsystem.IsTransferJobQueued({tid: tid}) + ' GetQueuedTransferJobCountByThing: ' + FileTransferSubsystem.GetQueuedTransferJobCountByThing ({ thingName: IPUName }))

// OUTPUT: IsTransferJobActive: false IsTransferJobQueued: false GetQueuedTransferJobCountByThing: 0

FileTransferSubsystem.Copy(fileTransferJob) // file 3
tid = ds.rows[0].transferId 
logger.debug(' IsTransferJobActive:' + FileTransferSubsystem.IsTransferJobActive({tid: tid}) + ' IsTransferJobQueued:' + FileTransferSubsystem.IsTransferJobQueued({tid: tid}) + ' GetQueuedTransferJobCountByThing: ' + FileTransferSubsystem.GetQueuedTransferJobCountByThing ({ thingName: IPUName }))

// OUTPUT: IsTransferJobActive: false IsTransferJobQueued: false GetQueuedTransferJobCountByThing: 0

....

My Bests,

Hung Tran

[slangley]

Hi @htran-21.

We performed some testing under ThingWorx 8.4.x to try to understand the behavior you're seeing.  We initiated the test from the platform side and did receive an error in the Application Log:

Unable to Invoke Service Copy on FileTransferSubsystem : Unable to enqueue file transfer since offline queue for the Thing is full. maxQueuePerThingSize=10

Notice the maxQueuePerThingSize is set to 10 in the Configuration for the FileTransferSubsystem:

Since you are running a ThingWorx version that is no longer supported, you will have to validate this option exists in your version.  If so, you could set it to a higher limit than you would expect to hit.

Also, if you are receiving an error, you could set up a subscription to be alerted when the queue size has exceeded the max allowed.

Hopefully, this provides some guidance.  If there is actually a bug in the version of the product you are running, you would need to upgrade to resolve it.

Regards.

--Sharon

[htran-21]

Hi slangley,

Thank you to look into that, however you misunderstood what i mean. I did not mean about the maxQueuePerThingSize, it exists. My test case is showing that there is no way to check the number of transfer jobs queued by a Thing, FileTransferSubsystem.GetQueuedTransferJobCountByThing() always returns the same value no matter how many transfer jobs are queued in scope of a service.

My Bests,

Hung Tran

[slangley]

Hi @htran-21.

When you say it's returning the same value, what is that value?  We have tested this under the latest version (8.4) and it appears to work as expected.  We recommend you test it under the latest release of the product, as the version you are currently running is no longer supported.

Regards.

--Sharon

[htran-21]

Hi Sharon,

I have run the test on ThingWorx 8.4.0-b1379, there is the same problem. The test case is attached.

My Bests,

Hung Tran

[slangley]

Hi @htran-21.

From reviewing your code, it doesn't appear that you are set up to perform a valid test.  You're simply moving files from one folder to another on the same system which will run so quickly, it may not even appear to queue up.  Have you tried this with an actual remote device while it's in a disconnected state?  That's what this is designed for.

Regards.

--Sharon

[htran-21]

Hi Sharon,

That's just a sample test code, the actual code is copying files to remote devices (14-26 file transfer tasks are queued to a device at a time, the device is connected at the time), the issue is there is no counter or a signal or an  error that let me know when reaches the maxOfflineQueuePerThingSize (= 10). You could update the test code to move some files to one of your remote device, or use big files (this is a copy, not move, it would take longer with big files) to reproduce the case.

At this moment, i have to add a run-time counter into the service to work around it, however that's not a safe solution in term of a concurrent processing.

My Bests,

Hung Tran

[slangley]

Hi @htran-21.

You could create your own custom service that would do 2 things:

1. Run an "if then else" statement to check the queue to determine the count, and if already equal to the max queue size, terminate the service and produce an error to advise users.
2. If below the max queue size, allow the copy to continue.

We validated that an error is thrown when the maxQueuePerThingSize is exceeded, so you could also subscribe to this event, though the first option would be a better solution.

Regards.

[htran-21]

Hi Sharon,

1/ I don't understand about your solution "if then else", may you create a pseudo code? Do you mean a local counter cause the GetQueuedTransferJobCountByThing() always return the same value no matter how many Copy tasks are queued in the execution scope of a service?

2/ Which event do you mention? FileTransfer?

My Bests,

Hung Tran

[slangley]

Hi @htran-21.

1)  Check this site for more info

2) I was referring to the error event thrown when the maxQueuePerThingSize is exceeded.  You can set up a subscription to be notified when this occurs.

Regards.

--Sharon

[htran-21]

Hi Sharon,

1/ I did not mean about the if else syntax, I mean about the pseudo logic that you mentioned.

2/ Please give me a bit more information how to set up a subscription to be notified when this error occurs. What is the name of Event, is it FileTransfer? Anyway, i don't think that will work, cause the code execution of main service will not never know about this notification.

My Bests,

Hung Tran

[slangley]

Hi @htran-21.

We generally don't provide code but here is a simple script that should help you to get started:

var currentQueueSize =  Subsystems["FileTransferSubsystem"].GetQueuedTransferJobCountByThing({
    thingName: "EdgeThingNameHere" /* STRING */
});
if(currentQueueSize < 10) {
   // result: INFOTABLE dataShape: "FileTransferJob"
    var FTCopy =  Subsystems["FileTransferSubsystem"].Copy({
        async: undefined /* BOOLEAN */,
        sourceRepo: undefined /* STRING */,
        metadata: undefined /* JSON */,
        targetRepo: undefined /* STRING */,
        targetFile: undefined /* STRING */,
        targetPath: undefined /* STRING */,
        queueable: undefined /* BOOLEAN */,
        sourceFile: undefined /* STRING */,
        sourcePath: undefined /* STRING */,
        timeout: undefined /* INTEGER */
    });
    
    result = true;
} else {
    result = false;
}

This example is checking the queue size for a single thing in question, but could write this in dynamic fashion to handle multiple devices.  The if condition checks the current queue size for the thing, and if less than the max specified, it allows the file transfer to proceed.  Otherwise, it will block the transfer.  In this case a boolean is being output but you could change it to output a message that could be tied to a button click in a mashup when a user tries to initiate a transfer.  No subscription would be required in this case.

The "FileTransfer" event should be thrown anytime the service is executed and you can subscribe to this event.  The event lives on the device.

Regards.

--Sharon

[htran-21]

Hi Sharon,

You are misunderstood the situation, the situation is that a service has to queue many files to device in one run, not 1 file like in your pseudo code.

I wrote a workaround that is exactly the same, use a local counter to keep queue size at begin and increase the counter when a Copy task is called, cause GetQueuedTransferJobCountByThing() always return the same value no matter how many Copy task is queued. However, this is not a good solution in a concurrent system because the queue size by that Thing may be changed in the middle by other services and that will lead to another issue.

This is just a report that GetQueuedTransferJobCountByThing() does not work as expected when many copy tasks are queued in one run.

My Bests,

Hung Tran

[slangley]

Hi @htran-21.

I think you misunderstood.  The service, GetQueuedTransferJobCountByThing, actually checks the queue for determining the count as opposed to using a local counter.  The service can be called as many times as needed.

Please also keep in mind that this code is being provided only as an example.  You will need to tailor it for your needs but at the most basic level, it contains everything needed to accomplish your task.

If you need assistance with your code, please reach out to our Sales team who can arrange consulting services to provide further  assistance.

Regards.

--Sharon

[htran-21]

Hi Sharon,

I think you did not read my sample code completely

It is always returning 0 no matter how many tasks are queued and no run-time exception is raised when queue exceed. That is obviously a BUG.

A test thing for demonstration is already attached in earlier post. Download link!

@slangley You don't need to give an answer if you feel uncomfortable or you think that is not a problem in Thingworx, that's just a waste of time on both.

My Bests,

Hung Tran

[slangley]

Hi @htran-21.

We actually did review the example you provided and set up our own test to validate the service was working successfully.  As previously stated, your example is not simulating a scenario that would typically be used for file transfer.  The way it's designed, it's really just moving files from one directory to another so the queue may be processing files so quickly, that you're not seeing them hit the queue.

If you would like to pursue this further and have support for ThingWorx, we can open a case in order to schedule a WebEx for further review and analysis.  We can demonstrate a more true-to-life use case, where you can see the queue count changing.  Please let me know if that is something you are interested in.

Regards.

--Sharon