Implement and manage a custom or inbuilt queue using single data writer to stream

[Soura]

I want to achieve a custom(or inbuilt) queue processor. The objective is to have a admin user who has Write permissions to write to a stream. Other non-admin users can queue write operations which later get processed by the admin user.

I tried the same using event subscription and scheduler but the security context is passed on to the final service call and as the final service doesn't have the permissions set for the non-admin users, they are not able to execute.

I hope my understanding of using the internal event queue is correct.

1. Is there a way to use the event subscription queue be processed by a specific admin user?
2. What would be the best way to implement and manage a queue using a single admin user who performs the write operations? Is there any option that the platform supports already?

Note: For custom queue implementation concurrency, synchronization could be potential issues.

[BABA-SHYAM]

Hi,

As per my experience with queue says whenever we handle queues with events of subscription, thingworx entities performs poorly or just ended up being ghost entities.

My suggestion would be whatever you are doing with queue, just create some datatables in external DB( sql server), now run some subscription which will pick that say top 50 stored data, perform your action may be in for loop and then remove those entities from that table and achieve them if business demands.

I hope I have understood your question correctly and answered it properly.

Regards

Shyam

[Soura]

Hi @BABA-SHYAM ,

Thanks for your response. I was expecting an inbuilt queue from platform.

Regards

[VladimirRosu · Accepted solution]

Hi Soura,

The built-in ThingWorx Stream and Event subsystems do contain a queue, but that is used internally by the platform APIs to queue events to them and are not expected to be directly used by any platform user. While you can see their state, that's more to understand platform health.

In your case, I would go with an approach similar to Baba Shyam, but using platform features.

That is: create a Stream A that holds only the data queued by normal users. When the Administrator would want to approve, he is going to basically move the data to the proper stream B (via querying data from Stream A,  adding data to Stream B, then deleting the processed data from Stream A). Users would have permissions to queue entries for Stream A, but no permissions for Stream B.

This is will work as far as I understood your case, but beware that within ThingWorx we don't have a way to impersonate another user in your services (besides the Scheduler/Timer capability, which is not a proper way for your usecase).

Another small note here, even if I don't really think this is what stopped you: I'm not sure if you used the System User capability before. This might help your usecase, because, if you give a Service permission to the System User, that service will be capable to be executed by another service, but not directly by an user. https://support.ptc.com/help/thingworx/platform/r9/en/index.html#page/ThingWorx%2FHelp%2FComposer%2FSecurity%2FSystemUser.html

[Soura]

Hi @VladimirRosu ,

Thanks for your response. I had similar thoughts for this implementation. Additionally I use a timer(at 5 -10 sec) to process Stream A by the Administrator user. As Stream A would have only unprocessed data it might not be a lot of data load. We want the Stream A to be processed as soon as it queues up. Is there an alternative to timers like events(without passing security context to the subscription) or notification etc. which can be used? Please clarify why timer wont be a proper way for this use case ?

System user did cross my mind and it would not fit our use case.

The idea is to have a single admin user have the data write permissions.

[VladimirRosu]

Timers are not a proper way to impersonate users because when users desire such a capability they don't think about just one user - they want to impersonate any user, no matter who that user would be. Since the Timer accepts only one user, to accomplish such an usecase, with N number of users to impersonate would require to create dynamically timers, etc...
Again, a concern not valid if you just have a single user for impersonation.

[Soura]

Hi @VladimirRosu,

The idea is to is to have a single admin user who has Write permissions to write to a stream, rest all can queue requests.

I was thinking timers may not be appropriate due to performance issues. Your thoughts wrt performance for timers for(5-10 secs)?

Regards

[VladimirRosu]

All depends on the performance needs. If you have a single Timer which does this process, and "processes" only a bulk of "10 entries" when it ticks, then all will be good. The key here is to make sure that you're processing ONLY the amount of data you're sure it can be processed until the next tick.