Which is the most efficient way for retrieve entities? TWX 8.5

[LuisCabello]

Hello!,

I'm developing a very huge aplication and we will have a lot of devices (more than 1 million).

The main thing is that the app has to have the posibility of retrieve one o two from the one million of device when It will be necesary. Which is the most efficient way for to do that?

These are the options that I thougth:

1. GetImplementingThingsWithData it's efficient (with a query parameter)?

2. It's better to have all this data in a database and make a query? 

3. Create a network tree entity it's a good idea?

Thank you in advance.

Luis :D.

[PaiChung]

Options 1 and 3 mean you will load one million rows of data into memory, that isn't a good way to go.

If you can group your devices into smaller numbers you can do an iterative approach or a 'pagination' approach

Database and Query would work

Any which way you do this, I recommend you device some way to 'paginate' so you aren't loading 1 million records at one time.

[LuisCabello]

Hello @PaiChung thank you a lot for your answer,

One question respect GetImplementingThingsWithData, when you use this fuction, which is the correct option?:

1. First it retieve all entities from a template and filter with the query ( so, first TWX has to load millions of devices)

2. Only retrieve the things that satisfy the query parameter (imagin that I have 1 million devices/things and I want retrieve only the device/thing with id=011, so only load in memory one device).

Thank you in advance.

Luis

[PaiChung]

If you have 1 million items, I recommend against using GetImplementingThingsWithData you could consider using QueryImplementingThingsWithData

With the GET it will always retrieve all entities and their property values, this would be too much.

With Query I would be careful, there are limited indexes that would allow you to use Query and not retrieve all 1 million directly after which the remainder of the query would be applied. I the case of entities I think only Model Tags would be an indexed field allowing you to retrieve less than your 1 million entities on a first request.

All that said, perhaps you can spec your system to have plenty of processing power and memory to handle this, however I recommend designing things such that you are always avoiding pulling 1 million entities and their info.

[LuisCabello]

Hello @PaiChung  thank for you answer :).

About the plenty of processing and memory this has to be solved, we have a High Availability architecture.  The most important thing it's how to seach in 1 million devices only one in a efficient way. If I understood well, the main approach it's use QueryImplementingThingsWithData + Model tags, isn't it?.

With this approach, I thinking in to have model tags with unique ID and seach with this ID. How do you see this approach?

Thank a lot for your answers.

Luis.

[PaiChung · Accepted solution]

Potentially a separate database with optimized views/queries might work better. Difficult to predict.

But else you just make sure you have grouped your Things with your tags so you can 'paginate' through them.