cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Community Tip - You can subscribe to a forum, label or individual post and receive email notifications when someone posts a new topic or reply. Learn more! X

Which is the most efficient way for retrieve entities? TWX 8.5

LuisCabello
13-Aquamarine

Which is the most efficient way for retrieve entities? TWX 8.5

Hello!,

 

I'm developing a very huge aplication and we will have a lot of devices (more than 1 million).

The main thing is that the app has to have the posibility of retrieve one o two from the one million of device when It will be necesary. Which is the most efficient way for to do that?

These are the options that I thougth:

1. GetImplementingThingsWithData it's efficient (with a query parameter)?

2. It's better to have all this data in a database and make a query? 

3. Create a network tree entity it's a good idea?

 

Thank you in advance.

 

Luis :D.

ACCEPTED SOLUTION

Accepted Solutions
PaiChung
22-Sapphire I
(To:LuisCabello)

Potentially a separate database with optimized views/queries might work better. Difficult to predict.

But else you just make sure you have grouped your Things with your tags so you can 'paginate' through them.

View solution in original post

5 REPLIES 5
PaiChung
22-Sapphire I
(To:LuisCabello)

Options 1 and 3 mean you will load one million rows of data into memory, that isn't a good way to go.

If you can group your devices into smaller numbers you can do an iterative approach or a 'pagination' approach

Database and Query would work

Any which way you do this, I recommend you device some way to 'paginate' so you aren't loading 1 million records at one time.

LuisCabello
13-Aquamarine
(To:PaiChung)

Hello @PaiChung thank you a lot for your answer,

 

One question respect GetImplementingThingsWithData, when you use this fuction, which is the correct option?:

1. First it retieve all entities from a template and filter with the query ( so, first TWX has to load millions of devices)

2. Only retrieve the things that satisfy the query parameter (imagin that I have 1 million devices/things and I want retrieve only the device/thing with id=011, so only load in memory one device).

 

Thank you in advance.

 

Luis

PaiChung
22-Sapphire I
(To:LuisCabello)

If you have 1 million items, I recommend against using GetImplementingThingsWithData you could consider using QueryImplementingThingsWithData

With the GET it will always retrieve all entities and their property values, this would be too much.

With Query I would be careful, there are limited indexes that would allow you to use Query and not retrieve all 1 million directly after which the remainder of the query would be applied. I the case of entities I think only Model Tags would be an indexed field allowing you to retrieve less than your 1 million entities on a first request.

 

All that said, perhaps you can spec your system to have plenty of processing power and memory to handle this, however I recommend designing things such that you are always avoiding pulling 1 million entities and their info.

LuisCabello
13-Aquamarine
(To:PaiChung)

Hello @PaiChung  thank for you answer :).

 

About the plenty of processing and memory this has to be solved, we have a High Availability architecture.  The most important thing it's how to seach in 1 million devices only one in a efficient way. If I understood well, the main approach it's use QueryImplementingThingsWithData + Model tags, isn't it?.

 

With this approach, I thinking in to have model tags with unique ID and seach with this ID. How do you see this approach?

 

Thank a lot for your answers.

 

Luis.

PaiChung
22-Sapphire I
(To:LuisCabello)

Potentially a separate database with optimized views/queries might work better. Difficult to predict.

But else you just make sure you have grouped your Things with your tags so you can 'paginate' through them.

Announcements


Top Tags