Skip to main content
L
LuisCabello13-Aquamarine
13-Aquamarine
January 18, 2022
Solved

Which is the most efficient way for retrieve entities? TWX 8.5

  • January 18, 2022
  • 1 reply
  • 2208 views

Hello!,

 

I'm developing a very huge aplication and we will have a lot of devices (more than 1 million).

The main thing is that the app has to have the posibility of retrieve one o two from the one million of device when It will be necesary. Which is the most efficient way for to do that?

These are the options that I thougth:

1. GetImplementingThingsWithData it's efficient (with a query parameter)?

2. It's better to have all this data in a database and make a query? 

3. Create a network tree entity it's a good idea?

 

Thank you in advance.

 

Luis :D.

Best answer by PaiChung

Potentially a separate database with optimized views/queries might work better. Difficult to predict.

But else you just make sure you have grouped your Things with your tags so you can 'paginate' through them.

1 reply

P
PaiChung22-Sapphire I
22-Sapphire I
January 18, 2022

Options 1 and 3 mean you will load one million rows of data into memory, that isn't a good way to go.

If you can group your devices into smaller numbers you can do an iterative approach or a 'pagination' approach

Database and Query would work

Any which way you do this, I recommend you device some way to 'paginate' so you aren't loading 1 million records at one time.

    L
    LuisCabello13-AquamarineAuthor
    13-Aquamarine
    January 19, 2022

    Hello @PaiChung thank you a lot for your answer,

     

    One question respect GetImplementingThingsWithData, when you use this fuction, which is the correct option?:

    1. First it retieve all entities from a template and filter with the query ( so, first TWX has to load millions of devices)

    2. Only retrieve the things that satisfy the query parameter (imagin that I have 1 million devices/things and I want retrieve only the device/thing with id=011, so only load in memory one device).

     

    Thank you in advance.

     

    Luis

    P
    PaiChung22-Sapphire I
    22-Sapphire I
    January 19, 2022

    If you have 1 million items, I recommend against using GetImplementingThingsWithData you could consider using QueryImplementingThingsWithData

    With the GET it will always retrieve all entities and their property values, this would be too much.

    With Query I would be careful, there are limited indexes that would allow you to use Query and not retrieve all 1 million directly after which the remainder of the query would be applied. I the case of entities I think only Model Tags would be an indexed field allowing you to retrieve less than your 1 million entities on a first request.

     

    All that said, perhaps you can spec your system to have plenty of processing power and memory to handle this, however I recommend designing things such that you are always avoiding pulling 1 million entities and their info.

      Terms & ConditionsCookie settingsAccessibility statement