Community Tip - You can subscribe to a forum, label or individual post and receive email notifications when someone posts a new topic or reply. Learn more! X
Hello!,
I'm developing a very huge aplication and we will have a lot of devices (more than 1 million).
The main thing is that the app has to have the posibility of retrieve one o two from the one million of device when It will be necesary. Which is the most efficient way for to do that?
These are the options that I thougth:
1. GetImplementingThingsWithData it's efficient (with a query parameter)?
2. It's better to have all this data in a database and make a query?
3. Create a network tree entity it's a good idea?
Thank you in advance.
Luis :D.
Solved! Go to Solution.
Potentially a separate database with optimized views/queries might work better. Difficult to predict.
But else you just make sure you have grouped your Things with your tags so you can 'paginate' through them.
Options 1 and 3 mean you will load one million rows of data into memory, that isn't a good way to go.
If you can group your devices into smaller numbers you can do an iterative approach or a 'pagination' approach
Database and Query would work
Any which way you do this, I recommend you device some way to 'paginate' so you aren't loading 1 million records at one time.
Hello @PaiChung thank you a lot for your answer,
One question respect GetImplementingThingsWithData, when you use this fuction, which is the correct option?:
1. First it retieve all entities from a template and filter with the query ( so, first TWX has to load millions of devices)
2. Only retrieve the things that satisfy the query parameter (imagin that I have 1 million devices/things and I want retrieve only the device/thing with id=011, so only load in memory one device).
Thank you in advance.
Luis
If you have 1 million items, I recommend against using GetImplementingThingsWithData you could consider using QueryImplementingThingsWithData
With the GET it will always retrieve all entities and their property values, this would be too much.
With Query I would be careful, there are limited indexes that would allow you to use Query and not retrieve all 1 million directly after which the remainder of the query would be applied. I the case of entities I think only Model Tags would be an indexed field allowing you to retrieve less than your 1 million entities on a first request.
All that said, perhaps you can spec your system to have plenty of processing power and memory to handle this, however I recommend designing things such that you are always avoiding pulling 1 million entities and their info.
Hello @PaiChung thank for you answer :).
About the plenty of processing and memory this has to be solved, we have a High Availability architecture. The most important thing it's how to seach in 1 million devices only one in a efficient way. If I understood well, the main approach it's use QueryImplementingThingsWithData + Model tags, isn't it?.
With this approach, I thinking in to have model tags with unique ID and seach with this ID. How do you see this approach?
Thank a lot for your answers.
Luis.
Potentially a separate database with optimized views/queries might work better. Difficult to predict.
But else you just make sure you have grouped your Things with your tags so you can 'paginate' through them.