DataTable rows limit

[fmanniti]

Hello, I took some data from a stream and copied into a datatables as entries.

I have a data every 30 minutes so it means 48 per day.

I copied about 6 months of data and, if I go to check the service GetDataTableEntryCount() it returns me 8995 but if I plot on a Time Series Chart the result of GetDataTableEntries() it only shows 10 days (so like 500 rows).

I get the same problem if I bid data with a grid.

I guess it is because it automatically takes the maxItem limit at its default value 500 although I left it blank ; but is there a way to leave it open?

I don't have the same problem if I use stream instead so do you think for a big amount of data (like over 10000 rows) stream is better than DataTable?

[AdamR · Accepted solution]

Hi Fabio,

The performace of DataTables vs Streams depends on the type of data and how it will be queried.  Stream generally speaking are best for time series data and are optimized to pull data quickly on a date range.  DataTables are faster when referencing a non datetime field as the primary filter.  HOWEVER, they are intended for small scale more static data, if volume exceed about 100,000 rows performance will drop quickly.  A more detailed explanation can be found here...

https://community.thingworx.com/community/developers/blog/2016/03/11/where-should-i-store-my-thingworx-data

Thanks,
Adam

[fmanniti]

Thank you very much, Adam. Your answer is really helpful.
Still I have some doubt. I mean: I have maximum 20.000 entries so Data Table could be good, but the primary field is a DateTime so probably Stream could be better.

I'm still thinking about this. I'll let you know how I'll come out.

Thanks again