Best practise to store historical data plus live streaming data in a Thing

[nagrawal-2]

Hi,

What should be the best practice to store the historical data from Excel and then later start storing Live streaming data in same properties of a Thing. I know as per the documentation, I should use Datatables to store static or historical data and Value Streams for Live streaming data but my setup is different. Initially, I need to push some Historical data of a device and later the same device will start pushing live data in TW. I am not sure how would I proceed to achieve this.

Please suggest me the best way possible.

Thanks in advance.

Best regards,

Neetu

[CarlesColl]

If it's the same data that will be recorded live, than all have to go to the same storage, in this case Stream or ValueStream.

[nagrawal-2]

Hi Carles,

Thanks for your response.

Yes format of data will be same when recording live. But initially I will be pushing the data from CSV ? Can I push the CSV data in Value Stream?

Regards,

Neetu

[CarlesColl]

Yes for sure, you can read CSV files with the following extensions:

• https://marketplace.thingworx.com/tools/csv-parser​
• https://marketplace.thingworx.com/tools/Parsley_Extension

An you can insert data directly on the ValueStream with property owner Thing service:

• Add+baseType+ValueStreamEntry

We usually combine booth without issues.

[nagrawal-2]

Thanks alot Carles, I will try this way.

Regards,

Neetu

[nagrawal-2]

Hi Carles,

I am able to read data from CSV file using CSV extension and display it in Mashup in Grid widget. I am not sure how would I push data in ValueStream.

Do I need to create ValueStream and related properties in a Thing?

Regards,

Neetu

[CarlesColl]

Hi Neetu,

Look at my original post the second part:

An you can insert data directly on the ValueStream with property owner Thing service:

• Add+baseType+ValueStreamEntry

[nagrawal-2]

Yes this is what I don't understand. Do you have any example to help me ?

Regards,

Neetu

[nagrawal-2]

Hi Carles,

I am able to store the historical data using UpdatePropertyValue service. But when I bind my Property to Time series chart, it is not showing all data values. It is only displaying 30 values out of 260. I don't know what went wrong?

Also, I want to use Date picker widget on my Time series chart so User can see some historical data, would it be possible?

Regards,

Neetu

[CarlesColl]

Hi Neetu,

Did you query Property values through the composer and checked that you have more than 30 values on the history?

Using date picker to query historical data it's not a problem at all, you just need to use startDate and endDate parameters on the property Query services.

[nagrawal-2]

I used QueryNumberPropertyHistory and just came to know the data is storing in chunks. I uploaded CSV file with 238 records and only data got saved in ValueStream is 30 records and strangely random data not in ascending or descending order.

I probably be using wrong format of date in CSV

2016-01-31,608,0.0976,20,7.1

2016-02-29,558.0457329,0.1052,18,7.2

2016-03-31,449.9114189,0.08536,22,7.2

2016-04-30,464.1673867,0.0876,25,7.1

I have uploaded so many data in many entities and now realized they all got uploaded wrongly.

[CarlesColl]

Hi,

For what I see on your csv file, you have only Date  and you don't have Time, then if you import that data all the Data at the same day will be overwritten ( I don't know if this is the problem ).

Query one property by one with "Query"+baseType+"PropertyHistory" to see which data it's recorded on each Property

[nagrawal-2 · Accepted solution]

Hi Carles,

I have resolved the issue. It's not about passing the date-time value in CSV, I had to change the date format in params object:

dateformat: "yyyy-MM-dd HH:mm:ss"

The string should be in exact above format (Case sensitive) to make it work.

Thanks for your support.

Regards,

Neetu