How to get remote csv data into ThingWorx?

[GaryMansell]

Hi,

I am new to ThingWorx and would really appreciate a bit of guidance as to the best way of approaching a new project we want to do.

We will have a number of remote Raspberry Pi's running an application that collects time series data into a csv file held local to the device. The csv data table on each device will not be huge - a few 1000's of rows at the most, but data will be continually added to the csv file all the time the application is running on the Pi.

The Rapsberry Pi will be connected via WIFI to a mobile phone WIiFi hotspot to provide Internet connectivity, but there may be periods of time where the connectivity is not 100% and it may drop for short periods of time. Hence, the csv file on the remote device would act as a permanent data store (and also a buffer) for when the connectivity may drop to the ThingWorx server briefly.

It seems to me that the ThingWorx server is going to have to regularly poll the remote Raspberry Pi (so that I can show a near real-time data mashup) and will need some "smarts" so that it only pulls/requests new data that it is missing rather than the whole file each time. ThingWorx will need to persist this data itself so that I can view the historical data on the mashup as well as real-time.

I would really appreciate some pointers as to how to best achieve this sort of solution with ThingWorx

Thanks in advance

Gary

[PaiChung]

Using the Edge Microserver you can easily do a filetransfer and parse Server side.

or even post content to Thingworx.

Not sure what restrictions you face, but I think that would work better.

[GaryMansell]

Thanks for your replying, it's much appreciated, I have been giving this more consideration, and I think I might have an idea as to how to do this now.

What I was thinking of doing was to setup apache on the Pi with a PHP script page that will parse the local CSV file and only send back the lines of data more recent than the timestamp that the PHP page receives from the ThingWorx Server. The ThingWorx server would send the timestamp of the last data that it successfully received and stored, so if there is a connectivity outage for a period, then it would collect the backlog of data from the Pi.

The data could be sent from the apache server on the Pi to the ThingWorx Server in JSON format so that it can then be decoded and stored.

I think I would need to setup dynamic DNS on the Pi so that it could be accessed from the ThingWorx server on the Internet by name, and I may need to ensure that the clock is correct on the Pi with ntp or similar, but it seems to me that this should work.

What do you think - do you see any flaws in my thinking?

Rgds

Gary

[PaiChung]

If you have Thingworx, I believe you also have access to our Agent which will broker a much better and more secure connection to the platform. (just a thought)

Not necessarily anything wrong with the way you are looking to do it, it'll be extra work to track 'offline time' of your devices.

As far as the timestamp, there are timesync servers out there that you can hit to set the clock on your PI and then just make sure you send time in UTC

[GaryMansell]

This is to be a simple Proof of Concept with very limited budget and limited skilled resource availability at this time - I just want to show the art of the possible for now on a small scale to try and engender some interest in others.

What is the agent that you refer to - are you referring to an SDK as I was trying to avoid too much programming as this is not my forte? A bit of PHP scripting i can manage, but java or c++ is a bit out of my comfort zone