Skip to main content
G
GaryMansell12-Amethyst
12-Amethyst
June 25, 2016
Question

How to get remote csv data into ThingWorx?

  • June 25, 2016
  • 1 reply
  • 2702 views

Hi,

I am new to ThingWorx and would really appreciate a bit of guidance as to the best way of approaching a new project we want to do.

We will have a number of remote Raspberry Pi's running an application that collects time series data into a csv file held local to the device. The csv data table on each device will not be huge - a few 1000's of rows at the most, but data will be continually added to the csv file all the time the application is running on the Pi.

The Rapsberry Pi will be connected via WIFI to a mobile phone WIiFi hotspot to provide Internet connectivity, but there may be periods of time where the connectivity is not 100% and it may drop for short periods of time. Hence, the csv file on the remote device would act as a permanent data store (and also a buffer) for when the connectivity may drop to the ThingWorx server briefly.


It seems to me that the ThingWorx server is going to have to regularly poll the remote Raspberry Pi (so that I can show a near real-time data mashup) and will need some "smarts" so that it only pulls/requests new data that it is missing rather than the whole file each time. ThingWorx will need to persist this data itself so that I can view the historical data on the mashup as well as real-time.

I would really appreciate some pointers as to how to best achieve this sort of solution with ThingWorx

Thanks in advance

Gary

1 reply

P
PaiChung22-Sapphire I
22-Sapphire I
June 28, 2016

Using the Edge Microserver you can easily do a filetransfer and parse Server side.

or even post content to Thingworx.

Not sure what restrictions you face, but I think that would work better.

G
GaryMansell12-AmethystAuthor
12-Amethyst
June 28, 2016

Thanks for your replying, it's much appreciated, I have been giving this more consideration, and I think I might have an idea as to how to do this now.

What I was thinking of doing was to setup apache on the Pi with a PHP script page that will parse the local CSV file and only send back the lines of data more recent than the timestamp that the PHP page receives from the ThingWorx Server. The ThingWorx server would send the timestamp of the last data that it successfully received and stored, so if there is a connectivity outage for a period, then it would collect the backlog of data from the Pi.


The data could be sent from the apache server on the Pi to the ThingWorx Server in JSON format so that it can then be decoded and stored.

I think I would need to setup dynamic DNS on the Pi so that it could be accessed from the ThingWorx server on the Internet by name, and I may need to ensure that the clock is correct on the Pi with ntp or similar, but it seems to me that this should work.

What do you think - do you see any flaws in my thinking?

Rgds

Gary

P
PaiChung22-Sapphire I
22-Sapphire I
June 28, 2016

If you have Thingworx, I believe you also have access to our Agent which will broker a much better and more secure connection to the platform. (just a thought)

Not necessarily anything wrong with the way you are looking to do it, it'll be extra work to track 'offline time' of your devices.

As far as the timestamp, there are timesync servers out there that you can hit to set the clock on your PI and then just make sure you send time in UTC

Terms & ConditionsAccessibility statement