Skip to main content
G
gagam1-Visitor
1-Visitor
January 25, 2018
Question

Best server architecture for 10M rows IoT stream and an advanced graphic front-end

  • January 25, 2018
  • 6 replies
  • 4292 views

Hi all,

I'm desperately trying to understand best practices for creating the platform described below. I can't find an answer for whether to use remote or internal DB, a Thing-centered data structure or traditional tables and whether to develop the front-end on mashups or on an external server with API to ThingWorx.

What we're trying to do is:

  1. IoT data stream is uploaded at a rate of 10 rows/sec from up to 20 devices simultaneously. we expect 10M rows a year. Would it require an external DB?
  2. This data is never displayed raw, but rather is used to calculate additional, much smaller tables, that are the source of our charts. I was told a non-relational DB is the best solution for this kind of data handling (Mongo, Google Firebase etc.) but I didn't manage to understand their compatibility with ThingWorx.
  3. We are developing consumer-grade front-end, customizing almost every component. Would't it be easier to develop and run it on an external server and just get the data from ThingWorx through its API, rather than trying to create it using Mashups and customized widgets?

And one last question I can't wrap my head around: If ThingWorx is a Thing-Centered system, wouldn't it 'degrade' its performance if we work with traditional table-based relational DB? Are we even utilizing any of ThingWorx advantages in the field of data management here, or just using it as a portal for accessing the external DB?

Iv'e already read Where Should I Store My Thingworx Data? and Remote or internal database? and many other community threads on these issue, but can't get the big picture.

Any help would be appreciated, thanks.

Gilad.

6 replies

W
wposner-21-Visitor
1-Visitor
January 25, 2018

Your IoT use  is fairly standard. The twx thing model and associated value streams are all you need.  You can build reports by issuing quer against your value streams.  As for your front end, go with something like Angular and Node/Hapi. Hapi will be your call mtrollwr layer and implement the REST calls back to your twx services. Angular will give you the front end flexibility you require.

G
gagam1-VisitorAuthor
1-Visitor
January 25, 2018

Thank you for the reply Wayne.

Sorry if I misunderstood, but do you mean I don't need a dedicated DB server? I mean, where are those value streams stored?

As for the front end, we do intend to use Angular / React, but can it be coded directly to TWX or is a separate server better?

W
wposner-21-Visitor
1-Visitor
January 25, 2018

TWX ships with support for various persistence providers--among them are PostgreSQL, SQL Server, and Cassandra.  Depending on the IoT use-case and licensing limitations, one of default providers will be more appropriate than the others.  PostgreSQL is free and is the default persistence provider option for TWX based on standard licensing.  The default persistence provider is where TWX will store it's representation of the thing model, data tables, and value streams. 

So for a true IoT deployment where incoming data is typical of sensor values and other similar attributes coming from edge devices, PostgresSQL is robust enough to handle the load and no additional DB is required.  I've been told that PTC has tested TWX using PostgresSQL with up to 1,000,000 incoming connections and they had no issues.  If you're going to be significantly above that number of edge devices (and depending on the amount of data being received per device) you may want to start to consider something like Cassandra as the default persistence provider. 

P
pselvaraj-11-Visitor
1-Visitor
May 7, 2018

Hi,

 

First, check if ThingWorx is really required for your use cases. Because for Non-IoT use cases you can develop using latest web technologies as already you might have few in the market. 

 

If ThingWorx is required then as per your data input frequency you must go for external database. PostgreSQL may drop in performance if the database size exceeds 500GB. In case if you use default persistance provider for your needs, you data ingestion thing always busy inserting data into you postgres database. you can feel the slowness over the period since your thingworx composer itself is stored and drwan from default persistance provider.

 

If at all you have developed a application using postgresql, it may respond quick enough for single user. But think for if multiple users logged in and perform some time consuming tasks on your application. So, it is advisible to go for external db for you data ingestion rate and volume.

 

Terms & ConditionsAccessibility statement