cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Community Tip - Need to share some code when posting a question or reply? Make sure to use the "Insert code sample" menu option. Learn more! X

Stream do not store any data

fpassiatore
7-Bedrock

Stream do not store any data

Hello,

I'm currently trying to set up a stream to store data.

I created a service that store data into that stream and update the properties of the related thing.
When I execute that service, the properties of the thing are updated but nothing appears in the stream.

The service is not returning any error.

I even tried to push data in the stream directly from the default service "AddStreamEntry" on the stream itself.
And even if the service return no error, nothing is added in the stream.

I'm wondering if it's not somehow related to the persistent provider (we are using Postgresql, and on a previous installation with default db everything was working fine)

Any idea on what I am missing here?

Thanks in advance

ACCEPTED SOLUTION

Accepted Solutions

Problem solved by downgrading the version of pgpool and postgresql...

Thanks for you help guys

View solution in original post

22 REPLIES 22
ttielebein
14-Alexandrite
(To:fpassiatore)

Are there any errors in the logs? Look under Monitoring > Application (top right corner of home-screen). Also check in the Script logs. Let me know!

Hi, no there is nothing in the logs

Aanjan
12-Amethyst
(To:fpassiatore)

Francesco, is it a Stream or a ValueStream? Are your properties logged? Is the user you are using an admin level user?

it's a simple stream, the properties are indeed logged and the user is admin yes

Edit: I tested the valuesStream as well, and I can say that I have the exact same issue as with simple Streams (no data saved in the valuestream, nothing in the DB).

ttielebein
14-Alexandrite
(To:fpassiatore)

How are you writing to the stream? Value Streams work with the "Logged" option on properties. To update streams, there must be some service somewhere doing this in Javascript. Can you provide that?

Thanks!

Tori

Hello Tori,

My thing's properties have the options "logged" and "persistent" enabled

Here is the setup:

I have something like 10 things corresponding to LoRa devices.

I have a "callback interface" that is used by those LoRa devices to update their own properties.

On the callback interface I have a service "UpdateData" that will, thanks to a (unique) deviceID received as a parameter, update the properties of the related thing and store those new properties in a stream.

The devices call that "UpdateData" service every 15 minutes.

Here is the script of the service :

// define the log prefix

var logPrefix = me.name +"::LoRa.CallbackDataSet: ";

//##################-- calling OpenWeather webservice --##################

var OWparams = {

    url: "http://api.openweathermap.org/data/2.5/weather?lat="+Latitude+"&lon="+Longitude+"&units=metric&appid=**************************************" /* STRING */

};

// result: JSON

var OWresult = Resources["ContentLoaderFunctions"].GetJSON(OWparams);

if (OWresult != undefined)

    var externalTemp = OWresult.main.temp;

else {

    var externalTemp = 0;

    logger.error(logPrefix + " Failed to retrieve the external temperature from OpenWeather.");

}

//---------------- end of calling of OpenWeather webservice ----------------

//#############-- try to find a thing that has the given deviceid --#############

var thingName = me.getLoRaDeviceFromDeviceID({deviceId: DeviceID});

if(thingName == "") {

    logger.error(logPrefix + " Failed to find  thing provisioned for sigfoxId: " + DeviceID);

    throw "Failed to find thing provisioned for sigfoxId: " + DeviceID;

}

//-------------------------------------------------------------------------------

var dateNow = new Date();

//####################-- Setting params --####################

// location:LOCATION

var myLocation = new Object();

myLocation.latitude = Latitude;

myLocation.longitude = Longitude;

myLocation.elevation = Altitude;

myLocation.units ="WGS84";

Things[thingName].Temperature = Temperature;

Things[thingName].ExtTemperature = externalTemp;

Things[thingName].BatteryLifetime = BatteryLevel;

Things[thingName].FridgeHumidity = Humidity;

Things[thingName].FridgeLocation = myLocation;

Things[thingName].RSSI = RSSI;

Things[thingName].SNR = SNR;

Things[thingName].SpreadFactor = SpreadFactor;

Things[thingName].LastCommunication = dateNow;

//-----------------------------------------------------------

//####################-- Saving fridge value into stream --####################

var values = Things["SmartFridgeDataStream"].CreateValues();

values.RSSI = RSSI; //NUMBER

values.Temperature = Temperature; //NUMBER

values.ExtTemperature = externalTemp; //NUMBER

values.SNR = SNR; //NUMBER

values.SpreadFactor = SpreadFactor; //NUMBER

values.FridgeHumidity = Humidity; //NUMBER

values.BatteryLifetime = BatteryLevel; //NUMBER

values.FridgeLocation = myLocation; //LOCATION

var params = {

    values: values /* INFOTABLE*/,

    location: myLocation /* LOCATION */,

    source: thingName /* STRING */,

    timestamp: dateNow /* DATETIME */

};

// no return

Things["SmartFridgeDataStream"].AddStreamEntry(params);

//-----------------------------------------------------------

Normally I'm supposed to receive the parameters through HTTP POST request (that part is working fine I could test it by logging the received parameters.

But simply by clicking the "test" button of that update data service, I can clearly see that the script is successfully executed and that the properties of my things are updated. But nothing is added in the stream, and nothing appears in the logs.

Another important detail to know, We have another Thingworx platform setup (default installation with H2 default DB) and everything is working fine on that one !
It is on our new production setup, with PostgreSQL that it is not working (everything in HA).

ttielebein
14-Alexandrite
(To:fpassiatore)

Can you tell me more about these properties? Are they remotely bound? Are they located on a thingshape? What version of ThingWorx?

Well they are simple properties, only numbers and strings. and no there is no binding, I just assign the values though the services.
The properties are all defines on 2 different shapes, and there is one template implementing the 2 shapes. The the things implement the template.

But I just did a test and i'm now 99% sure that the issue is coming from the PostGreSQL setup (maybe HA is related?)

I just exported my whole project and imported it on a default thingworx installation (with H2 DB), and without any modifications the streams are working fine !
So it's definitely not a thing/service issue. I'm nearly sure it's coming from a communication issue between PostGreSQL and Thingworx, or something is not configured properly on that side...

ttielebein
14-Alexandrite
(To:fpassiatore)

Aanjan Ravi​ what do you make of this last comment?

Aanjan
12-Amethyst
(To:fpassiatore)

Francesco, how are the permissions setup on the server? Can you open/ access the 'ThingworxPersistenceProvider' present in the Persistence Providers section?

Hello Aanjan, where exactly do you want me to check the permissions on the server?

I can open the ThingworxPersistentProdiver but in readonly, I cannot edit it. ( It is tagged as a system object)

Note that I can create things, templates, shapes, media, datashape, mashups, etc..

Any other object than streams data and value streams DATA (that i tried at least, i've not tried with data table) are properly saved in the database. It's not working only for stream data/value stream data.

Aanjan
12-Amethyst
(To:fpassiatore)

Ah, you should be able to edit it (atleast the configuration section). Streams, Value Streams, Data Tables, Blogs and Wiki data are stored/ maintained through the PersistenceProvider. Can you try giving your user Design and Runtime permissions on the PersitenceProvider?

Well, my user is part of the default administrator group (and even if I use the default administrator user it's the same) and when I open the ThingworxPersistentProvider I have this beautiful message (which makes senses):

Capture.PNG

I don't know if it's normal but I'm not able to edit the configuration of the ThingworxPersistentProvider, even with design and runtimes rights.

Aanjan
12-Amethyst
(To:fpassiatore)

What version of ThingWorx is this? You should be able to access the Provider if you're an admin user (as that's where you would change the Stream and Value Stream processor settings). Is your connection going through a Connection Server by any chance?

Well, what you say make sense but it's not the case here

ThingWorx 7.1.3-b14

and for the rest:

     PostgreSQL 9.4.8

     pgpool-II version 3.5.2 (ekieboshi)

Yes I am indeed passing by a connection server


Aanjan
12-Amethyst
(To:fpassiatore)

Instead of going through the Connection Server, can you try hitting your platform directly and check if you can access the Provider then?

I connected remotely on the server itself and opened thingworx through the localhost address and I still have read-only access even with the default admin user.

I got some news today from the engineer who installed the platform.
He have trace of the requests in the logs of PostGreSQL, but still, even if the request appears, nothing is stored and there is no errors.

More than ever I think something is wrong on the PostGreSQL side.

Here are the logs (test done with all values to 1):

2016-08-30 16:51:29 CEST LOG:  execute pgpool21452/pgpool21452: SELECT current_setting('transaction_isolation')

2016-08-30 16:51:29 CEST LOG:  execute <unnamed>: select * from upsert_stream_entry($1,$2,$3,$4,$5,$6,$7) as result

2016-08-30 16:51:29 CEST DETAIL:  parameters: $1 = 'SmartFridgeDataStream', $2 = 'fakefridge', $3 = '2016-08-30 16:52:55.611', $4 = '{"SNR": 1.0, "RSSI": 1.0, "Temperature": 1.0, "SpreadFactor": 1.0, "ExtTemperature": 25.26, "FridgeHumidity": 1.0, "FridgeLocation": {"latitude": 1.0, "elevation": 1.0, "longitude": 1.0}, "BatteryLifetime": 1.0, "isGPSLocationFresh": true}', $5 = '1.0,1.0,1.0', $6 = 'Thing', $7 = '[]'


I don't know if this helps but... we'll see

Aanjan
12-Amethyst
(To:fpassiatore)

Do you have an export of all your entities and data? Would it be possible for your to import this into a different system and check there?

Yes, I exported everything (except for the data since we could not save anything anyway..)

I imported the whole project on another server (default H2 installation) and everything works perfectly fine, including streams.

Also, note that this is our actual structure (on top of Pgpool we have Thingworx of course)

When we bypass the pgpool and connect one of the PostGreSQL database directly to thingworx, the data stream are saved properly!

But when we pass through the Pgpool it is not working anymore.

I feel we are getting close but we still don't understand what's wrong with Pgpool... any idea?

Problem solved by downgrading the version of pgpool and postgresql...

Thanks for you help guys

Aanjan
12-Amethyst
(To:fpassiatore)

Ah thanks for the update! I must've overlooked that, my apologies. Postgres 9.4.5 is the recommended version.

Announcements


Top Tags