Thingworx Analytics: Send Data From Dataconnect To Analytics Server

[mdeshpande-2]

Hello,

I have just started with thingworx analytics 52.0 and DataConnect 52.0. I have some issues in sending data from dataconnect to analytics server.

I have followed these steps:

1] Set up Analytics Server

2] Import Analytics extension

3] Set up Dataconnect

4] Configure Platform Subsystem with dataconnect params

5] Configure MachineLearningAll thing with Analytics server params

6] Create a Data Analysis definition in TW and execute it

7] The zip file with property data and payload.json are being created in thingworx repository

8] Sending the zip(payload.json+csv) file to analytics server?

Step 8 is where I am facing problems. How to upload the created zip file to the analytics server? Do I have to use the service SubmitUploadDataset?

If so, then I am not able to test the service in Composer. When I Edit the UploadData field and try to add a row it tells me "Must have a datashape defined to add rows".

Am I approaching it the wrong way? Do I need to write a custom service to send the data?

I am having problems with understanding how exactly data from dataconnect is transfered to analytics server. I would appreciate any help on understanding this better and solving the issue.

Thanks & Regards,

Madhumita

[cmorfin]

Hi Madhumita

When all is configured ok, the step 8 is done automatically.

The flow is:

1) DAD collect logged data from ThingWorx

2) DAD send the data (as skinny csv file + payload.json) to DataConnect

3) DataConnect transform the data and create one single resulting csv file

4) DataConnect create a dataset inside ThingWorx Analytics Server with the resulting csv file loaded as data. The dataset created has got the same name as the DAD.

If you do not have a dataset created in ThingWorx Analytics, then you probably have a problem in the configuration or in the way data is collected by the DAD.

You should start by reviewing the DataConnect log files under /opt/dataconnect/logs.

Note that I am in the process of writing a blog on this specific topic. I am hoping to have it published by next week.

If you can wait, I think it would be a good reading to understand how things work.

If you need further help I would probably recommend to open a case to Technical Support, because that is a topic that may need more data to progress.

Kind regards

Christophe

[mdeshpande-2]

Hi Christophe,

Thanks for the help.I will be opening a case for the same.Also looking forward to the blog.

Thanks & Regards,

Madhumita

[cmorfin]

Hi Madhumita

Just to let you know, I have published the blog at :

How to use DataConnect when sending property values from the ThingWorx platform to ThingWorx Analytics.

I hope you'll find it of interest

Kind regards

Christophe

[mdeshpande-2]

Hi Christophe,

I went through the blog. It was really helpful. I followed your configuration steps and was able to establish connection between DataConnect and Analytics Server.

However I am still facing an issue with data transfer from dataconnect to Analytics server. I checked the logs and I found that the error is "File Not Found".

The transformer job is not able to find the zip file in system repository. But the file is present there.I haven't been able to resolve this issue.

I would appreciate any help regarding the same. I have attached the dataconnect logs for reference.

Now I am using Thingworx Analytics and DataConnect 52.2.

Thanks & Regards,

Madhumita

[cmorfin · Accepted solution]

Hi Madhumita

I think you missed one step during the installation of DataConnect - see the DataConnect User's guide, chapter "Integrate ThingWorx and DataConnect"

you need to create a symbolic link between /repository and /ThingworxStorage/repository: ln -s /ThingworxStorage/repository /repository .

Hope this helps

Kind regards

Christophe

[mdeshpande-2]

Hi Christophe,

Thanks for the quick reply.The symlink wasn't present in the required directory. I had created it in some other directory. Now, its working okay.

Thanks & Regards,

Madhumita

[cmorfin]

Hi Madhumita

Great I am happy that it works for you.

If you don't mind it would be good if you could mark the answer as correct answer as this will help others in the Community to find what could be wrong.

Thank you

Kind regards

Christophe

[ldell'aversano]

Hi Madhumita Deshpande ,

I'm in the same your situation.

Dataconnect makes skinny csv files and payload.json and *.zip but I don't see dataset (Tractor_DAD) on Analytics Server.

In which position I should create the link? In the root position?

Thank's

Luigi Dell'Aversano

[cmorfin]

Hi Luigi

The link to be created is

ln -s /ThingworxStorage/repository /repository

if you have DataConnect and ThingWorx on the same CentOS box.

The above ln command has absolute path so it does not matter where you execute it from, the link will be created under /.

It might also be helpful if you can provide the DataConnect logs from /opt/dataconnect/logs as your issue might be different.

Hope this helps.

Kind regards

Christophe

[ldell'aversano]

Hi Christophe,

I've already created the link under "/" but it doesn't work yet.

Maybe there are some other errors, I've attached DataConnect logs that aren't empties, and DataConnect configuration file.

At the moment we've got:

1. A message "I'm Alive" from http://localhost:8080/1.0 url;
2. A message "{"implementationVersion":"52.2.0"}" from http://localhost:8080/1.0/about/versioninfo url;
3. Under /ThingworxStorage/repository/SystemRepository we have Tractor_DAD folder:
   1.   inside it we've Tractor_DAD1489746987677.zip and 1489746987677 folder with skinny csv and payload.json;

We 've on the same VMware Centos 7 station Analytics server and Thingworx composer and DataConnect (MED-61147-CD-520_F000_ThingWorx-Machine-Learning-52-0-dataconnect.rpm).

Where are the mistakes?

Thank you for support.

Luigi Dell'Aversano

[cmorfin]

Hi Luigi

Thank you for all the details, they are quite useful indeed.

The additional challenge that you have got here is that you have deployed ThingWorx Analytics and DataConnect on the same machine.

I will need some additional info to see if we can move forward with this setup.

Could you please send me:

-  the output of ls -l <Tomcat>/webapps  (possibly ls -l /usr/share/tomcat/webapps )

- a screenshot of SYSTEM > Subsystems > PlatformSubsystem: showing the ThingWorx DataConnect URL Path

- output of http://localhost:8080/twxml-connect/about/versioninfo

- output of http://localhost:8080/twxml-connect/1.0/about/versioninfo

Thank you

Kind regards

Christophe

[ldell'aversano]

Hi Christophe,

Attached there are the additional information requests.

In the output of http://localhost:8080/twxml-connect/about/versioninfo there is a response error like you will see in the attach.

Thank you so much!

Luigi Dell'Aversano

[cmorfin]

Hi Luigi

ok, so because you have deployed ThingWorx Analytics and DataConnect on the same machine, the URI to access DataConnect is not the default http://<host>:<port>/1.0 but http://<host>:<port>/twxml-connect/1.0 .

So you will need to update the  URI pointing to DataConnect with this information under SYSTEM > Subsystems > PlatformSubsystem:  ThingWorx DataConnect URL Path.

Also in dataconnect.conf, the property dataconnect.transfer.uri should point to a directory (that you can create somewhere on your disk), not the zip file as you have it now. See How to use DataConnect when sending property values from the ThingWorx platform to ThingWorx Analytics. for an example of a setting for this property. (under DataConnect configuration)

After those changes you will need to restart Tomcat.

Kind regards

Christophe

[ldell'aversano]

Hi Christophe,

I've changed configuration file and ThingWorx DataConnect URL Path.

Now dataconnect-rest.log file is populates too, I've attached it.

Inside it there is an error "

2017-03-20T11:55:36.720Z [ERROR] [http-bio-8080-exec-10] [com.coldlight.dataconnect.rest.exception.HttpExceptionResolver] - [[1596f8ed-4338-460d-8ce9-090add31090c] Unknown error: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection]

org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection" and "Wrong user name or password [28000-190]".

At which username and password it refers to? H2 password or APP KEY?

Thank you for the fast support.

Luigi Dell'Aversano

[cmorfin]

Hi Luigi

It is complaining abut the database access.

I did notice that you have got

dataconnect.database.username=

dataconnect.database.password=

I was assuming you remove the values for security purpose, but just want to make sure, you do need to have values here, the property should not be left without values. You can use any username and password, it will be created at runtime.

Can you please confirm you have got some values there ?

Regards

Christophe

[ldell'aversano]

Hi Christophe ... IT WORKS !!!

Thank you for support and very precious informations!

I had deleted username and password from database credentials, this was a(nother) mistake.

Furthermore, I've select a correct day-hour-minutes like below

and I've got on Thingworx Analytics Builder

Thank's a lot

Luigi Dell'Aversano

[cmorfin]

Hi Luigi

Brilliant, I am happy it works for you

Regards

Christophe

[pfu]

Hi,Christophe

I have a problem about that my service GetHistory cannot display In this area:

best,

fu

[cmorfin]

Hi Peng Fu

In case the below does not help, it would be best that you open a new thread with your specific issues as to not add complexity to an already long thread.

I would advise you check How to use DataConnect when sending property values from the ThingWorx platform to ThingWorx Analytics. and check the entities and theirs services to compare to yours.

For example if your GetHistory service has an InfoTable as output without a Datashape associated, then it won't be listed.

hope this helps

Regards

Christophe

[jzhao1]

Hi,Christophe Morfin!

I have the same problem,my Thingworx and Dataconnect are not on the same machine, I have created a shared file. But the runtime still reported an error.

I added an attachment.

We look forward to your reply, thank you!

Kind regards

Juanjuan

[cmorfin]

Hi JuanJuan

Could you please open a  new thread ?

It will help dissociate the different issues and help others to find answers in the future.

This thread is already very long and did cover several issues so it is difficult to follow.

If the information in it do not help you to fix your issue, then please open a new thread and we will take it from there.

Thank you

Kind regards

Christophe

[jzhao1]

ok