Skip to main content
S
skef1-Visitor
1-Visitor
July 30, 2018
Question

Process large datasets

  • July 30, 2018
  • 1 reply
  • 2592 views

I need to process large datasets with thingworx analytics. By using the API, I can create datasets with 1 million - 6 million rows. After I created the datasets, the analytics builder becomes awesome slow and is throwing the following error:

 

GetDatasetConfigurationAMS: ERROR: JavaException: java.util.concurrent.TimeoutException: Timed out APIRequestMessage [requestId: 633, endpointId: -1, sessionId: -1, method: POST, entityName: localAnalytics_DataThing, characteristic: Services, target: GetDatasetSchema]

 

Which would be the maximum dataset size thingworx analytics is capable to process?

1 reply

C
cmorfin19-Tanzanite
19-Tanzanite
July 30, 2018

Hi Skef

 

I don't have numbers for ThingWorx Analytics Builder but indeed for large dataset the Help Center does recommend to use a direct upload to the repository. See https://support.ptc.com/help/thingworx_hc/thingworx_analytics_8/#page/thingworx_analytics_8%2Fanalytics-data-upload-large.html 

 

Hope this helps

Kind regards

Christophe

 

S
skef1-VisitorAuthor
1-Visitor
July 31, 2018

The problem is not storing the data with thingworx analytics, its about processing it with the analytics builder. The described error message indicates, that an error occurred. But why? Maybe the dataset is to large? Is it possible to configure API timeouts? 

C
cmorfin19-Tanzanite
19-Tanzanite
July 31, 2018

Hi Skef

 

Apologies I thought you had the error when uploading the dataset.

So If I understand well you are able to uplaod a dataset of between 1 and 6 millions row, but you get this error when working with it.

If that is correct could you clarify when exactly do you get the error ?

What operation do you do to receive this error ?

If you repeat the same operation do you always have this error or does it work sometimes ?

How many rows and columns has your dataset ?

What datatype are they ?

 

Also regarding your deployment:

Which version of ThingWrox Analytics is it ?

Which version of OS are you using ?

Is it a native or docker deployment ?

How much RAM and processors does the server got ?

 

Thank you

Kind regards

Christophe

 

Terms & ConditionsAccessibility statement