cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

ROC and Confustion Matrix failed with wrong values

Highlighted
Level 10

ROC and Confustion Matrix failed with wrong values

Hi,

I'm working with Analytics 8.1 and get strange results with my data. To find the problem I try the "Analytics Builder Quickstart" tutorial with the vibrations data. Everything works find until the step "Generate and Enhance Model". The Model will be build but all ROC values and the Confusion Matrix are wrong.

The other stuff Profiles and Signales are ok.

26 REPLIES 26

Re: ROC and Confustion Matrix failed with wrong values

Hi David

Could you please clarify at what step of "Generate and Enhance Model you get this result.

Is it at the first model creation or after some of the revise steps ?

Also could you please attach the screenshot of:

- Help > About from Composer

- Settings > Verify Configuration from Builder

- Model Details in Builder form the view page of the model as seen in your screenshot

- Model Configuration  in Builder form the view page of the model as seen in your screenshot

We can then test here with the same version to check if we see the same thing.

Regards

Christophe

Re: ROC and Confustion Matrix failed with wrong values

Hi Christophe,

this result comes by creating the model, but also by enhance the model.

Help > About from Composer (Version: 8.1.1-b108)

Settings > Verify Configuration from Builder (Version: 8.1.040000)

Model Details

Model Configuration

So I hope this is helpful.

Re: ROC and Confustion Matrix failed with wrong values

Hi David

Thank you for this.

Regardign the Model Details screenshot, the one you posted is not theone I wanted. The one you posted is the Model Job details, but I am after the model details. you can find it by:

- selecting the model in the model list page

- select View

- select the Model Details button

Also could you please, send screenshots of your extensions : in Composer Import/Export > Manage

Thank you

Kind rgeards

Christophe

Re: ROC and Confustion Matrix failed with wrong values

So I hope this is the right one:

Composer Import/Export > Manage

Re: ROC and Confustion Matrix failed with wrong values

I'm getting the same results after building the model on Thingworx 8.1.0-b52 and Analytics 8.1.04 (ROC = 0.5 for false positives)

Anyone can clarify / explain such results?

Re: ROC and Confustion Matrix failed with wrong values

Hi Dmitry

This is odd because I do not reproduce this result either with ThingWorx 8.1.0 or 8.1.1 and Analytics 8.1.

Is it possible (or have you already tried) to create a new model (just give a new different name) with same dataset ? do you get the same 0.5 result with the new model ?

Also what happen if you upload the data again into a new dataset and create a new model ?

If you always get the 0.5 results, could you possibly upload the dataset (json + csv) you used ?

I understand this is the one from the QuickStart guide but since I am not reproducing the issue with it, maybe something happens at some point on that dataset , so I would rather have your json and csv in a zip file.

Also are you using your own ThingWorx and Analytics instances or are you using some trial hosted ones ?

If you are usign your own:

- what is the OS used for both ThingWorx and Analytics server ?

- what is the regional settings / locale environment of those servers ?

Thanks

Kind regards

Christophe

Re: ROC and Confustion Matrix failed with wrong values

I try this with so many times with different model names, with different datasets, with different goals (boolean types).

Here some of used datasets:

- AnalyticsTestStream.zip (goal: OverheatingError)

- analytics_vibration (test data of the current tutorial, goal: low_grease)


Both instances run on the same machine: Ubuntu 16.04.3 LTS

Regional settings: Deutsch (Deutschland) -> German (Germany)

Re: ROC and Confustion Matrix failed with wrong values

Hi Christophe Morfin

Thank you for the prompt response. I created a new dataset and a new model - got the same result. Please see the dataset (.csv and .json), as well as the video showcasing the process and the result, attached to this post.

I'm doing the tests on my development machine - it's RedHat 7.4 (both TW Foundation and Analytics are installed on the same PC). Regional settings and date / number formats are set to German (showcased in the video).

Re: ROC and Confustion Matrix failed with wrong values

Given Christophe Morfin​ 's interest in Regional Settings and seeing that both I and David Richter​ have German (which has "," (comma) instead of "." (point) as the decimal separator), I changed data format to English / UK, and was pretty sure that it was the cause. But re-uploading the dataset and recreating the model didn't help - still no luck.


PS I only did logout / login after changing the data format, which should be enough, but probably rebooting is still needed and would resolve the issue.