Anomaly Detection Calibration Fails

drieder · ‎May 28, 2018

Hello everyone,

while trying to follow this guide I had a lot of problems which I was able to solve thanks to this community, but now I am facing the next one:

While trying to create an anomaly alert, it fails after a long time of Calibration. As I suspect from the log files, it seems that the data is collected properly during Calibration but it fails to switch to training mode.

Also I checked the ports for the training and result standalones and the ones I used for them are free. I also get the expected results described in the guide when visting http://localhost:8082/results/models and http://localhost:8084/training.

Attached you can find the output of the command prompts where I executed the Jar Files. (One of them is very large, sorry for that)

Any help is appreciated very much!

Best Regards,

Dominik

Edit: Maybe it is the similar case like here?

mgoel · ‎May 28, 2018

@drieder

Thank you for opening this Thread here. Could you please confirm if you are using TWX version 8.0, or any other?

You can follow this article: https://www.ptc.com/en/support/article?n=CS271378&posno=2&q=calibrating%20fails&source=search

Regards,

Mohit

drieder · ‎May 28, 2018

Hello Mohit,

Thank you for your reply.

I am working with TWx Analytics 8.2. In my training-standalone-single.yml file I didn't see anything about "file storage". this is how my yml file looks like

Regards,

Dominik

BryanK · ‎May 29, 2018

Hi,

I think that you problem is related to windows permissions.

If you have a look a the results log file you will see this error.

Failed to upload results document: Could not create folder "file:///C:/Users/d.rieder/Desktop/ThingWorxProjekt/Analytics/AnomalyMicroservices-Windows/AnomalyMicroservices-Windows/Usersd.riederDesktopThingWorxProjektAnalyticsAnomalyMicroservices-WindowsAnomalyMicroservices-Windowsiles"

Try to run the jar files from an Admin command prompt or change the folder permissions.

drieder · ‎May 30, 2018

Hello,

thank you for your suggestions but that didn't work for me unfortunately. It is realy strange that the folder can't be created since the db and the training folders were created by the jars the first time I ran them.

Also I have no idea what else you have to take into account when executing these jars. The ports I setted are free and I configured the AlertProcessingSubsystem properly.

Regards,

Dominik

BryanK · ‎May 30, 2018

Hi Dominik,

I eventually gave up on trying to use the PTC windows installer (where it creates the services and runs as a windows service) and decided to use the standalone jars like you are using.

Essentially I had the same issues that you had.

If you add the following to the logging of the training.yml and the results.yml file there is a lot more info available for trouble shooting. **take note of the paths**

Training.yml

logging:
# The default level of all loggers. Can be OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, or ALL.
level: INFO
# Logger-specific levels.
loggers:
     org.apache.spark: WARN
logging:
# The default level of all loggers. Can be OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, or ALL.
level: ${logging.default.level}
appenders:
    # write log statements to console (stdout).
    - type: console
      # TimeZone ID as recognized by Java e.g. UTC, EST5EDT
      timeZone: EST5EDT
      # Do not display log statements below this threshold to stdout.
      threshold: ALL
      target: stdout
    - type: file
      # TimeZone ID as recognized by Java e.g. UTC, EST5EDT
      timeZone: EST5EDT
      # Do not display log statements below this threshold to stdout.
      threshold: ALL
      currentLogFilename: C:\PTC\ThingworxAnalytics\logs/training.log
      archive: true
      archivedLogFilenamePattern: C:\PTC\ThingworxAnalytics\logs/training-%d.log.gz
      archivedFileCount: 10

Results.yml

logging:
# The default level of all loggers. Can be OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, or ALL.
level: INFO
# Logger-specific levels.
loggers:
    com.thingworx.analytics.model: INFO
appenders:
    # write log statements to console (stdout).
    - type: console
      # TimeZone ID as recognized by Java e.g. UTC, EST5EDT
      timeZone: EST5EDT
      # Do not display log statements below this threshold to stdout.
      threshold: ALL
      target: stdout
    - type: file
      # TimeZone ID as recognized by Java e.g. UTC, EST5EDT
      timeZone: EST5EDT
      # Do not display log statements below this threshold to stdout.
      threshold: ALL
      currentLogFilename: C:\PTC\ThingworxAnalytics\logs/results.log
      archive: true
      archivedLogFilenamePattern: C:\PTC\ThingworxAnalytics\logs/results-%d.log.gz
      archivedFileCount: 10

I made changes to both files as I went and eventually I have mine working now.

I have attached my results, training and microservices files for you have a look. hopefully they will help.

Areas to look at are.

1. database

2. server

3. file.storage.path.

These are the commands that I use to launch the 2 jars.

java -Dproperties.file=C:\PTC\ThingworxAnalytics\microservices.properties -jar "C:\PTC\ThingworxAnalytics\result-standalone-2.0.3-bin.jar" server "C:\PTC\ThingworxAnalytics\results-h2-file.yml"
java -Dproperties.file=C:\PTC\ThingworxAnalytics\microservices.properties -jar "C:\PTC\ThingworxAnalytics\training-standalone-2.0.5-bin.jar" server "C:\PTC\ThingworxAnalytics\training-standalone-single.yml"

Please let me know if you need anything else.

****Edit*****

One more thing, After I sent this I restarted my VM. Tried to retrain and it failed again.

To fix this.

stop everything. (the microservices)

Delete the Databases (models-db.mv.db and training.mv.db)

Start everything

Retrain.

***Edit off ****

drieder · ‎Jun 04, 2018

Hello,

thank you for your detailed reply, I will try it out.

Regards,

Dominik

BryanK · ‎May 30, 2018

Hi,

For some reason my detailed response was marked as spam and deleted???

Attached are my training, results, and micro files.

Have a look at them and add in the logging as I think it may help a lot.

I had to delete my model-db.mv.db and training.mv.db (which are automatically created when you start up) as they somehow became corrupted.

Regards,

Bryan

cmorfin · ‎May 31, 2018

Hi

The error you have in training.log is the same as the one reported at https://www.ptc.com/en/support/article?n=CS270006.

The reason in that case is due to the syntax used to defined the file.storage.path property in microservice.properties file. The syntax does indeed depends on the version of the jar file you use.

The two syntax are described at https://developer.thingworx.com/resources/guides/anomaly-detection-how-guide/set-and-run-thingwatcher-micro-services under Configure File Paths.

Hope this helps

Kind regards

Christophe

BryanK · ‎May 31, 2018

Hi,

Thanks that's a useful article.

Would I need to open another topic to discuss getting "ThingWorx-Analytics-Server-Windows-Standalone" working?

It uses these files

analytics-server.properties
system-environment-variables.properties

It installs and starts but I'm getting the "same" issues described in this thread.

This is from the Training log file.

INFO [2018-05-31 18:16:52,613] org.eclipse.jetty.server.AbstractConnector: Started admin@64b3b1ce{HTTP/1.1,[http/1.1]}{0.0.0.0:9401}
INFO [2018-05-31 18:16:52,613] org.eclipse.jetty.server.Server: Started @25338ms
INFO [2018-05-31 18:27:40,433] io.dropwizard.jersey.jackson.JsonProcessingExceptionMapper: Mapping exception: class com.fasterxml.jackson.databind.JsonMappingException
! java.lang.IllegalArgumentException: The dataSource field "dataRef" cannot be null, empty or missing.
! at com.thingworx.analytics.ccc.data.ref.DatasetRefJsonDeserializer.deserialize(DatasetRefJsonDeserializer.java:142)
! at com.thingworx.analytics.ccc.data.ref.DatasetRefJsonDeserializer.deserialize(DatasetRefJsonDeserializer.java:29)
! at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)
! at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:101)
! at com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:156)
! ... 70 common frames omitted
! Causing: com.fasterxml.jackson.databind.JsonMappingException: The dataSource field "dataRef" cannot be null, empty or missing. (through reference chain: com.thingworx.analytics.training.params.TrainingRequest["dataSource"])
! at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:379)

cmorfin · ‎Jun 01, 2018

Hi @BryanK

yes please open a new thread as this seems a very different issue.

Thanks

Christophe