Community Tip - Learn all about PTC Community Badges. Engage with PTC and see how many you can earn! X
Hello, I am having some issues with Anomaly Alert after Training
Here is everything I've done:
results.destination.uri=results://52.169.91.141:8096/results/models file.storage.path=file:/ThingworxMicroservices/data/results dataset.webservices.url= dataset.app.id= dataset.app.key=
database: driverClass: org.h2.Driver url: jdbc:h2:file:./data/results/model-db # server config stuff server: applicationConnectors: - type: http port: 8096 reuseAddress: true adminConnectors: - type: http port: 8097 reuseAddress: true registerDefaultExceptionMappers: false
# server config stuff server: # softNofileLimit: 1000 # hardNofileLimit: 1000 applicationConnectors: - type: http port: 8090 adminConnectors: - type: http port: 8091 registerDefaultExceptionMappers: false
java -Dproperties.file=/etc/thingworx/microservices.properties -jar 'result-standalone-2.0.2-bin.jar' server results-h2-file.yml > logs/results.log & java -Dproperties.file=/etc/thingworx/microservices.properties -jar 'training-standalone-2.0.4-bin.jar' server training-standalone-single.yml > logs/training.log &
Returning a FAILED state for TimedValue [timestamp=1535104090265, value=1011.0362249999962]. ThingWatcherMessage [timestamp=2018-08-24T11:48:10.767, severity=ERROR, state=ThingWatcherInternalState [internal=TRAINING, external=TRAINING], messageCode=WAT9999E, messageText=Unexpected exception. {Throwable=[RetryableTrainingRequestException: Received error from POST of Training request with Code(500)_Message(Internal Server Error)_Details:_{code=500, message=Failed during training, details=Caused by: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: org.h2.jdbc.JdbcSQLException: Unique index or primary key violation: "CONSTRAINT_INDEX_C ON PUBLIC.JOB_EXEC_INFO(CLAIM_CHECK) VALUES ('1', 1)"; SQL statement:_/* JobExecutionInfoDAO.add */ insert into job_exec_info (claim_check, created, request) values (?, ?, ?) [23505-192] [statement:"insert into job_exec_info (claim_check, created, request) values (:claimCheck, :created, :request)", located:"insert into job_exec_info (claim_check, created, request) values
Anomaly Alert [AnomalyAlertTest_Thing-propTest1-AnomalyAlertTest] Failed when in [TRAINING] State. Thingwatcher Error Code :: [WAT9999E] Message :: Unexpected exception. {Throwable=[RetryableTrainingRequestException: Received error from POST of Training request with Code(500)_Message(Internal Server Error)_Details:_{code=500, message=Failed during training, details=Caused by: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: org.h2.jdbc.JdbcSQLException: Unique index or primary key violation: "CONSTRAINT_INDEX_C ON PUBLIC.JOB_EXEC_INFO(CLAIM_CHECK) VALUES ('1', 1)"; SQL statement:_/* JobExecutionInfoDAO.add */ insert into job_exec_info (claim_check, created, request) values (?, ?, ?) [23505-192] [statement:"insert into job_exec_info (claim_check, created, request) values (:claimCheck, :created, :request)", located:"insert into job_exec_info (claim_check, created, request) values (:claimCheck, :created, :request)", rewritten:"/* JobExecutionInfoDAO.add */ insert into job_exec_info (claim_check, created, request) values (?, ?, ?)", arguments:{ positional:{}, named:{request:'{"goalName":"Goal","dataSource":{"datasetUri":"body:///","format":"json","exclusions":[],"data":"[{\"EntityId\":\"EntityId1\",\"Goal\":84.4649635544983,\"Time\":1535104002582},{\"EntityId\":\"EntityId1\",\"Goal\":66.53744259471634,\"Time\":1535104002735},{\"EntityId\":\"EntityId1\",\"Goal\":52.42285620347115,\"Time\":1535104002888},{\"EntityId\":\"EntityId1\",\"Goal\":89.6188606133255,\"Time\":1535104003041},{\"EntityId\":\"EntityId1\",\"Goal\":85.13964386168999,\"Time\":1535104003194},{\"EntityId\":\"EntityId1\",\"Goal\":67.83531276896812,\"Time\":1535104003347},{\"EntityId\":\"EntityId1\",\"Goal\":61.62003522072911,\"Time\":1535104003500},
Solved! Go to Solution.
For the record, this has been resolved offline, by clearing the result db file.
See https://www.ptc.com/en/support/article?n=CS292986 for details.
Hi
This error is reported at https://www.ptc.com/en/support/article?n=CS270206.
Could you apply the resolution provided and check if it helps here.
Thanks
Kind regards
Christophe
I tried that solution but it didn't worked for me.
Still the same error
Hi
Could you upload the complete yml files for result and training ?
Could you also specify which version of ThingWorx you are using ?
It could also be useful to get the training and result output in a file and upload it - the following article should help for this https://www.ptc.com/en/support/article?n=CS266118
Kind regards
Christophe
appCtx: - classpath:/spring/aws-s3nfs.xml - classpath:/spring/service.xml propertyReaderCtx: - classpath:/spring/properties-wiring.xml database: driverClass: org.h2.Driver url: jdbc:h2:file:./data/results/model-db # server config stuff server: applicationConnectors: - type: http port: 8096 reuseAddress: true adminConnectors: - type: http port: 8097 reuseAddress: true registerDefaultExceptionMappers: false logging: # The default level of all loggers. Can be OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, or ALL. level: INFO # Logger-specific levels. loggers: com.thingworx.analytics.model: INFO appenders: # write log statements to console (stdout). - type: console # TimeZone ID as recognized by Java e.g. UTC, EST5EDT timeZone: EST5EDT # Do not display log statements below this threshold to stdout. threshold: ALL target: stdout flyway: # The schemas managed by Flyway. (default: default schema of the connection) schemas: ["PUBLIC"] # Locations to scan recursively for migrations. (default: db/migration) locations: - /migration # Allows migrations to be run "out of order". If you already have versions 1 and 3 applied, and now a version 2 is found, it will be applied too instead of being ignored. (default: false) outOfOrder: false # The description to tag an existing schema with when executing baseline. (default: << Flyway Baseline >>) baselineDescription: "<< MMS Flyway Baseline >>" # Whether to automatically call baseline when migrate is executed against a non-empty schema with no metadata table. (default: false) # Be careful when enabling this as it removes the safety net that ensures Flyway does not migrate the wrong database in case of a configuration mistake! baselineOnMigrate: true # Whether to automatically call validate or not when running migrate. (default: true) validateOnMigrate: true swagger: resourcePackage: com.thingworx.analytics.results
# stuff for connection to neuron appCtx: - classpath:/spring/job-executor-asynchronous-memory-results.xml - classpath:/spring/service.xml - classpath:/spring/defaultmodelsource.xml - classpath:/spring/modelHealthCheck.xml - classpath:/spring/datasource.xml propertyReaderCtx: - classpath:/spring/properties-wiring.xml # server config stuff server: # softNofileLimit: 1000 # hardNofileLimit: 1000 applicationConnectors: - type: http port: 8090 adminConnectors: - type: http port: 8091 registerDefaultExceptionMappers: false logging: # The default level of all loggers. Can be OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, or ALL. level: INFO # Logger-specific levels. loggers: org.apache.spark: WARN appenders: # write log statements to console (stdout). - type: console # TimeZone ID as recognized by Java e.g. UTC, EST5EDT timeZone: EST5EDT # Do not display log statements below this threshold to stdout. threshold: ALL target: stdout database: driverClass: org.h2.Driver url: jdbc:h2:mem:db.training flyway: # Locations to scan recursively for migrations. (default: db/migration) locations: - /migration # The description to tag an existing schema with when executing baseline. (default: << Flyway Baseline >>) baselineDescription: "<< Training Microservice Flyway Baseline >>" # Whether to automatically call baseline when migrate is executed against a non-empty schema with no metadata table. (default: false) # Be careful when enabling this as it removes the safety net that ensures Flyway does not migrate the wrong database in case of a configuration mistake! baselineOnMigrate: true swagger: resourcePackage: com.thingworx.analytics.training
I tried to follow the article to get the output but I get an error when I try to run them
Hi
Can you try to:
- stop the training and results microservices
- delete all the db files under ./data/results and also any training db that you may have from previous run.
- start the training and results microservices and try again.
Note that you are using a very old version of those services, so should this not help, it probably will make sense to use and 8.2 or 8.3 version.
Thank you
Kind regards
Christophe
I'm using ThingWorx-Analytics-ThingWatcher-8-0-1 because from the PTC software download it's the Most Recent Datecode. Could you tell me how can I get 8.2 or 8.3 version?
Thank you
Hi
All the versions are available from the PTC Software Download page https://support.ptc.com/appserver/auth/it/esd/product.jsp?prodFamily=NRN .
If you do not see newer version, I would advise you open a Licensing case so they can check what is going on.
for this, access the Case Logger, enter ThingWorx Analytics for product, enter the description stating you cannot see newer version of ThingWatcher.
In the second steps, select Licensing for the field Which Specific Area Does Your Issue Relate to?
Hope this helps
Kind regards
Christophe
I cannot set Thingworx Analytics as product because I get this
Product ThingWorx Analytics is not supported under SCN 5A1978605. Please enter correct SCN or Product
Hi
Thanks to the SCN I am able to identify you, so I have opened a case on your behalf to the Licensing team who is in charge of what is or not activated on the Support Download page. You should be able to follow the case at https://support.ptc.com/appserver/cs/track/casetracker.jsp. Let me know if you do not see it.
However based on the error you get for product ThingWorx analytics, the behaviour of the software download page seems in line.
Having said that my colleagues from Licensing team will be able to give you details on what is happening.
Thank you
Kind rgeards
Christophe
Thank you very much. I actually cannot see the case from the link you sent me. Could you please send me the case number?
Hi
This is case C14536680
Kind regards
Christophe
That's fine. I'll let you know.
Thank you very much
Hello
I am now using 8.3 version but I am still having the same issues.
I tried to delete files under results and training db but nothing changed. I'm still having failure
Error
Anomaly Alert [AnomalyAlertTest_Thing-propTest1-AnomalyAlertTest] Failed when in [TRAINING] State. Thingwatcher Error Code :: [WAT1001E] Message :: Operation exception. {Throwable=[Trainer.TrainingJobErrorException: Training job with id 2 entered into an incomplete state [FAILED] unable to process models. Error message: [UnsupportedOperationException: empty collection]}]
Hi
thank you for this.
Just to note this is not the same issue as before, we do still have a failure but the error is very different now.
Could you make sure the property monitored is logged.
Also could you use the attached yml files, they contains statement to store output in log file.
Make sure to update the port number in those yml file.
Then restart the training and result microservices (with those yml) and reproduce the error.
Could you then please upload:
- the trainingms.log and resultsms.log file
- ThingWorx logs folder
Thank you
Kind regards
Christophe
Fine, may I just ask you one thing before I'll give you the informations you asked?
I saw all manuals talking about Remote Things but I'm using either a Generic Thing or a Timer Thing. Is that a issue?
Is RemoteThing the only way to make Anomaly Alert work or it can work also with other kind of entities?
Thank you
Hi
The usual case would indeed be to have a remote thing, but I do have the anomaly working with a generic thing too, so I would think it should work for you too.
Kind regards
Christophe
Ok, here you are.
trainingms.log
[...] INFO [2018-09-06 11:16:10,846] org.eclipse.jetty.setuid.SetUIDListener: Opened application@672a7893{HTTP/1.1,[http/1.1]}{0.0.0.0:8090} INFO [2018-09-06 11:16:11,465] org.eclipse.jetty.server.handler.ContextHandler: Started i.d.j.MutableServletContextHandler@7e4d2615{/,null,AVAILABLE} INFO [2018-09-06 11:16:11,470] io.dropwizard.setup.AdminEnvironment: tasks = POST /tasks/log-level (io.dropwizard.servlets.tasks.LogConfigurationTask) POST /tasks/gc (io.dropwizard.servlets.tasks.GarbageCollectionTask) INFO [2018-09-06 11:16:11,475] org.eclipse.jetty.server.handler.ContextHandler: Started i.d.j.MutableServletContextHandler@5a905a16{/,null,AVAILABLE} INFO [2018-09-06 11:16:11,482] org.eclipse.jetty.server.AbstractConnector: Started application@672a7893{HTTP/1.1,[http/1.1]}{0.0.0.0:8090} INFO [2018-09-06 11:16:11,489] org.eclipse.jetty.server.AbstractConnector: Started admin@2e18c730{HTTP/1.1,[http/1.1]}{0.0.0.0:8091} INFO [2018-09-06 11:16:11,490] org.eclipse.jetty.server.Server: Started @11587ms
resultsms.log
[...] INFO [2018-09-06 11:16:00,069] org.eclipse.jetty.server.handler.ContextHandler: Started i.d.j.MutableServletContextHandler@29174934{/,null,AVAILABLE} INFO [2018-09-06 11:16:00,075] org.eclipse.jetty.server.AbstractConnector: Started application@1ef8f1ca{HTTP/1.1,[http/1.1]}{0.0.0.0:8096} INFO [2018-09-06 11:16:00,076] org.eclipse.jetty.server.AbstractConnector: Started admin@2066f0d3{HTTP/1.1,[http/1.1]}{0.0.0.0:8097} INFO [2018-09-06 11:16:00,076] org.eclipse.jetty.server.Server: Started @7254ms INFO [2018-09-06 11:16:20,123] com.thingworx.analytics.results.resources.ModelResource: List of models with name [null], sortBy [], limit null ,tag [] INFO [2018-09-06 11:16:20,214] org.hibernate.engine.internal.StatisticalLoggingSessionEventListener: Session Metrics { 82101 nanoseconds spent acquiring 1 JDBC connections; 0 nanoseconds spent releasing 0 JDBC connections; 947007 nanoseconds spent preparing 2 JDBC statements; 574904 nanoseconds spent executing 2 JDBC statements; 0 nanoseconds spent executing 0 JDBC batches; 0 nanoseconds spent performing 0 L2C puts; 0 nanoseconds spent performing 0 L2C hits; 0 nanoseconds spent performing 0 L2C misses; 0 nanoseconds spent executing 0 flushes (flushing a total of 0 entities and 0 collections); 142301 nanoseconds spent executing 2 partial-flushes (flushing a total of 0 entities and 0 collections) } INFO [2018-09-06 11:16:39,813] com.thingworx.analytics.results.resources.ModelResource: List of models with name [null], sortBy [], limit null ,tag [] INFO [2018-09-06 11:16:39,817] org.hibernate.engine.internal.StatisticalLoggingSessionEventListener: Session Metrics { 46200 nanoseconds spent acquiring 1 JDBC connections; 0 nanoseconds spent releasing 0 JDBC connections; 147201 nanoseconds spent preparing 2 JDBC statements; 263202 nanoseconds spent executing 2 JDBC statements; 0 nanoseconds spent executing 0 JDBC batches; 0 nanoseconds spent performing 0 L2C puts; 0 nanoseconds spent performing 0 L2C hits; 0 nanoseconds spent performing 0 L2C misses; 0 nanoseconds spent executing 0 flushes (flushing a total of 0 entities and 0 collections); 20100 nanoseconds spent executing 2 partial-flushes (flushing a total of 0 entities and 0 collections) } INFO [2018-09-06 11:18:30,559] com.thingworx.analytics.results.resources.ModelResource: List of models with name [null], sortBy [], limit null ,tag [] INFO [2018-09-06 11:18:30,562] org.hibernate.engine.internal.StatisticalLoggingSessionEventListener: Session Metrics { 40301 nanoseconds spent acquiring 1 JDBC connections; 0 nanoseconds spent releasing 0 JDBC connections; 66402 nanoseconds spent preparing 2 JDBC statements; 119601 nanoseconds spent executing 2 JDBC statements; 0 nanoseconds spent executing 0 JDBC batches; 0 nanoseconds spent performing 0 L2C puts; 0 nanoseconds spent performing 0 L2C hits; 0 nanoseconds spent performing 0 L2C misses; 0 nanoseconds spent executing 0 flushes (flushing a total of 0 entities and 0 collections); 63601 nanoseconds spent executing 2 partial-flushes (flushing a total of 0 entities and 0 collections) }
Thingworx log
Returning a FAILED state for TimedValue [timestamp=1536247208958, value=29.934394445547596]. ThingWatcherMessage [timestamp=2018-09-06T17:20:08.959, severity=ERROR, state=ThingWatcherInternalState [internal=COLLECTING_TRAINING_DATA, external=CALIBRATING], messageCode=WAT9999E, messageText=Unexpected exception. {Throwable=[ArithmeticException: / by zero}]]_java.lang.ArithmeticException: / by zero_ at com.coldlight.cybertron.timeseries.utils.TimestampUtil.floorTimestamp(TimestampUtil.java:29)_ at com.coldlight.cybertron.timeseries.utils.TimestampUtil.calculateWeightedValues(TimestampUtil.java:50)_ at com.coldlight.cybertron.timeseries.streaming.OrderedWeightedValueSampler.offer(OrderedWeightedValueSampler.java:81)_ at com.coldlight.cybertron.timeseries.streaming.OrderedWeightedValueSampler.offer(OrderedWeightedValueSampler.java:75)_ at com.thingworx.analytics.thingwatcher.ThingWatcherInternal.lambda$runStateMachineFor$7(ThingWatcherInternal.java:393)_ at java.util.ArrayList.forEach(ArrayList.java:1249)_ at com.thingworx.analytics.thingwatcher.ThingWatcherInternal.runStateMachineFor(ThingWatcherInternal.java:393)_ at com.thingworx.analytics.thingwatcher.ThingWatcherInternal.bareMonitor(ThingWatcherInternal.java:302)_ at com.thingworx.analytics.thingwatcher.ThingWatcherInternal.monitor(ThingWatcherInternal.java:229)_ at com.thingworx.analytics.thingwatcher.ThingWatcher.monitor(ThingWatcher.java:57)_ at com.thingworx.system.subsystems.alerts.anomalyalert.AnomalyMonitor.monitor(AnomalyMonitor.java:256)_ at com.thingworx.system.subsystems.alerts.data.Alert.doAnomalyEvaluation(Alert.java:565)_ at com.thingworx.system.subsystems.alerts.handlers.NumberAlert.evaluateAlarm(NumberAlert.java:138)_ at com.thingworx.system.subsystems.alerts.data.AlertList.checkAlerts(AlertList.java:87)_ at com.thingworx.things.Thing.rawSetPropertyVTQ(Thing.java:4093)_ at com.thingworx.things.Thing.rawSetPropertyVTQ(Thing.java:3913)_ at com.thingworx.things.Thing.handleSetPropertyWithUpdate(Thing.java:4155)_ at com.thingworx.things.Thing.internalSetPropertyVTQ(Thing.java:4148)_ at com.thingworx.things.Thing.setPropertyVTQ(Thing.java:4270)_ at com.thingworx.things.Thing.setPropertyValue(Thing.java:4197)_ at com.thingworx.dsl.engine.adapters.ThingworxEntityAdapter.put(ThingworxEntityAdapter.java:296)_ at org.mozilla.javascript.ScriptableObject.putProperty(ScriptableObject.java:2347)_ at org.mozilla.javascript.ScriptRuntime.setObjectProp(ScriptRuntime.java:1598)_ at org.mozilla.javascript.ScriptRuntime.setObjectProp(ScriptRuntime.java:1592)_ at org.mozilla.javascript.gen.Event_Timer_1214._c_script_0(Event.Timer:14)_ at org.mozilla.javascript.gen.Event_Timer_1214.call(Event.Timer)_ at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)_ at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3090)_ at org.mozilla.javascript.gen.Event_Timer_1214.call(Event.Timer)_ at org.mozilla.javascript.gen.Event_Timer_1214.exec(Event.Timer)_ at com.thingworx.dsl.engine.DSLProcessor.executeService(DSLProcessor.java:171)_ at com.thingworx.dsl.DSLServiceHandler.processService(DSLServiceHandler.java:38)_ at com.thingworx.handlers.ServiceHandlerBase.processServiceWithMetrics(ServiceHandlerBase.java:48)_ at com.thingworx.things.Thing.handleEvent(Thing.java:7645)_ at com.thingworx.things.Thing.processEvent(Thing.java:7581)_ at com.thingworx.system.subsystems.eventprocessing.EventInstance.run(EventInstance.java:61)_ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)_ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)_ at java.lang.Thread.run(Thread.java:748)_
Anomaly Alert [AnomalyAlertTest_Thing-value-anomalia] Failed when in [CALIBRATING] State. Thingwatcher Error Code :: [WAT9999E] Message :: Unexpected exception. {Throwable=[ArithmeticException: / by zero}]
Do you need anything else?
Hi
It would look like something wrong with the incoming data.
You previously indicated using a Thing with a timer.
If this is still the case during the above error, is it possible for you to export all the required entities and attach them here so I can test it.
I would also need to know what exact version of ThingWorx and training and result jar (just give the name of the jar) you are using.
To attach file, select the Choose Files button at the bottom of the text box when replying.
Thanks
Christophe
Yes, the thing is this:
I'm using a Timer thing because I'm creating a simulator for data so, every 2 seconds, I have a subscription which updates my data around a central value and, sometimes, generates a pick.
So it is possible my simulation of data is not proper; in case it would be useful to know a good way to generate a set of random data in order to study how anomaly alert works.
Anyway, in order to give you the information you want, I attach here the thing; plus you will also see a stream with those data and a mashup where you can see the plot of data itself.
My Thingworx version is ThingWorx 8.3.1
my Jars are: result-standalone-2.0.3-bin, training-standalone-2.0.5-bin
Thank you
Hi
I seem to be missing AnomalyAlert_DataShape and AnomalyAlertTest_ValueStream to be able to import the entities.
Could you provide them too ?
Thank you
Christophe
You're right, sorry
I guess you have all now
Thanks
I seem to still be missing some entities, Stream_Analysis_ThingShape
Thank you, I am able to import and run everything.
Interestingly it work fine on my side in a ThingWorx 8.3.1.
There might be some local settings on your machine.
Here are a few ideas:
1) Is it possible for you to check the Regional Settings and make sure you run i English , then restart Tomcat and the training and results microservices.
2) Could you post the property of the anomaly alert to confirm the settings for Outbound Anomaly Rate and other values.
3) If you still have failure, could you set the Application log to INFO and reproduce the error, then upload the complete ApplicationLog.log and training and result log files. It is best to upload the file rather than pasting in the body.
For testing purpose you can maybe change the timer updateRate so Calibrating will be quicker.
Thank you
Christophe
LANG=en_US.UTF-8 LANGUAGE= LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=
Thank you
Fabio
For the record, this has been resolved offline, by clearing the result db file.
See https://www.ptc.com/en/support/article?n=CS292986 for details.