Training failed when monitoring remote thing from kepserver

[weli]

Hi,

I created a remotething in new composer by binding tag from kepservex,and added a anomly alert, then I go to mashup Anomaly_Dashboard to monitoring my newly created thing, but it always failed of training.

Here is error log:

2017-08-08 12:32:58.757+0800 [L: ERROR] [O: S.c.t.d.e.DSLProcessor] [I: ] [U: Administrator] [S: ] [T: http-nio-80-exec-12] Execution error in service script [AnomalyServices GetAnomalyAlertsByEntity] : TypeError: Cannot call method "GetPropertyDefinitions" of null

Error in invoking service Things_AnomalyServices1, GetAnomalyAlertAttributes, Things, AnomalyServices, Services, GetAnomalyAlertAttributes. Error from the server: "EOF"

Returning a FAILED state for TimedValue [timestamp=1502168642040, value=95.0]. ThingWatcherMessage [timestamp=2017-08-08T13:04:02.508, severity=ERROR, state=ThingWatcherInternalState [internal=OBTAINING_MODEL, external=TRAINING], messageCode=WAT1001E, messageText=Operation exception. {Throwable=[Trainer.TrainingJobErrorException: Training job with id 39 entered into an incomplete state [FAILED] unable to process models.  Error message: [ConnectException: Connection refused: connect]}]]

Anomaly Alert [Thing4Anomaly2-Simulation_Examples_Functions_Ramp1-amomalydemo2] Failed when in [TRAINING] State. Thingwatcher Error Code :: [WAT1001E]  Message :: Operation exception. {Throwable=[Trainer.TrainingJobErrorException: Training job with id 39 entered into an incomplete state [FAILED] unable to process models.  Error message: [ConnectException: Connection refused: connect]}]

Unable to persist entries because ForsetiClient[id=15785,tx=Transaction[eventId=26137,thread=1369:'ForkJoinPool-34-worker-12',status=STATUS_ACTIVE,resourceCount=1]] can't acquire ExclusiveLock{owner=ForsetiClient[id=15794,tx=Transaction[eventId=26145,thread=1444:'ForkJoinPool-34-worker-30',status=STATUS_ACTIVE,resourceCount=1]]} on NODE(5602), because holders of that lock are waiting for ForsetiClient[id=15785,tx=Transaction[eventId=26137,thread=1369:'ForkJoinPool-34-worker-12',status=STATUS_ACTIVE,resourceCount=1]]._ Wait list:ExclusiveLock[ForsetiClient[id=15794,tx=Transaction[eventId=26145,thread=1444:'ForkJoinPool-34-worker-30',status=STATUS_ACTIVE,resourceCount=1]] waits for [15785, 15793, 15794, ]]

Error occurred during commit for the ThingworxPersistenceProvider persistence provider.

Anyone can help? Thanks a lot.

[CRArko]

Hello, Tom.

We found some internal documentation which suggest trying the following:

• Use the fully qualified hostname or ip address of the results micro service server in the results.destination.uri property in microservice.properties, for example

results.destination.uri=results://myserver.mydomain.com:8096/results/models

• Restart the training and results microservices
• Then reset the alert using the RetrainAlert service or  DisableAlertsForProperty /  EnableAlertsForPropertyservices of the monitored Thing

Can you try this and let us know the results?

Thanks,

-- Craig A.

[weli]

Hi Craig,

Thanks for your help, here is my configruation of  microservices.properties

dataset.webservices.url=

dataset.app.id=

dataset.app.key=

results.destination.uri=results://192.168.31.202:8090/results/models

file.storage.path=tempStorage

I can access http://192.168.31.202:8090/results/models​ in browser at Thingworx server. And I already restart Tomcat of Thingworx, and restart the training and results microservices.  Then using RestartAnomalyAlert service to retrain,but I got the same error. I doubt something wrong with my database.

It seems I can't use DisableAlertsForProperty or EnableAlertsForProperty services in  AnomalyServices, because it doesn't accept parameter of thing name. Also I can't use service of RetrainAnomalyAlert, it says:

JavaException: com.thingworx.common.exceptions.InvalidRequestException: Attempted to retrain ThingWatcher [Thing4Anomaly2-Simulation_Examples_Functions_Ramp1-amomalydemo2] that is in an unsupported state [FAILED] (RetrainAnomalyAlert#38)

Thanks & Best Regards

Tom

[cmorfin]

Hi Tom

Could you provide some more details as below:

- are you deploying the microservices as docker or jar file ?

- could you provide the full command you execute to start the microservices or docker run command

- could you send a screenshot of AlertProcessingSubsytem > Configuration page

- are ThingWorx and microservices on the same machine or different ?

- what is the OS of ThingWorx server and microservices server ?

Could you uplaod:

- ThingWorxStorage/logs/ApplicationLog.log

- Tomcat/logs from ThingWorx machine

Note to reset the alert you can maybe try the disableAllAlert and EnableAllAlert service instead for the time being.

Thanks

Christophe

[weli]

Hi Christophe,

Thanks for your help. I believe my configuration is correct. Because the monitoring once working properly. And I just tried restart VMware(Thingworx installed in VMware), it is working again,don't know why.Although I still got the error messages which I putted in my original post. At least I can say those error dose not affect anomaly detection.

Thanks & Best Regards

Tom