Property transform not working on platform 8.5.2

[olatz]

Hello,

After installing Platform-Analytics, Property Transform is not working. 

My environment is:

- Virtual Machine 1

     - THW Platform: 8.5.2

     - SSL:443

     - Autogenerated certificate

- Virtual Machine 2

     - Analytics Server: 8.5.2

          - Connected to the platform and working

    - Platform-Analytics: 8.5.1

          - RabbitMQ and Apache Flink from the installer

          - Erlang 21.0

          - The installation summary said that everything went OK and Things were created

I have the following error in configuration.log: (attached)

2020-09-17 08:59:41 [pool-1-thread-1] ERROR c.t.a.d.c.FlinkStatusCheck - Error while checking status of Flink cluster
org.apache.flink.util.FlinkException: Could not connect to the leading JobManager. Please check that the JobManager is running.
at org.apache.flink.client.program.ClusterClient.getJobManagerGateway(ClusterClient.java:862)
at org.apache.flink.client.program.ClusterClient.listJobs(ClusterClient.java:700)
at com.thingworx.analytics.derived.configuration.FlinkStatusCheck.check(FlinkStatusCheck.java:45)
at com.thingworx.analytics.derived.configuration.FlinkStatusCheck.lambda$new$0(FlinkStatusCheck.java:33)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

In the flink1-4 log file, there is no error printed (see attached), the only thing that seems strange is the message "JAVA HOME: (not set)" because in /etc/environments 

            export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_141/

is set.

I can't find any reference related with the error I am getting with flink in the knowledge database.

Thanks in advance for your help.

Olatz.

[cmorfin]

Hi @olatz 

Could you please execute TWALogCollector on the PlatformAnalytics machine and upload the gz file created

Thanks

Christophe

[olatz]

Hi @cmorfin 

Attached the file.

Thanks for your help,

Olatz

[olatz · Accepted solution]

Hi @cmorfin 

I think that finally, I have discovered where the error is. 

I have multiple hostnames asigned to the same IP in /etc/hosts, this was causing that the flink jobmanager was listening in a hostname, and property transform was trying to connect to another hostname.

Anyway, thank you for your willingness to help me with the problem.

Olatz.

[cmorfin]

Hi @olatz 

Thank you for the update, that is great news.

I was in the process of setting up a system similar to yours as I did not spot much in the log.

Could I ask you what pointed you in this direction ? Did you see something in the logs ?

Thank you

Christophe

[olatz]

Hi @cmorfin,

Yes, the hostnames where different in different log files. There are lot of logs and some of them print the IP, others print the hostname. In my case, all the services are in the same machine (property transform, flink, rabbitmq, ...).

• In configuration.log:

2020-09-17 08:59:31 [flink-akka.actor.default-dispatcher-4] INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@thingworx-analytics-8.5:37249]

• In flink-twxanalytics-jobmanager-0-Thingworx-analytics-8.log:

2020-09-17 08:59:13,673 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager at akka.tcp://flink@localhost:6123/user/jobmanager.

2020-09-17 08:59:14,999 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at Thingworx-analytics-8 (akka.tcp://flink@thingworx-analytics-8:37113/user/taskmanager) as 632a7f71e5f9acc802885788a8c9808b. Current number of registered hosts is 1. Current number of alive task slots is 200.

• flink-twxanalytics-taskmanager-0-Thingworx-analytics-8.log

2020-09-17 08:59:14,089 INFO org.apache.flink.runtime.taskmanager.TaskManager - Actor system started at akka.tcp://flink@thingworx-analytics-8:37113
2020-09-17 08:59:14,095 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
2020-09-17 08:59:14,097 INFO org.apache.flink.runtime.taskmanager.TaskManager - Starting TaskManager actor
2020-09-17 08:59:14,100 INFO org.apache.flink.runtime.io.network.netty.NettyConfig - NettyConfig [server address: Thingworx-analytics-8/127.0.1.1, server port: 0, ssl enabled: false, memory segment size (bytes): 32768, transport type: NIO, number of server threads: 200 (manual), number of client threads: 200 (manual), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)]

We don't need to have two hostnames assigned to the same IP. I really don't know why it was configured this way. The IP 127.0.1.1 was assigned two hostnames, Thingworx-analytics-8 and thingworx-analytics-8.5. I don't remember editing the /etc/hosts file to include 127.0.1.1 IP, nor adding two hostnames to the same IP address. Could be possible that this information was added during rabbitmq installation?

Thanks in advance for your help.

Olatz.

[cmorfin]

Hi @olatz 

Thank you very much for the details, very interesting.

I have just made an installation of Platform Analytics 8.5.1 on Ubuntu 18.04 and there has been no modification of /etc/hosts, so I would think those entries have been added by someone.

Since you have identified the root cause, it would be useful if you can mark your answer as Solution as it will help others in the future who may have a similar situation.

Thank you

Christophe