Raspberry PI OS (Buster) EMS - SSL handshake error / Verify Failed / twWs_Connect: Error restarting

[dlolling]

Hello,

I am currently running EMS version MED-61060-CD-054_SP9_Microserver-Linux-arm-hwfpu-openssl-5-4-9-1154 on Raspberry PI OS (Linux ems01 4.19.118+ #1311 Mon Apr 27 14:16:15 BST 2020 armv6l GNU/Linux) and am trying to connect to the 30 day hosted trial of TW.

I am using this guide "Setup a Raspberry Pi as an IoT Device".  This guide specifies to use Buster and version SP5 of the ems, which I have also tried.  I have worked through several of these guides and many of the solutions offered on this forum but still cannot get the Linux RPI version to work.  I have been able to get the Windows version of the EMS to connect to my online instance of TW.

As it stands, I am now getting the following error:

TW_NEW_SSL_CLIENT: SSL handshake error. Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed.

SDK: twWs_Connect: Error restarting socket. Error 0

If I follow the suggestion to set encryption to:

"ws_connection": {
"encryption": "none"

I get:

[TRACE] 2020-06-15 19:01:53,984 SDK: twWs_Connect: Got Response from Server:

HTTP/1.1 400
Content-Type: text/plain;charset=ISO-8859-1
Connection: close

Bad Request
This combination of host and port requires TLS.[ERROR] 2020-06-15 19:01:53,991 SDK: twWs_Connect: Error initializing web socket. Response code: 400

SDK: twWs_Connect: Error initializing web socket. Response code: 400

I have tried various combinations of raspbaerry pi OS version from "jessie, "sketch" and "buster" using each of the suggested EMS versions and continue to end up with various errors.  This is the closest I have come to getting it to work.  From the logs it appears to be connecting to the remote server.

Below is my config.json

{
"ws_servers": [{
"host": "pp-##########.devportal.ptc.io",
"port": 443
}],
"resource": "/Thingworx/WS",
"appkey": "##################",
"logger": {
"level": "TRACE",
"publish_directory": "/home/pi/microserver/logs",
"publish_level": "TRACE",
"max_file_storage": 2000000,
"auto_flush": true
},
"http_server": {
"ssl": false,
"authenticate": false
},
"ws_connection": {
"encryption": "ssl"
},
"certificates": {
"validate": false
},
"tunnel": {
"buffer_size": 8192,
"read_timeout": 10,
"idle_timeout": 300000,
"max_concurrent": 4
},
"file": {
"buffer_size": 8192,
"max_file_size": 8000000000,
"virtual_dirs": [{
"other": "/home/pi/microserver/other"
}, {
"tw": "/home/pi/microserver/tw"
}, {
"updates": "/home/pi/microserver/updates"
}],
"staging_dir": "/home/pi/microserver/staging"
},
"data_security": {
"key_hash": "#############"
}
}

I enabled TRACE to hope to find some additional info in the logs.  I am attaching it as a file.

I appreciate any help you can offer.

Thanks,

Dave

[jwyatt]

Greetings,

You mention that you're using the Setup a Raspberry Pi as an IoT Device guide. However, we're seeing multiple differences between your config.json file and the one provided in said guide.

The guide's code for config.json is shown in this step: https://developer.thingworx.com/en/resources/guides/setup-raspberry-pi-iot-device/mlp-pi-configure-ems

For clarity, here it is in a code block:

{
    "ws_servers": [{
        "host": "YOUR_IP_ADDRESS_HERE",
        "port": 443
    }],
    "appKey": "YOUR_APP_KEY_HERE",
    "logger": {
        "level":             "INFO",
        "publish_directory": "\/home\/pi\/Downloads\/microserver\/logs",
        "publish_level":     "INFO",
        "max_file_storage":  2000000,
        "auto_flush":        true
    },
    "http_server":  {
        "ssl": false,
        "authenticate": false
    },
    "ws_connection": {
        "encryption": "ssl"
    },
    "certificates": {
        "validate": false
    },
    "tunnel": {
        "buffer_size":    8192,  
        "read_timeout":   10,  
        "idle_timeout":   300000,
        "max_concurrent": 4
    },
    "file": {
        "buffer_size": 8192, 
        "max_file_size": 8000000000,  
        "virtual_dirs": [   
            {"other":   "\/home\/pi\/Downloads\/microserver\/other"},
            {"tw":      "\/home\/pi\/Downloads\/microserver\/tw"},     
            {"updates": "\/home\/pi\/Downloads\/microserver\/updates"}          
        ],
        "staging_dir":  "\/home\/pi\/Downloads\/microserver\/staging" 
    }   
}

Compared to your previously-linked code, we see the following:

1. You have a resource block, which isn't in the default code.
2. You also have a data_security block with a reference to key_hash, which also isn't present in ours.
3. Also, it looks like you're not escaping your pathways with a \ character, i.e. a forward slash. Without the escape character, I'm uncertain if any of your pathways will work correctly. 

At this point, I would recommend trying to get the connection working by strictly following the provided DevPortal guide config.json code, and see if you can establish a basic connection. 

If so, then you could go back and try to begin modifying on your own.

Thank you,

Jason

[jwyatt]

Sorry, meant "back slash"... not "forward slash".

[dlolling]

Thank you, I did try the code exactly as outlined in the step and replaced with the IP and key.  I noticed after trying the start the EMS additional code was being added to config.json.  I began experimenting with different config options trying to get at least something to work.

I will go back and start with the config as you outline and post any errors I encounter.

Thanks for responding.

Dave

[dlolling]

No luck yet!

I went through the guide again, recreating all of the files as outlined.  My only change was installing the EMS in /home/pi/microserver.  I think when I made my original post I had tried so many different things the last config I tried did not have the escaped back slashes.  I followed the same guide and was able to get it to work on my Windows 10 laptop, so I know the portal side is working.

Below is the output of my ssh session:

 sudo ./wsems
[WARN ] 2020-06-24 21:56:10,852 jsonConfigurator::addItem: Parent data_security not found.  Creating it
[ERROR] 2020-06-24 21:56:10,865 jsonConfigurator::setEncryptedString: Key or Parent cannot be null for call
[FORCE] 2020-06-24 21:56:10,883 ./wsems: Initializing EMS ....
[FORCE] 2020-06-24 21:56:10,886 main: Using config file /home/pi/microserver/etc/config.json
[INFO ] 2020-06-24 21:56:10,887 ./wsems: Creating the WsEms proxy.
[INFO ] 2020-06-24 21:56:10,893 ./wsems: Starting HTTP Server.
[WARN ] 2020-06-24 21:56:10,897 Main: Encryption is disabled on HTTP Server.
[WARN ] 2020-06-24 21:56:10,901 Main: Authentication is disabled on the HTTP Server.
[INFO ] 2020-06-24 21:56:10,904 ./wsems: Initializing the ThingWorx REST interface.
[INFO ] 2020-06-24 21:56:10,908 wsEmsProxy::initialize: FIPS is disabled.
[INFO ] 2020-06-24 21:56:10,910 wsEmsProxy::initialize: Encryption is enabled on Web Socket connection.
[FORCE] 2020-06-24 21:56:10,911 httpServer: starting http server port=8000
[WARN ] 2020-06-24 22:56:10,928 SDK: SDK Version: 2.2.11
[WARN ] 2020-06-24 22:56:10,933 SDK: TLS Library: OpenSSL
[WARN ] 2020-06-24 22:56:10,948 SDK: TLS Library Version: 1.1.1c
[INFO ] 2020-06-24 21:56:11,84 WsProxy::initialize: EMS Version 5.4.9.1154
[INFO ] 2020-06-24 21:56:11,87 WsProxy::initialize: twApi singleton initialized
[WARN ] 2020-06-24 21:56:11,91 WsProxy::initialize: Certificate validation is disabled.
[INFO ] 2020-06-24 21:56:11,98 wsEmsProxy::initialize: Initialization complete!
[INFO ] 2020-06-24 21:56:11,109 ./wsems: Starting the connection.

PP-2006111944YT.devportal.ptc.io:443-->[ERROR] 2020-06-24 22:56:20,377 SDK: TW_NEW_SSL_CLIENT: SSL handshake error. Error: error:1416F086:SSL r                                                  outines:tls_process_server_certificate:certificate verify failed.
[ERROR] 2020-06-24 22:56:20,382 SDK: twWs_Connect: Error restarting socket.  Error 0
[INFO ] 2020-06-24 21:56:23,964 WsEmsProxy::registerVirtualDirs: Adding virtual dir other: /home/pi/microserver/other
[INFO ] 2020-06-24 21:56:23,973 WsEmsProxy::registerVirtualDirs: Adding virtual dir tw: /home/pi/microserver/tw
[INFO ] 2020-06-24 21:56:23,983 WsEmsProxy::registerVirtualDirs: Adding virtual dir updates: /home/pi/microserver/updates
[ERROR] 2020-06-24 22:56:24,33 SDK: twMessage_Send: Error sending Message with RequestId 2
[ERROR] 2020-06-24 22:56:24,42 SDK: twMessage_Send: Error sending request
[ERROR] 2020-06-24 22:56:24,145 SDK: twMessage_Send: Error sending Message with RequestId 3
[ERROR] 2020-06-24 22:56:24,149 SDK: twMessage_Send: Error sending request
[ERROR] 2020-06-24 22:56:25,514 SDK: TW_NEW_SSL_CLIENT: SSL handshake error. Error: error:1416F086:SSL routines:tls_process_server_certificate:                                                  certificate verify failed.
[ERROR] 2020-06-24 22:56:25,518 SDK: twWs_Connect: Error restarting socket.  Error 0
[ERROR] 2020-06-24 22:56:30,654 SDK: TW_NEW_SSL_CLIENT: SSL handshake error. Error: error:1416F086:SSL routines:tls_process_server_certificate:                                                  certificate verify failed.
[ERROR] 2020-06-24 22:56:30,659 SDK: twWs_Connect: Error restarting socket.  Error 0
[ERROR] 2020-06-24 22:56:35,839 SDK: TW_NEW_SSL_CLIENT: SSL handshake error. Error: error:1416F086:SSL routines:tls_process_server_certificate:                                                  certificate verify failed.
[ERROR] 2020-06-24 22:56:35,844 SDK: twWs_Connect: Error restarting socket.  Error 0
[ERROR] 2020-06-24 22:56:40,973 SDK: TW_NEW_SSL_CLIENT: SSL handshake error. Error: error:1416F086:SSL routines:tls_process_server_certificate:                                                  certificate verify failed.
[ERROR] 2020-06-24 22:56:40,986 SDK: twWs_Connect: Error restarting socket.  Error 0
[ERROR] 2020-06-24 22:56:46,115 SDK: TW_NEW_SSL_CLIENT: SSL handshake error. Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed.
[ERROR] 2020-06-24 22:56:46,120 SDK: twWs_Connect: Error restarting socket.  Error 0
[ERROR] 2020-06-24 22:56:51,252 SDK: TW_NEW_SSL_CLIENT: SSL handshake error. Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed.
[ERROR] 2020-06-24 22:56:51,257 SDK: twWs_Connect: Error restarting socket.  Error 0
[ERROR] 2020-06-24 22:56:56,409 SDK: TW_NEW_SSL_CLIENT: SSL handshake error. Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed.
[ERROR] 2020-06-24 22:56:56,414 SDK: twWs_Connect: Error restarting socket.  Error 0
[ERROR] 2020-06-24 22:57:01,544 SDK: TW_NEW_SSL_CLIENT: SSL handshake error. Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed.
[ERROR] 2020-06-24 22:57:01,548 SDK: twWs_Connect: Error restarting socket.  Error 0
^C

Here is my config.json before starting:

{
    "ws_servers": [{
        "host": "XXXXXXXXX.devportal.ptc.io",
        "port": 443
    }],
    "appKey": "XXXXXXXXXXXXXXX",
    "logger": {
        "level":             "INFO",
        "publish_directory": "\/home\/pi\/microserver\/logs",
        "publish_level":     "INFO",
        "max_file_storage":  2000000,
        "auto_flush":        true
    },
    "http_server":  {
        "ssl": false,
        "authenticate": false
    },
    "ws_connection": {
        "encryption": "ssl"
    },
    "certificates": {
        "validate": false
    },
    "tunnel": {
        "buffer_size":    8192, 
        "read_timeout":   10, 
        "idle_timeout":   300000,
        "max_concurrent": 4
    },
    "file": {
        "buffer_size": 8192,
        "max_file_size": 8000000000, 
        "virtual_dirs": [  
            {"other":   "\/home\/pi\/microserver\/other"},
            {"tw":      "\/home\/pi\/microserver\/tw"},    
            {"updates": "\/home\/pi\/microserver\/updates"}         
        ],
        "staging_dir":  "\/home\/pi\/microserver\/staging"
    }  
}

AFTER trying to start wsems, I noticed config.json gets modified.  Is that normal behavior?

The appKey is changed and is much longer and a data_security stanza is added to the end:

        "data_security":        {
                "key_hash":     "2zoylXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
        }

I keep searching for a solution and try the suggestions and example I see on the forum and Google searches and with each change, new errors appear so it seems like an endless loop of trial and error.

I have tried MED-61060-CD-054_SP9_Microserver-Linux-arm-hwfpu-openssl-5-4-9-1154 and MED-61060-CD-054_SP9_Microserver-Linux-arm-openssl-5-4-9-1154 on Raspberry OS (Buster) as well as older version, 5-4-4 and 5-4-5 on older versions of the OS just for the sake of using the recommended OS with each version of the EMS.

Any suggestions would be great.

[jwyatt]

Are you using the EMS version provided in this step of the guide?

https://developer.thingworx.com/en/resources/guides/setup-raspberry-pi-iot-device/mlp-pi-install-ems

It's the version of EMS for Linux, with an ARM CPU *WITH* a hardware floating point unit (hwfpu), and it's version 5.4.5.512. 

Because, to the best of my knowledge, your config.json file should not be being modified with that version. In fact, I've never heard of any EMS version changing the config.json file.

If the config.json file doesn't work, then the EMS should reject it, and it should default back to the "last known good" config file, i.e. config.booted.

Ah, and I see at the bottom of your comment that you appear to be trying to match OS-versions to recommended EMS versions.

Unfortunately, the TWX DevPortal can only support the exact versions against which the guide was written. 

It's *possible* that a new version of the EMS does modify the config.json file. Each version of the EMS has its own requirements for the config.json file, and it's possible that some of these sections that are getting added are now mandatory in newer versions. 

But until we're able to update the guide to something newer, we'll have to require the versions listed for any DevPortal support.

However, it's possible someone else on the Community is familiar with different EMS versions. 

[dlolling]

I have tried two Raspberry PI related guides with the recommended OS and EMS versions.

Guide 1:
https://developer.thingworx.com/en/resources/guides/thingworx-raspberry-pi-quickstart
Using Raspbian Jessie with MED-61060-CD-054_F000_Microserver-Linux-arm-hwfpu-openssl-5-4-5-512.zip
I followed the guide exactly and encountered a variety of errors.

Guide 2:
https://developer.thingworx.com/en/resources/guides/setup-raspberry-pi-iot-device.
All of the links refer to Rasbian OS and the links in the guide take you to the latest version of
the OS which is Buster and EMS MED-61060-CD-054_SP5_Microserver-Linux-arm-hwfpu-openssl-5-4-5-512.zip. I originally tried this combination and encountered numerous errors.

I also tried https://developer.thingworx.com/en/resources/guides/use-edge-microserver-ems-connect-thingworx
using Windows 10 and MED-61060-CD-054_F000_Microserver-Windows-x86-32-openssl-5-4-0-114.zip which worked just fine.

Lastly, the Guide https://developer.thingworx.com/en/resources/guides/raspberry-pi-ar-tutorial does not
require an EMS. It just uses Python scripts.  I have not tried this yet.

I will follow Guide 2 again using the specified EMS and report back.  I am assuming because the links in the guide refer  to the latest Raspberry Pi OS that is what I need to use?

Thanks,

[jwyatt]

In the "Setup a Raspberry Pi as an IoT Device" guide, it appears we need to update the link. Thank you for pointing it out. 

The link goes to "latest", while the screenshot shows 2019-07-10-raspbian-buster-full.zip.

As such, that was the OS against which the Linus ARM hwfpu 5.4.5 EMS was tested for that guide.

We'll update the guide to restrict the OS version further. 

[dlolling]

I did a clean install using 2019-07-10-raspbian-buster-full.zip and did not apply any OS updates.  I unzipped

MED-61060-CD-054_SP5_Microserver-Linux-arm-hwfpu-openssl-5-4-5-512.zip in /home/pi/microserver .

I created all of the necessary directories, config files and test files as per the guide.  I receive the following error when starting wsems.

pi@raspberrypi:~/microserver $ sudo ./wsems
[FORCE] 2020-06-26 16:01:18,350 ./wsems: Initializing EMS ....
[FORCE] 2020-06-26 16:01:18,350 main: Using config file /home/pi/microserver/etc/config.json
[INFO ] 2020-06-26 16:01:18,350 ./wsems: Creating the WsEms proxy.
[INFO ] 2020-06-26 16:01:18,351 ./wsems: Starting HTTP Server.
[WARN ] 2020-06-26 16:01:18,352 Main: Encryption is disabled on HTTP Server.
[WARN ] 2020-06-26 16:01:18,352 Main: Authentication is disabled on the HTTP Server.
[INFO ] 2020-06-26 16:01:18,353 ./wsems: Initializing the ThingWorx REST interface.
[INFO ] 2020-06-26 16:01:18,354 wsEmsProxy::initialize: FIPS is disabled.
[INFO ] 2020-06-26 16:01:18,355 wsEmsProxy::initialize: Encryption is enabled on Web Socket connection.
[FORCE] 2020-06-26 16:01:18,368 httpServer: starting http server port=8000
[WARN ] 2020-06-26 11:01:18,381 SDK: SDK Version: 2.2.0
[WARN ] 2020-06-26 11:01:18,388 SDK: TLS Library: OpenSSL
[WARN ] 2020-06-26 11:01:18,391 SDK: TLS Library Version: 1.0.2l-fips
[WARN ] 2020-06-26 11:01:18,394 SDK: FIPS Capable
Illegal instruction

Config files are below:  (The cut and paste messed up the formatting)

config.json

{
"ws_servers": [{
"host": "XXXXXXXXXX.devportal.ptc.io",
"port": 443
}],
"appKey": "XXXXXXXXXXXX",
"logger": {
"level": "INFO",
"publish_directory": "\/home\/pi\/microserver\/logs",
"publish_level": "INFO",
"max_file_storage": 2000000,
"auto_flush": true
},
"http_server": {
"ssl": false,
"authenticate": false
},
"ws_connection": {
"encryption": "ssl"
},
"certificates": {
"validate": false
},
"tunnel": {
"buffer_size": 8192,
"read_timeout": 10,
"idle_timeout": 300000,
"max_concurrent": 4
},
"file": {
"buffer_size": 8192,
"max_file_size": 8000000000,
"virtual_dirs": [
{"other": "\/home\/pi\/microserver\/other"},
{"tw": "\/home\/pi\/microserver\/tw"},
{"updates": "\/home\/pi\/microserver\/updates"}
],
"staging_dir": "\/home\/pi\/microserver\/staging"
}
}

config.lua

scripts.log_level = "WARN"

scripts.script_resource_ssl = false
scripts.script_resource_authenticate = false

scripts.PiThing = {
file = "thing.lua",
template = "YourEdgeThingTemplate",
scanRate = 120000,
sw_update_dir = "\/home\/pi\/microserver\/updates"
}

/home/pi/microserver/etc/custom/templates/YourEdgeThingTemplate.lua

require "shapes.swupdate"

module ("templates.YourEdgeThingTemplate", thingworx.template.extend)

I have searched PTC and Google looking for solutions but have yet to find the answer.

[dlolling]

On ‎06-12-2020 12:34 PM I created this post "EMS fails to start on Raspberry PI OS (Buster) - Illegal Instruction". This was my first attempt at installing the EMS on Raspberry Pi. I later moved on to trying different EMS software versions, different OS versions and changed the config.json based on other solutions I found on the forums and Google.

[jwyatt]

Greetings again.

To level-set, the "Setup a Raspberry Pi as an IoT Device" guide on the TWX DevPortal was written using the following:

1. Raspbian Buster Full (unupdated)
2. Linux ARM HWFPU 5.4.5 version of the EMS
3. Foundation 8.5 (DevPortal hosted trial)

One additional note that has not been mentioned thus far is that it was written against a Pi 2, and tested by another developer using a Pi 3. I'm unsure of the Pi version you've been using, but I don't see a reason why that would cause an issue.

Investigating further, we *HAVE* seen some modifications to the config.json file using newer versions. In particular, we've noted the need for the following in the "certificates" section:

"certificates":  {
    "validate": false,
    "disable_hostname_validation": true
},

However, the "disable_hostname_validation" flag is something that should *NOT* be required when strictly following the guide and using the components mentioned above, nor should you see config.json modifications, so *something* is going on.

Moving forward, you may wish to reach out to PTC Tech Support. You may do so here:

https://support.ptc.com/apps/case_logger_viewer/cs/auth/ssl/log

However, filing a Support ticket requires that you have a currently-valid PTC support-contract. 

If that is not the case, we recommend reaching out to either your Sales person or Customer Success agent to see if they can assist. 

Thank you,

Jason

[dlolling]

Thank you again for your response.

I am running the same hardware (PI 2), OS and EMS as you stated above and following the guide exactly as outline.  As I mentioned before I had previously posted about encountering an illegal instruction when I started the guide the first time.

I added the additional line in my config.json and it threw the same error.   Illegal instruction.

Any way to determine what is throwing the illegal instruction or what part of the config?

Any other suggestions as to try to determine the source of the error?

I ran strace against wsems and attached the files for your entertainment.

Unless something obvious pops out and solves this, I am going to have to move on.  We do have support but I have spent too much time already.

Are the versions of EMS that are provided for the guides the same as something that would be deployed by a customer in a production environment or are these just for demo purposes only?

Thanks,

[dlolling · Accepted solution]

@jwyatt 

Hello,

As a follow up, I decided to transfer the contents of the microserver directory over to my other Raspberry PI 4 (8gb version) running the latest up to date version of Raspberry PI OS (Buster) and everything worked like a champ.

So not sure what was causing the problems as I was using exactly what was called out in the guide.  

But anyway, thanks for the help, glad to get it working as intended.  Planning on trying an Arduino next!

Dave

[slangley]

Hi @dlolling.

Since you have found a solution to your problem, please mark the appropriate response to this topic as the Accepted Solution (even if its yours) for the benefit of others with the same issue.

Thank you for your participation in our community!

Regards.

--Sharon