IoT & Connectivity Tips

In the process of working with a customer, I was curious as to the throughput of a file sent via the Axeda Connected Content feature to one of the Axeda Agent Gateways.   I took a random 50 megabyte blob of data (/dev/urandom) and sent it to one of my test Gateways via a Package deployment: DEBUG   xgEnterpriseProxy: Enterprise Queue Empty INFO    xgSM:  ... Download percent done = 11% INFO    xgSM:  ... Download percent done = 21% INFO    xgSM:  ... Download percent done = 31% INFO    xgSM:  ... Download percent done = 41% INFO    xgSM:  ... Download percent done = 51% INFO    xgSM:  ... Download percent done = 61% INFO    xgSM:  ... Download percent done = 71% INFO    xgSM:  ... Download percent done = 81% INFO    xgSM:  ... Download percent done = 91% INFO    xgSM:  ... Download percent done = 100% DEBUG   xgSM: >>  INTERNAL DEBUG MESSAGE << :  Download time is 4 seconds DEBUG   xgSM: >>  INTERNAL DEBUG MESSAGE << :  Upgrading.  Backing up files to C:\temp\CFKGW\AxedaBackup DEBUG   xgSM: >>  INTERNAL DEBUG MESSAGE << :  Extracting downloaded files from DefaultProject\CFKGW\Downloads\141581_143281.tar.gz to directory C:\temp\CFKGW\ DEBUG   xgSM: >>  INTERNAL DEBUG MESSAGE << :  Extraction Finished About 12MB per second.  This was a sandbox in the PTC On-Demand Center.  Not bad, but not necessarily representative of a real production system.  This sandbox doesn't have 1000 devices trying to get this file at once.  So some benchmarking in your configuration and environment certainly needs to be done. So that done, I thought I'd up the ante - 700 megabytes this time! DEBUG   xgEnterpriseProxy: Enterprise Queue Empty INFO    xgSM:  ... Download percent done = 10% INFO    xgSM:  ... Download percent done = 20% INFO    xgSM:  ... Download percent done = 30% INFO    xgSM:  ... Download percent done = 40% INFO    xgSM:  ... Download percent done = 50% INFO    xgSM:  ... Download percent done = 60% INFO    xgSM:  ... Download percent done = 70% INFO    xgSM:  ... Download percent done = 80% INFO    xgSM:  ... Download percent done = 90% INFO    xgSM:  ... Download percent done = 100% DEBUG   xgSM: >>  INTERNAL DEBUG MESSAGE << :  Download time is 66 seconds So 10MB per second. Directory of C:\temp\cfkgw 05/17/2017  01:32 PM    <DIR>          . 05/17/2017  01:32 PM    <DIR>          .. 05/17/2017  01:03 PM         1,048,576 1mb.dat 05/17/2017  01:04 PM        52,428,800 50meg-randomdata.dat 05/17/2017  01:32 PM       734,003,200 700mb.dat 05/17/2017  01:03 PM    <DIR>          AxedaBackup                3 File(s)    787,480,576 bytes Not bad at all! 

Twilio extends the ThingWorx functionality to send SMS and voice messages in variety of languages. Starting from scratch I'll cover the steps required to setup the Twilio extension and the basic account registration (trial version) and setup at Twilio.com Twilio allows registering with the trial account, which comes with $15 as initial account balance, something which is quite useful for testing the initial setup and testing of Twilio extension in conjunction to ThingWorx. Prerequisite 1. Sign up, if not already done, with Twilio.com for free trail account 2. Configure the account to setup a verified phone number to receive text and voice messages, free trial account allows setting up 1 verified phone number that can receive text and voice messages Note: See What's the difference between a verified phone number and a Twilio phone number? 3. Choose and configure a Twilio phone number capable of sending text and voice messages, free trial account allows setting up of 1 Twilio number which will be used in the Extension configuration in ThingWorx for sending the messages 4. Download Twilio Extension from ThingWorx Marketplace Note that Twilio phone number should be enabled with the send SMS or send voice message capability, depending on what your use case is. Failing to do so will lead to error while testing the sendSMS or sendVOICE service (two default services provided with the Twilio Extension when imported in ThingWorx) Signing up and setting up Twilio account 1. Sign up on Twilio.com with your details and emailID. When registering for the first time you'll be prompted for your personal phone number as part of the security requirement. Also with free trial account this will be the only phone number where you'll be able to send SMS or Voice message via Twilio Extension in ThingWorx 2. Next would be to setup the Twilio number with voice and text messaging, navigate to https://www.twilio.com/console/phone-numbers/getting-started 3. Do ensure that you setup the voice or text capabilities for the new Twilio number, failing to do so will lead to error in sending message via Twilio Extension in ThingWorx 4. Different Twilio Number have different capabilities, it's quite possible that you don't find the number with required capabilities i.e. Voice, SMS - in that scenario simply search for different number, one that has the capabilities you are looking for 5. With trial account only 1 Twilio number is avilable for setup 6. While setting up the Twilio number you will be required to provide a valid local address of your country The cost of setting up the number and further sending the SMS will only be deducted from the $15 made available when you signed up for the first time. Navigate to your account details on Twilio.com to make note of the following information which will be used when configuring the Twilio Extension in ThingWorx: a. Account SID b. Authentication ID c. Your Verified Phone Number (will be used for receiving the messages) d. Twilio Phone number created with required capabilities, e.g. that number is capable of sending text message Here's how it shows up once you are successfully registered and logged on to the twilio.com/console To check your Twilio Phone Number and the Verified Phone Number navigate to twilio.com/phone-numbers As it can be seen in the above screenshot, the number I have bought from Twilio is capable of sending Voice and SMS. With this all's set  and ready to configure the Twilio Extension in ThingWorx, which is pretty straight forward. Setup Twilio Extension in ThingWorx 1. Let's setup the extension by navigating to the ThingWorx Composer > Import/Export > Extensions > Import > browse to the location where you saved the Twilio Extension from ThingWorx Marketplace and click Import 2. Once imported successfully refresh the composer and navigate to Modeling > Things , to create a new Thing implementing the Twilio Template (imported with the extension) like so, in screenshot below Thing is named DemoTwilioThing 3. Navigate to the the Configuration section of that Thing, to provide the information we noted above from Twilio account Note: this demo thing is setup to send the text SMS to my verified phone number only, therefore the CallerID in the configuration above is the Twilio number I created after signing up on Twilio with text message capability. 4. Save the created the Thing 5. Finally, we can test with the 1 of the 2 default services provided with the Twilio Template i.e. SendSMSMessage service 6. While testing the SendSMSMessage service use your verified phone number in To for receiving the message Troubleshooting some setup related issues I'll try to cover some of the generic errors that came up while doing this whole setup with Twilio Extension Error making outgoing SMS: 404 / TwilioResponse 20404 Reason If the Twilio number is not enabled with SMS sending capabilities you may run into such an error when testing the service. Here's the full error stack Unable to Invoke Service SendSMSMessage on DemoTwilioThing : Error making outgoing SMS: 404 <?xml version='1.0' encoding='UTF-8'?><TwilioResponse><RestException><Code>20404</Code><Message>The requested resource /2010-04-01/Accounts/<AccountSID>/SMS/Messages was not found</Message><MoreInfo>https://www.twilio.com/docs/errors/20404</MoreInfo><Status>404</Status></RestException></TwilioResponse> Resolution Navigate back to the Phone Number section on Twilio's website and as highlight in screenshot above check if the Twilio number is enabled with SMS capabilities. Error making outgoing SMS:400 / TwilioResponse 21608 Reason You may encounter following error when attempting to send SMS/Voice message, here's the full error detail Wrapped java.lang.Exception: Error making outgoing SMS: 400 <?xml version='1.0' encoding='UTF-8'?><TwilioResponse><RestException><Code>21608</Code><Message>The number <somePhoneNumber>is unverified. Trial accounts cannot send messages to unverified numbers; verify <somePhoneNumber>at twilio.com/user/account/phone-numbers/verified, or purchase a Twilio number to send messages to unverified numbers.</Message><MoreInfo>https://www.twilio.com/docs/errors/21608</MoreInfo><Status>400</Status></RestException></TwilioResponse> Cause: Error making outgoing SMS: 400 <?xml version='1.0' encoding='UTF-8'?><TwilioResponse><RestException><Code>21608</Code><Message>The number <somePhoneNUmber>is unverified. Trial accounts cannot send messages to unverified numbers; verify <somePhoneNumber> at twilio.com/user/account/phone-numbers/verified, or purchase a Twilio number to send messages to unverified numbers.</Message><MoreInfo>https://www.twilio.com/docs/errors/21608</MoreInfo><Status>400</Status></RestException></TwilioResponse> Resolution As the error points out clearly the number used in To section while testing the SendSMSMessage service didn't have verified number. With free trial account you can only use the registered verified phone number where SMS/Voice message can be sent. If you want to use different number an account upgrade is required. Error making outgoing SMS:400 / TwilioResponse 21606 Reason Following error is thrown while testing SendSMSMessage service with different Twilio number which is either not the same as the number you bought when setting up the trial account or it doesn have SMS sending capabiltiy. Here's the full error stack Wrapped java.lang.Exception: Error making outgoing SMS: 400 <?xml version='1.0' encoding='UTF-8'?><TwilioResponse><RestException><Code>21606</Code><Message>The From phone number <TwilioNumber>is not a valid, SMS-capable inbound phone number or short code for your account.</Message><MoreInfo>https://www.twilio.com/docs/errors/21606</MoreInfo><Status>400</Status></RestException></TwilioResponse> Cause: Error making outgoing SMS: 400 <?xml version='1.0' encoding='UTF-8'?><TwilioResponse><RestException><Code>21606</Code><Message>The From phone number <TwilioNumber> is not a valid, SMS-capable inbound phone number or short code for your account.</Message><MoreInfo>https://www.twilio.com/docs/errors/21606</MoreInfo><Status>400</Status></RestException></TwilioResponse> Resolution Check the configuration in the ThingWorx Composer for the Thing created out of Twilio Template whether or not the callerID is configured with correct Twilio account number

It usually happens that we need to copy a large file to ThingWorx server periodically, and what's worse, the big file is changing(like a log file). This sample give a simpler way to implement. The main idea in the sample is: 1. Lower the management burden from ThingWorx server and instead it put all the work in edge SDK side 2. Save network burden with only uploading the incremented file and append it to the older file on ThingWorx server   Java SDK version in this sample: 6.0.1-255

Performance and memory issues in ThingWorx often appear at the Java Virtual Machine (JVM) level. We can frequently detect these issues by monitoring JVM garbage collection logs and thread dumps. In this first of a multi-part series, let's discuss how to analyze JVM garbage collection (GC) logs to detect memory issues in our application. What are GC logs? When enabled on the Apache server, the logs will show the memory usage over time and any memory allocation issues (See: How to enable GC logging in our KB for details on enabling this logging level).  Garbage Collection logs capture all memory cleanup operations performed by the JVM.  We can determine the proper memory configuration and whether we have memory issues by analyzing GC logs. Where do I find the GC logs? When configured as per our KB article, GC logs will be written to your Apache logs folder. Check both the gc.out and the gc.restart files for issues if present, especially if the server was restarted when experiencing problems. How do I read GC logs? There are a number of free tools for GC log analysis (e.g. http://gceasy.io, GCViewer). But GC logs can also be analyzed manually in a text editor for common issues. Reading GC files: Generally each memory cleanup operation is printed like this: Additional information on specific GC operations you might see are available from Oracle. A GC analysis tool will convert the text representation of the cleanups into a more easily readable graph: Three key scenarios indicating memory issues: Let's look at some common scenarios that might indicate a memory issue.  A case should be opened with Technical Support if any of the following are observed: Extremely long Full GC operations (30 seconds+) Full GCs occurring in a loop Memory leaks Full GCs: Full GCs are undesirable as they are a ‘stop-the-world’ operation. The longer the JVM pause, the more impact end users will see: All other active threads are stopped while the JVM attempts to make more memory available Full GC take considerable time (sometimes minutes) during which the application will appear unresponsive Example – This full GC takes 46 seconds during which time all other user activity would be stopped: 272646.952: [Full GC: 7683158K->4864182K(8482304K) 46.204661 secs] Full GC Loop: Full GCs occurring in quick succession are referred to as a Full GC loop If the JVM is unable to clean up any additional memory, the loop will potentially go on indefinitely ThingWorx will be unavailable to users while the loop continues Example – four consecutive garbage collections take nearly two minutes to resolve: 16121.092: [Full GC: 7341688K->4367406K(8482304K), 38.774491 secs] 16160.11: [Full GC: 4399944K->4350426K(8482304K), 24.273553 secs] 16184.461: [Full GC: 4350427K->4350426K(8482304K), 23.465647 secs] 16207.996: [Full GC: 4350431K->4350427K(8482304K), 21.933158 secs]      Example – the red triangles occurring at the end of the graph are a visual indication that we're in a full GC loop: Memory Leaks: Memory leaks occur in the application when an increasing number of objects are retained in memory and cannot be cleaned up, regardless of the type of garbage collection the JVM performs. When mapping the memory consumption over time, a memory leaks appears as ever-increasing memory usage that’s never reclaimed The server eventually runs out of memory, regardless of the upper bounds set           Example(memory is increasing steadily despite GCs until we'll eventually max out): What should we do if we see these issues occurring? Full GC loops and memory leaks require a heap dump to identify the root cause with high confidence The heap dump needs to be collected when the server is in a bad state Thread dumps should be collected at the same time A case should be opened with support to investigate available gc, stacktrace, or heap dump files

Here is a demo that uses repeater widget and smart grid widget to display a nested table. Note: This demo is for testing use. Create a datashape for  the nested datatable named DataTableStructure Create DataShape named InfoStructure for Obj Create the nested datatable named DataTableTest as below Create a service named AddObjInput as below in this datatable Crate a Mashup named RepeaterMsh Add widgets like textbox, label, button and smart grid on the mashup There are some parameters we added before, please bind them to the widgets that we just added to the mashup Add DataTableTest service AddOrUpdateDataTableEntry to the mashup. Bind textbox and smart grid edited table to AddOrUpdateDataTableEntry parameter. Bind button click event to trigger the service AddOrUpdateDataTableEntry The smart grid widget need a special attribute named TableDefinition This demo sample is{"columns":[{"name":"PrimarySchool","type":"text","display":"PrimarySchool"},{"name":"SecondarySchool","type":"text","display":"SecondarySchool"},{"name":"HighSchool","type":"text","display":"HighSchool"}]} Create mashup which has repeater widgets The mashup over view is like below The first part is aimed to add a new record, the second part (repeater widget)is aimed to display table data, and also update data. In the first part, there are textboxs, labels, smart grid and button widget. They are banded to the service named AddDataTableEntry, aimed to add a new record in the table. (Smart grid is used to add the nested table data) There is a button named AddRecords, bind it’s clicked event to AddDataTableEntry service. And bind the AddDataTableEntry service’s event ServiceInvokeCompleteted to GetDataTableEntries The smart grid widget in here is an empty grid, but it should have a structure at beginning. That’s why to create a service named AddObjInput in step2. It returns an empty table but with the datashape. The second part is a repeater widget. Repeater widget has an attribute named Mashup, bind the mashup Repeatermsh. Bind GetDataTableEntries all data to the repeater widget. There should be some parameters that need to bind like gender, InfoTable, Sname… Testing the demo Note: Smart Grid widget is not a ThingWorx OOTB functionality The smart grid widget can be edited flexible. Please download with is Link https://marketplace.thingworx.com/Items/Smart%20Grid%20Widget There is a thread teach how to use smart grid: https://community.thingworx.com/message/53379#53379

This is just a quick note/reminder that starting 7.4, the utilities installation process has slightly changed. Several extensions are no longer bundled with ThingWorx Utilities and must be downloaded from the ThingWorx Marketplace and installed separately. For more information, see the “Prerequisites” topic in the ThingWorx Utilities Installation Guide: http://support.ptc.com/WCMS/files/172616/en/ThingWorxUtilitiesInstall.pdf The following extensions must be installed prior to installing ThingWorx Utilities available at ThingWorx IoT Marketplace ○ Google Maps Widget ○ Mail Extension ○ Web Sockets Tunnel Widget and Library

About This is the third part of a ThingBerry related blog post series. ThingBerry is ThingWorx installed on a RaspBerry Pi, which can be used for portable demonstrations without the need of utilizing e.g. customer networks. Instead the ThingBerry provides its own custom WIFI hotspot and allows Things to connect and send / receive demo data on a small scale. In this particual blog post we'll discuss on how to setup the ThingBerry for encrypted communication and trust via certificates. We will create a custom Root and Intermediate Certificate Authority as well as a server specific certificate. More information on the theory of Trust & Encryption can be found here: Trust &amp; Encryption - Theory As the ThingBerry is a highly unsupported environment for ThingWorx, please see this blog post for all related warnings. Preparing the environment To get started and keep all the working materials safe and secure within the scope of the pi user account, let's start creating a workspace directory and then install the openssl package cd ~ mkdir certs chmod 700 certs cd certs sudo apt-get install openssl Troubleshooting Tips Keytool To look at a specific certificate use the keytool and specify the .crt file to look at keytool -printcert -file rootca.crt This will show information on the certificate's configuration, such as signing authority or validity. Timestamps It's important, that the Chain of Trust follows a timely fashion. The Root CA has the longest expiration date. Within its validity the Intermediate CA must be valid. The server specific certificate must be valid in the timeframe of the Intermediate CA validity. As the validity is calculated based on seconds of the current (signing) action, I'm using valid days of Root CA 999 days Intermediate CA 888 days Server specific certificate 777 days Any other times, that make sense can be used, as long as the time constraints are followed. Otherwise the certificate will not be recognized as valid and will therefore not be trusted. Creating a Chain of Trust For this example we'll create a Root and Intermediate Certificate Authority (CA) as well as the server specific certificate. In a first step we can create all of the required keys. openssl genrsa -des3 -out rootca.key 4096 openssl genrsa -out intermediateca.key 4096 openssl genrsa -out server.key 4096 The -des3 option for the rootca will store a password and encrypt it within the key for more secure access. This tightens up security, so that the key for the rootca is not easily corrupted or exposed. Root Certificate Authority Having the keys, we now create the Root CA itself openssl req -new -x509 -sha256 -days 999 -key rootca.key -reqexts v3_req -extensions v3_ca -out rootca.crt Enter the information, starting with the password for the Root CA key. Use a easy to identify name for the Common Name (CN), e.g. "My Root CA" Intermediate Certificate Authority As the Intermediate CA needs to be signed / approved by the Root CA, we need a Certficiate Signing Request (CSR) openssl req -new -key intermediateca.key -reqexts v3_req -extensions v3_ca -out intermediateca.csr The v3_ca extension will mark this certificate as CA itself. This means that we can use the intermediate CA to sign other certificates as well. The CSR will be signed with the Root CA's key and certificate. For this the CSR must be sent to the whoever own the Root CA. In our case, we're the owner ourselves and have everything in the same directory, so no need to send it off to Let's Encrypt, Google or VeriSign. For signing the Intermediate CA as an actual CA, the Root CA must have a specific configuration to allow the v3_ca extension. This needs to be created via nano v3_ca.ext Into this new file, copy & paste the following: basicConstraints        = CA:TRUE subjectKeyIdentifier    = hash authorityKeyIdentifier  = keyid:always,issuer:always keyUsage                = cRLSign, dataEncipherment, digitalSignature, keyCertSign, keyEncipherment, nonRepudiation Save and exit. Finally, the Root CA is signing the Intermediate CA, using the v3_ca extension via: openssl x509 -req -in intermediateca.csr -CA rootca.crt -CAkey rootca.key -CAcreateserial -CAserial intermediateca.srl -extfile v3_ca.ext -days 888 -sha256 -out intermediateca.crt Certificate Authority Chain Having now two CAs, they need to be chained together. This is required for later to create the keystore used in Tomcat. Creating the Chain is probably the easiest part of this blog: cat intermediateca.crt rootca.crt > cachain.crt Server Specific Certificate The server specific certificate has to be signed and obtained by the Intermediate CA; for this a CSR is required. openssl req -new -key server.key -out server.csr The request is then signed with the Intermediate CA's key and certificate. For commerical certificates this signing process will be done by publicly trusted CAs, like Let's Encrypt, VeriSign or Google. In our case, we're the Intermediate CA ourselves and can sign the CSR. openssl x509 -req -in server.csr -CA intermediateca.crt -CAkey intermediateca.key -CAcreateserial -CAserial server.srl -days 777 -sha256 -out server.crt Creating Keystores For the keystore as used in Tomcat we need the CA Chain as well as the server specific certificate and the corresponding key. To generate a keystore that can be used in Tomcat, we first need to create a PKCS12 type keystore with openssl openssl pkcs12 -export -certfile cachain.crt -in server.crt -inkey server.key -out server.p12 This PKCS12 keystore can now be transformed into a JKS keystore, by importing the PKCS12 information into a new JKS keystore keytool -importkeystore -srckeystore server.p12 -srcstoretype PKCS12 -destkeystore keystore.jks -deststoretype JKS Important: the passwords for the PKCS12 keystore and the JKS keystore must match! Tomcat requires those passwords to be same - in case they are different, the configuration will not work and ThingWorx cannot be accessed through a secure connection. Configuring ThingWorx For ThingWorx, only the keystore is required. It can be copied over e.g. to certificates area in the storage directory. sudo cp keystore.jks /thingworx/storage/certificates sudo chmod 640 /thingworx/storage/certificates/keystore.jks sudo chown tomcat:tomcat /thingworx/storage/certificates/keystore.jks To enable it, Tomcat's server.xml needs to be updated. sudo nano $CATALINA_HOME/conf/server.xml Find the connector for port 80 and add a new connector after the port 80 connector: <Connector port="443" protocol="org.apache.coyote.http11.Http11NioProtocol"            maxThreads="150" connectionTimeout="20000"  redirectPort="8443"            SSLEnabled="true" scheme="https" secure="true" clientAuth="false" sslProtocol="TLSv1.2" enableLookups="false"            keystoreFile="/thingworx/storage/certificates/keystore.jks" keystorePass="<keystorePass>" /> Don't forget to update the password. Stop Tomcat sudo service tomcat stop Ensure Tomcat is actually down with ps -ef | grep tomcat If it's still running, just sudo kill it and start Tomcat again to activate the new configuration sudo service tomcat start And now? Congratulations! ThingWorx is now running on your ThingBerry using a secure channel. Access it via https://<thingberry>/Thingworx As HTTPS is automatically using port 443, you will automatically connect to the port configured in the server.xml Classic HTTP connections on port 80 can still be used. For more information about setting up ThingWorx with (self-signed) certificates also see https://support.ptc.com/appserver/cs/view/solution.jsp?n=CS193947 - there's also a note on forwarding all HTTP traffic to HTTPS. See also https://support.ptc.com/appserver/cs/view/solution.jsp?n=CS246292 for more information about using specific SSL protocols or cipher suites.

In this blog post I'm covering the pratical aspects of setting up a Chain of Trust and configure ThingWorx to use it. There are already some examples available on how to do this via command line using a self-signed certificate, e.g. https://support.ptc.com/appserver/cs/view/solution.jsp?n=CS193947 In this example I'll be using the KeyStore Explorer, a graphical tool for managing keystores. http://keystore-explorer.org/ To learn more about the theory behind trust and encryption, see also Trust &amp; Encryption - Theory The Chain of Trust Creating a Root CA Open the KeyStore Explorer and create a new jks KeyStore File > New > JKS Via Tools > Generate Key Pair create a new key-pair. Use the RSA algorithm with a key size of 4096. This size is more than sufficient. Set a validity period of e.g. 302 days. It's important that all subsequent certificates have a lower validity period than the original signer. Leave the other defaults and fill out the Name (addressbook icon). Only the Common Name is required - usually I never use the Email field, as I want to avoid receiving more spam than necessary. Click OK and enter an alias for the Root CA (leave the default). A new password needs to be set for the private key part of the certificate. The new Root CA will now show up in the list of certificates. Creating an Intermediate CA The intermediate CA is signed by the Root CA and used to sign the server specific certificate. This is a common approach as - in case the signing CA for the server specific certificate gets corrupted, not all of the created certificates are affected, but only the ones signed by the corrupted CA. In the list right-click the Root CA and Sign > Sign New Key Pair Use the RSA algorithm with a key size of 4096 again. Ensure the validity period is less than for the Root CA, e.g. 301 days. (Because every second counts) Fill out the Name again - this time as Intermediate Certificate. A Intermediate Certificate is a Certification Authority (CA) in itself. To reflect this, a Basic Constraint must be added via Add Extensions. Mark the checkbox for Subject is a CA Confirm with OK, set the alias (or leave the default) and set the password for the private key part. The new Intermediate CA will now show up in the list of certificates. Creating a Server Specific Certificate In the list right-click the Intermediate CA and Sign > Sign New Key Pair Use the RSA algorithm with a key size of 4096 again. Ensure the validity period is less than for the Root CA, e.g. 300 days. (Because still, every second counts) Fill out the Name again - this time it's important to actually use the public servername, so that browsers can match the server identity with its name in the browser's address bar. If the servername does not match exactly, the certificates and establishing trust will not work! In addition to this, a Subject Alternative Name must be defined - otherwise the latest Chrome versions consider the certificate as invalid. Click on Add Extensions and the + sign, choose Subject Alternative Name. Add a new name with the + sign. Make it the same as the Common Name (CN) above. As before, confirm everything with OK, enter an alias and a password for the private key part. Saving the keystore All certificates should now show with their corresponding properties in the certificate list: Notice the expiry date - the higher up the chain, the sooner the expiry date. If this is not the case - the certificates and establishing trust will not work! Save the keystore via File > Save As and set a password - this time for the keystore itself. As Tomcat has a restriction the password for the keystore must be the password for the private key of the server specific certificate. It's highly recommended and (hopefully) obvious that passwords for the other private keys should be different and that the password for the keystore should be completely different as well. The keystore password will be shared with e.g. Tomcat - the private key password should never be shared with a source you do not trust! Configuring ThingWorx In ThingWorx, configure the <Tomcat>/conf/server.xml as e.g. mentioned in https://support.ptc.com/appserver/cs/view/solution.jsp?n=CS193947 I'm using a connector on port 443 (as it's the default HTTPS port): <Connector port="443" protocol="org.apache.coyote.http11.Http11NioProtocol" maxThreads="150"           SSLEnabled="true" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" enableLookups="false"           connectionTimeout="20000" keystoreFile="C:\ThingWorx\myKeystore.jks" keystorePass="changeme" redirectPort="8443" /> Restart Tomcat to consider the new configuration. ThingWorx can now be called via https://<myserver.com>/Thingworx It's not yet trusted by the client - so it shows as potentially unsafe, and for me it shows in German as well *sorry* However, it's already using encrypted traffic as every HTTPS connection will use TLS. In this case we're communicating in a secure way, but do not know if the server (and its providers) can be trusted or not. Sharing credit card information or other personal details could be dodgy in such a scenario. Trusting the Root CA To trust our Root CA, we first have to export it. Every browser allows to view the certificate, e.g. in Chrome this is done via the Developer Tools > Security As we're using KeyStore Explorer, we can use it to export the Root CA. Right-click it in the certificate list and Export > Export Certificate Chain. Leave the defaults and export as .cer file. Open the Certificate Manager, by typing certmgr into the Windows search field. Open the Trusted Root Certification Authorities and right-click on the Certificates folder. Under All Tasks > Import. Choose the file you've just exported for the Root CA. The Root CA is now a trusted certificate authority. As the ThingWorx server specific certificate will fall back to the Root CA, it's now also recognized by the browser and the connection is now considered as secure and trustworthy. Congratulations!

Do you trust me? if (true) { why } else      { !why } Communication between client and server can be encrypted or not. Communication between client and server can be between trusted parties or not. Both statements are independant of each other. There's no need to trust a server - but still traffic can be encrypted. There's no need to encrypt traffic - but still a server can be trusted. For secure and safe communication in an IoT environment encryption as well as trust should be an integral part of exchanging data. ​ Trust Certificates and Chains of Trust Trust establishes the identity of the parties involved, e.g. is server I'm contacting really the server it pretends to be? This can be ensured with server certificates. On the other hand, client certificates ensure that the client is actual the client it pretends to be. This can be used as an authentication approach and if the identity of the server as well as the client is confirmed, this is also known as mutual authentication. Trust is usually invoked through certificates. Certificates have a subject, usually the servername that the certificate belongs to. This can also be a wildcard subject like *.myserver.com which will be valid for all subdomains of myserver.com. A certificate can be digitally signed to ensure its identity. Usually a Certificate Authority (CA) signs the certificate. This CA can in turn be signed by another CA and so an, creating a whole chain of CAs. This chain is commonly known as the Chain of Trust or the certification path. If a certificate is not signed by a CA it's called a "self-signed certificate". These are mostly used for test scenarios, but should never be used in real production environments. Certificates can also expire by using dates like "Valid From" and "Valid To" to determine if a certificate is valid or not. In case a certificate or CA is expired, all following certificates in the Chain of Trust will no longer be considered as valid. For "commercial" certificates, certificates that are signed by an official CA like Google, Let's Encrypt or VeriSign, a Certificate Signing Request (CSR) is used. For this a key-pair is generated and the CSR is signed with the private key. The CSR contains the public key and is sent to and finally signed by the CA itself, returning the public certificate e.g. signed by VeriSign. This mechanism allows to generate certificates by preparing the main information and just having the CA sign it as a trusted authority. Key-pair All certificates consist of a key-pair - a private key and a public key. The private key is indeed very private to the server and MUST NOT be shared with anyone else. The public key is indeed very public and can be shared with anyone. The key-pair consists of mainly two very large prime numbers and some mathematical magic allows for a mathematical one-way function ​with a trap door. This function allows to en-crypt any message with the public key and to de-crypt this message with the private key. So by giving out the public key to anyone, anyone can en-crypt a message to the server but only the server alone can de-crypt it with its private key. Therefore it's important that the private key is not shared with anyone, otherwise messages can be de-crypted by anyone possessing the private key and secure communication can be compromised. This method is called public-key encryption, or asymmetric encryption (as the key for en- and de-crypting are different). For more information, check out https://en.wikipedia.org/wiki/Public-key_cryptography This key-pair is not related to any encryption while sending data between client and server. Encryption on a transport level is accomplished by using different, random keys. It's just used to generate a unique certificate with a unique fingerprint as a public certificate. The private key part is amongst other things used for signing other certificates in the Chain of Trust. Tomcat needs access to the private key of the server specific certificate, to actually verify that the certificate is authentic. However, certificates can also be used for e.g. emails where the private key is in fact used to en- or decrypt the mail on a content level! ThingWorx ThingWorx is mainly configured for server side trust. In this case all connecting clients can ensure that the identity of the ThingWorx server is a trusted one - and not a fake server or an untrusted source. Certificates and trust are usually triggered when a secure connection is established, via TLS / SSL - e.g. by calling https://<myserver.com>:443/Thingworx The default port for a HTTPS (HTTP Secure / over TLS) connection is 443. For "normal" HTTP configuration, certificates are not required. Keystore To establish trust, any client needs to know the public certificate of the ThingWorx server. This is usually stored in a keystore. In ThingWorx, Tomcat needs to be configured with the location and the password of the keystore containing the certificate. For each web-request Tomcat will then present the server certificate to the client. The client can then verify through its own keystore if the certificate is trusted or not. When operating in browsers, this is done via e.g. the Windows-Keystore. For devices a custom, e.g. a Java-Keystore is required. The keystore must contain the whole Chain of Trust as well as the server specific certificate and its key. If the Chain of Trust is not stored completely in the keystore, or the key is missing / not accessible, the certificate validation process will already fail on server side. Certificates in Windows The default Windows-Keystore is actually pre-filled with lots of Root-Certificates, which start the Chain of Trust - such as Google, VeriSign or Microsoft. Trust needs to start somewhere… If the browser can trace the CAs of certificate back to a trusted Root CA, it's indicated by a green lock and the server will be trusted. If the Root CA cannot be found, it's indicated by a red lock (not a purse) - meaning that the server is not trusted. The server can still be contacted anyway, by manually acknowleding the risks (like identity theft or credit card scams). As an example, here's the certificate for Wikipedia and its Chain of Trust, starting with the GlobalSign Root CA: Trustworthy? In the end, certificates only clarify the identity of a server. If it is trusted or not, is completely up to the user / device. This trustful relationship all comes down to the client keystore / truststore. While the presenting server holds its certificate in the keystore, the verifying client holds the trusted certificates in the truststore. Any certificate in the truststore will ensure the correct server identity. Any unknown certificate will trigger a "Are you sure?" question in the browser. And as devices usually cannot respond to such a question, communication with an untrusted server can be forbidden by default. There are also products, like PTC Navigate which will require a client certificate to not only ensure server identity, but also ensure that also specific clients are able to connect to the server. Encryption No matter if a server is trusted or not, as soon as HTTPS is used as a protocol, communication will be encrypted. The certificate plays a role, but no matter if it's in the client's truststore, TSL forces encryption. Client Hello Usually the client contacts the server and sends a Client Hello. With this it also sends a list of available cipher suites, the intended receiver (server name) and a list of available Signature Hash Algorithms to be used. Cipher suites are used to determine which algorithms are used to establish a connection. The Signature Hash Algorithms define the Hash function, like SHA256 and the key / signature, like RSA. In my environment usually the Elliptic Curve Diffie-Hellman key exchange is used to transfer public keys between server and client. During the Client Hello information about the elliptic curves are also transferred. For more information on these topics, see https://en.wikipedia.org/wiki/SHA-2 https://en.wikipedia.org/wiki/RSA_(cryptosystem) https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange https://www.youtube.com/watch?v=YEBfamv-_do (Diffie-Hellman explained) https://en.wikipedia.org/wiki/Elliptic_curve_cryptography Server Hello After receiving the Client Hello, the server sends back the Server Hello. This consists of multiple handshakes. In this phase, the server sends the preferred cipher suite (intersection of client and server suites) including the Signature Hash Algorithm to be used. In the same step the server is sending off the certificate to the client for identity validation. For establishing trust and validating the certificate, the server needs to know the private key for the server specific certificate. The server also announces in the server key exchange what public key it's using for encryption to the client. This key is not related to the certificate, but a random number from a newly generated key-pair. Keys are exchanged between server and client and vice versa via the Diffie-Hellman key exchange. Client Response The client now sends in the client key exchange what public key it's using for encryption to the server. This is also a random key transmitted via the Diffie-Hellman key exchange. Initiating Communication After announcing the public keys, both server and client will now communicate encrypted by using their custom key-pairs for asymmetric encryption of the payloads. This can also include an additional change of the cipher specification. As the public keys have been exchanged via the Diffie-Hellman key exchange, server and client are now bi-directionally communicating using each other's public key to en-crypt a message and each's own private key to de-crypt the message again. For more information, also see https://tools.ietf.org/html/rfc5246 Trust & Encryption - Hands On Do you trust me now? if (true) { why } else      { !why } To see how this all works in a ThingWorx enviroment, and how easy it actually set up what has been discussed above, see also Trust &amp; Encryption - Hands On

This script is provided as is and is not officially supported by PTC. Users are welcome to edit or change this script as desired. This script automates the installation steps outlined in the ThingWorx Installation Guide. If you are not familiar with these processes or are installing ThingWorx for the first time, it is recommended to refer to the guide for additional information. Requirements:   64-bit Linux: Ubuntu (all flavors) 14/15/16, Red Hat Enterprise Linux 6/7, or CentOS 6/7   Root privileges   Internet connection   A ThingWorx platform installation file for either H2 or PostgreSQL (available in the PTC downloads center https://support.ptc.com/appserver/cs/software_update/swupdate.jsp )   *NOTE: This script does not require a GUI and can be run in headless mode Minimal download option: The script will look in the working directory for pre-downloaded Java, Tomcat, and (for RHEL and CentOS) PostgreSQL installation files. If the installer detects one of the supported files, the user will be given the option to install with that file and skip the automated download. This feature is limited to specific versions and requires one (or more) of the following files:     jdk-8u92-linux-x64.tar.gz     apache-tomcat-8.0.33.tar.gz     pgdg-redhat94-9.4-3.noarch.rpm     NOTE: This reduces the amount of data downloaded by the script, but does not eliminate it entirely. An active internet connection is still required. Support for existing SSL certificates: If an existing SSL certificate exists and the user wishes to use it instead of automatically generating one with the script, the user can copy the '.keystore' file to the working directory before running the script. During the installation process the user will be prompted with the option to use this keystore file, and will need to provide the keystore password created with the certificate.     NOTE: this script does not support keystore passwords containing the following characters: '!/@\"#$%^&*()_+]    *Note for Red Hat Enterprise Linux users*     This script was designed to work on servers with an active Red Hat subscription. It has been tested and appears to work on unsubscribed servers if the following applications are installed: gcc make epel repository (see This Red Hat Announcement for more information) Download the file attached to this document Steps to use this script: 1. Create a new folder on your Linux server. (This will be referred to as the 'working folder') 2. Download the ThingWorx platform installer file from the PTC Software downloads page. (For example, MED-61111-CD-072_SP2_ThingWorx-Platform-Postgres-7-2-2.zip) 3. Copy the installer file to the working folder. Do not unzip. 4. Copy the installer.sh file to your working folder. 5. In the command prompt, cd to your working directory. 6. Make the installer.sh file excutable:   > chmod +x installer.sh 7. Run the installer.sh file either using sudo or logging in as root   > sudo bash installer.sh   OR   > su   > bash installer.sh   *NOTE: The 'bash' command is mandatory, running the installer with the 'sh' command will not work

Unless created and owned by the Administrator user, by default MySQL Database Thing will not connect to the database as it requires certain permissions on the user. In order for a user other than an administrator to create a working database thing, they need three permissions (in addition to the typical subsystem and resource permissions - refer to https://www.youtube.com/watch?v=HzFqxvgHtpI&index=8&list=PLz1ppcU_kaneagUT9qgQfz3HByf6-9zTF ​ ):: Visibility to the Database Thing Template. Execute service permission on the EncryptPropertyValue service in the Encryption Services resource. Visibility to the DatabaseThing Thing Package. Typically to track down permissions issue, the most convenient and easy way is to use browser developer tools. For example in Chrome, developer tools can be used to view the API calls being sent by Composer, and the errors sent in response.   ThingWorx Composer doesn’t expose Thing Packages, so in order to set visibility to the DatabaseThing Thing Package, one would need to throw a REST API call at it. Hope this information helps in setting up a non-administrator own MSSQL database thing! *In addition refer to The use of System User

Wanting to build a simple environment with ThingWorx, using a Raspbery Pi mini-computer and an Edge MicroServer (EMS), one has to bear in mind some apparently insignificant tips, that can make a huge difference in a well-connected platform.    The Raspbery Pi can have either a few sensors attached on it and/or a Sense HAT integrated with it. A Sense HAT is an add-on board for Raspbery Pi that has an 8x8 LED matrix, a five-button joystick and a sensor, gyroscope. Tips: The ThingWorx guide "Connecting ThingWorx EMS with Rasberry Pi Quickstart" can be expanded to read Sense HAT specific sensor information A Sense HAT integrated with Raspberry Pi can be accessed from anywhere, using the existing APIs to read/write Getting location information while using the Pi board (because The Sense HAT doesn't have this feature): The Raspberry Pi has several USB ports, therefore inserting a USB GPS can be used as a workaround for connecting and reading location information (or the GPIO pins could be used to connect a GPS board) To include services and properties to read Sense HAT sensors data, start from the existing LUA example.lua template and built a Pi specific one. The EMS is a pre-built application that can be installed on Windows or Linux and it provides a bridge between the remote device and the ThingWorx platform, using the REST API AlwaysOn protocol. Tips: To read data from a sensor, it is important to use the corresponding library for that sensor; if LUA scripting is used, then this library has to be easily integrated with LUA (LSR is optional with EMS; EMS can be used standalone). Using the correct library, the EMS will be able to gather the data and push it to ThingWorx. The LSR sends properties to the EMS over the HTTP(s) protocol, which converts it to the AlwaysOn Protocol, and sends it to the Platform LUA is a process manager useful if having to integrate devices from different vendors using different modes to interact To test if the LSR is working: go to Composer -> Properties tab of the remote thing, click "Manage Bindings", select the "Remote" tab, next to "Local" - here can be seen all the remote properties added to the thing by the LSR. For creating and biding more things: create the remote thing in ThingWorx and then add them to the config.lua file in the same way you have the gateway Listing any "Remote Things" that are ready to be connected, and are not associated with any Thing, yet: Monitoring; one method to try to bind them: type the name of the thing in the Identifier section of General Information of the Thing To request a local REST API call to the default EMS port 8000, from a LSR, make sure that the GET works fine (i.e for a property update:  localhost:8000/Thingworx/Things/MyThing/Properties/myString It should return you something like this: {"rows":[{"myString":""}],"fieldDefinitions":{"myString":{"name":"myString","description":"","type":"STRING","aspects":{}}}}

This is part 3 out of 3 videos on Getting Started with ThingWorx Analytics During this video you will learn:   Executing a “Signals” Job Getting the results of the “Signals” Job Executing a “Training Model” Job Getting the results of the “Training Model” Job   Updated Link for access to this video:  Getting Started with ThingWorx Analytics: Part 3 of 3

In this blog I will be covering the initial setup of ThingWorx Android SDK with a sample app (supplied with the Android SDK) and set it up with a web based revision control system like Bitbucket's (free account plan). I'll also be covering quick information on how to enable the HTTPS connection for the ThingWorx server on Windows platform. This will allow for secured connection to the ThingWorx server from the Android application. Do note this is only a reference guide for setting it up with revision control system, you’re free to choose to setup the Android Studio without Bitbucket or with any other revision control system. It’ll be just fine to have a local Git/SVN/Perforce etc. to manage the code repository, setting Android Studio with Bitbucket is officially not supported. Pre-requisite: 1. Download and unzip the ThingWorx Android SDK from https://support.ptc.com 2. An account with Bitbucket is required 3. Download and install Android Studio Project Setup 1. Unzip the downloaded Android SDK on a local drive 2. Start the Android Studio > Import project > navigate to the sample applications location provided with the ThingWorx Android SDK e.g. Thingworx-Android-SDK-X.x.x\samples 3. Select one of the sample application e.g. androidShell, with ThingWorx Android SDK X.x.x there are 3 sample applications currently available when the Android SDK is downloaded: a. Android File Brwoser b. Android Shell c. Android Steam Thing 4. For this blog I'll be setting up the androidShell Android Application 5. Do note that all the sample projects are built using Gradle, so while importing if required select Gradle as the build system for the sample application 6. Once imported successfully in Android Studio you should be able to see the Android project and its file structure like so 7. We'll need an account with Bitbucket, create one if you don't have already 8. Logon to Bitbucket and create a team and new repository under that team 9. Navigate to the repository created in Bitbucket 10. Create a local GIT code repo if you don't already have one, and copy over all the contents from the Android SDK x.x.x.zip to that location 11. On your local machine open a command prompt and navigate to the drive where the local GIT's code repository resides, i.e. the folder where you unzipped the android SDK and execute the commands as mentioned: a. git remote add origin https://<accountName@bitbucket.org>/<teamName>/<projectName>.git b. git push -u origin master Note: This will add the contents of your local GIT repository to the empty code repository you’ve created under the team on Bitbucket. 12. You can also use SourceTree UI application on windows for creating, GIT and Mercury based code repository and connect it to your Bitbucket account 13. On successful commit following will be logged in the command prompt Enabling HTTPS on Tomcat and connecting to Android Application Securing Tomcat on Windows You can skip this section if you already have Tomcat running ThingWorx configured for HTTPS connection. 1. Execute the command in a command prompt C:\>"%JAVA_HOME%\bin\keytool" -genkey -alias tomcat -keyalg RSA -keystore C:\KeystoreTomcat\tomcat.keystore Note: Executing above step will require you to add additional information to the keystore like, Org name, full name location, etc. 2. Edit the <tomcatInstallation>\conf\server.xml file   <Connector   port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"               maxThreads="200" SSLEnabled="true" scheme="https" secure="true" keystoreFile="C:\KeystoreTomcat\tomcat.keystore" keystorePass="<giveYourPasswordHere>" clientAuth="false" sslProtocol="TLS">       <!-- <SSLHostConfig>             <Certificate certificateKeystoreFile="conf/localhost-rsa.jks"                         type="RSA" />         </SSLHostConfig>-->     </Connector> 3. Restart the tomcat service Note: For detailed information on securing the Tomcat on Windows refer to SSL/TLS Configuration How-TO . Note HTTPS setup is only for the Tomcat where ThingWorx is deployed and does not involve certificate setup on the Android Application side Finally,to test if the HTTPS setup was successful or not navigate to https://<serverName/IP>:8443/Thingworx in a web browser. Port 8443 is the default HTTPS port. Starting and connecting Android Application to ThingWorx Now that we have a working secured Tomcat and Android Studio setup with the sample Android Application, androidShell. Let's build and run the application using an emulated Android device in Android Studio 1. Navigate to https://localhost:8443/Thingworx > Import/Export > Import from file and import the Thing entity which will connect to the SampleThing when you’ll run the Android application, for e.g. I imported the Thingworx-Android-SDK-X.X.X\samples\android-shell\entity\Things_AndroidSampleThing.xml because I will be running the androidShell application 2. Attempting to run the Android application without the entity created in ThingWorx first, Thing will show as unbound in RemoteThing. 3. Create an AppKey in ThingWorx > Security > Application Keys with sufficient rights 4. Go to the Android Studio's toolbar and click on AVD Manager icon 5. This will open the Android Virtual Device Manager and lets you create a virtual Android device 6. You can of course use your own actual device to connect over USB and install and test the application on actual hardware 7. If you already have a Virtual or actual device connected to the system, click on Make Project icon in the toolbar 8. Once the Make finishes run the sample application, in my case androidShell application with the Run icon, like so 9. You may now be prompted with the options to select a device virtual or actual 10. Select as desired and click OK 11. This will now launch the application on the selected device, I have selected to launch on the virtual device which will start and emulated Android Device 12. When initiating/running the application for the first time you will be directed to the Settings screen allowing you to enter the connection URI and the Application Key to connect to the ThingWorx server 13. You have to follow one of the following two URI schemes while attempting to connect to a ThingWorx Server a. For HTTP connection use : ws://<machineIP/Name>:8080/Thingworx/WS b. For HTTPS connection use : wss://<machineIP/Name>:8443/Thingworx/WS Note: Ports may differ as these are the default ports, if you are running ThingWorx on different port enhance the URI accordingly 14. Since my ThingWorx is reachable on HTTPS connection i'll use the HTTPS connectino URI scheme and the application key that I have already created in the ThingWorx, which is bound to Administrator user 15. Once done press the back button on the screen to initiate the connection attempt 16. If all's set as it should be you will be able to see the Connected to Server option checked and the Property count for the Serial Number Property being updated every second For more detail on ThingWorx Android SDK refer to the ThingWorx Edge SDKs and WebScocket based Edge MicroServer (WS EMS) Help Center

This blog addresses a few points that are related to scoring with ThingWorx Analytics. It, particularly, brings a clearer understanding of the concepts behind the values of the scores that are generated when performing a scoring job.   Scoring Outputs:   It is important to note that when training an analytics model, the method is to create a generalizable model from a relatively small training dataset.   By its nature, we expect the training process to see a limited subset and not an exhaustive list of all possible values for many constraints, especially for time and practicality.   As such, these generalized models will be expected to handle unseen data in the form of new combinations or values outside of previously observed ranges (more on this below).   One common way to see scores that exceed the observed ranges in training, under the assumption that the goals are continuous, is to use prescriptive scoring.   Prescriptive scoring attempts to find optimal values for a lever, meaning tunable, features in order to maximize or minimize score values. See the prescriptive scoring documentation and functionality for more information.   min/max constraints: these are constraints that are placed upon the inputs for training and expected inputs for scoring.   •          For training: If theses ranges were provided as part of the upload process, then training will raise exceptions regarding invalid data. However, if the ranges are not provided - they will be inferred from the data and, as such, training will not see values outside of observed ranges.   •          For scoring: Validation of the ranges will only be performed on the inputs - not the outputs. It is very important to note that the handling of these "constraints" is dependent upon the data type.   For categorical (e.g. colors) and ordinal data (e.g. shirt sizes), the constraints are strict and data that was not observed in training will raise exceptions during scoring.   However, for continuous values (e.g. temperature ranges) these constraints are more informational in nature. For predictive scoring, our code will accept records with values outside of those ranges.   The rule of thumb is that values slightly outside these ranges are acceptable and that as the values stray farther from the ranges, the accuracy of the model degrades very quickly.   For prescriptive scoring, these constraints are used to determine the acceptable ranges of values to try when determining the optimal values. Values outside of these constraints will NOT be tried.

A Feature - a piece of information that is potentially useful for prediction. Any attribute could be a feature, as long as it is useful to the model. Feature engineering – Feature engineering is the process of transforming raw data into features that better represent the underlying problem to the predictive models, resulting in improved model accuracy on unseen data. It’s a vaguely agreed space of tasks related to designing feature sets for Machine Learning applications. Components: First, understanding the properties of the task you’re trying to solve and how they might interact with the strengths and limitations of the model you are going to use. Second, experimental work were you test your expectations and find out what actually works and what doesn’t. Feature engineering as a technique, has three sub categories of techniques: Feature selection, Dimension reduction and Feature generation. Feature Selection: Sometimes called feature ranking or feature importance, this is the process of ranking the attributes by their value to predictive ability of a model. Algorithms such as decision trees automatically rank the attributes in the data set. The top few nodes in a decision tree are considered the most important features from a predictive stand point. As a part of a process, feature selection using entropy based methods like decision trees can be employed to filter out less valuable attributes before feeding the reduced dataset to another modeling algorithm. Regression type models usually employ methods such as forward selection or backward elimination to select the final set of attributes for a model. For example: Project Development decision-tree:                                                  Dimension Reduction: This is sometimes called feature extraction. The most classic example of dimension reduction is principle component analysis or PCA. PCA allows us to combine existing attributes into a new data frame consisting of a much reduced number of attributes by utilizing the variance in the data. The attributes which "explain" the highest amount of variance in the data form the first few principal components and we can ignore the rest of the attributes if data dimensionality is a problem from a computational standpoint. Feature Generation or Feature Construction: Quite simply, this is the process of manually constructing new attributes from raw data. It involves intelligently combining or splitting existing raw attributes into new one which have a higher predictive power. For example a date stamp may be used to generate 2 new attributes such as AM and PM which may be useful in discriminating whether day or night has a higher propensity to influence the response variable. Feature construction is essentially a data transformation process. Tips for Better Feature Engineering Tip 1: Think about inputs you can create by rolling up existing data fields to a higher/broader level or category. As an example, a person’s title can be categorized into strategic or tactical. Those with titles of “VP” and above can be coded as strategic. Those with titles “Director” and below become tactical. Strategic contacts are those that make high-level budgeting and strategic decisions for a company. Tactical are those in the trenches doing day-to-day work.  Other roll-up examples include: Collating several industries into a higher-level industry: Collate oil and gas companies with utility companies, for instance, and call it the energy industry, or fold high tech and telecommunications industries into a single area called “technology.” Defining “large” companies as those that make $1 billion or more and “small” companies as those that make less than $1 billion.   Tip 2: Think about ways to drill down into more detail in a single field. As an example, a contact within a company may respond to marketing campaigns, and you may have information about his or her number of responses. Drilling down, we can ask how many of these responses occurred in the past two weeks, one to three months, or more than six months in the past. This creates three additional binary (yes=1/no=0) data fields for a model. Other drill-down examples include: Cadence: Number of days between consecutive marketing responses by a contact: 1–7, 8–14, 15–21, 21+ Multiple responses on same day flag (multiple responses = 1, otherwise =0) Tip 3: Split data into separate categories also called bins. For example, annual revenue for companies in your database may range from $50 million (M) to over $1 billion (B). Split the revenue into sequential bins: $50–$200M, $201–$500M, $501M–$1B, and $1B+. Whenever a company falls with the revenue bin it receives a one; otherwise the value is zero. There are now four new data fields created from the annual revenue field. Other examples are: Number of marketing responses by contact: 1–5, 6–10, 10+ Number of employees in company: 1–100, 101–500, 502–1,000, 1,001–5,000, 5,000+ Tip 4: Think about ways to combine existing data fields into new ones. As an example, you may want to create a flag (0/1) that identifies whether someone is a VP or higher and has more than 10 years of experience. Other examples of combining fields include: Title of director or below and in a company with less than 500 employees Public company and located in the Midwestern United States You can even multiply, divide, add, or subtract one data field by another to create a new input. Tip 5: Don’t reinvent the wheel – use variables that others have already fashioned. Tip 6: Think about the problem at hand and be creative. Don’t worry about creating too many variables at first, just let the brainstorming flow.

Hi,   I have just been playing around with the Edge MicroServer on a couple of Raspberry Pi's.  I obviously want them connected to ThingWorx, however they stop when the SSH session closes which isn't ideal.  I thought about doing something really quick and dirty using 'nohup', but this could have lead to many running processes, and wouldn't still not have started automatically when the Pi booted.  So, I did it right instead using init.d daemon configurations.   There are two files; one for the EMS and one for the Lua Script Resource.  You need to put these in /etc/init.d, then make sure they are owned by root.   sudo chown root /etc/init.d/thingworx* sudo chgrp root /etc/init.d/thingworx*   You'll need to modify the paths in the first lines of these files to match where you have your microserver folder.  Then you need to update the init.d daemon configs and enable the services.   sudo update-rc.d thingworx-ems defaults sudo update-rc.d thingworx-ems enable sudo update-rc.d thingworx-lsr defaults sudo update-rc.d thingworx-lsr enable   You can then start (stop, etc.) like this:   sudo service thingworx-ems start sudo service thingworx-lsr start   They should both also start automatically after a reboot.   Regards,   Greg

Merging InfoTables can seem to be an overwhelming task in ThingWorx. What even IS an InfoTable?? The short answer is that an InfoTable is a specifically structured JSON object. Merging InfoTables is therefore as easy as incrementing through each, adding the columns of each to another table, and then populating the new table with data. For InfoTables with the same DataShapes, the built in "Union" service can be used. Find this snippet under the InfoTableResources section. There is also a snippet for incrementing through DataShape fields if the DataShape is known. For InfoTables which don't have the same DataShapes or for which the DataShape is not known, things get a bit trickier. Thankfully, some new KCS documentation provides example code to step you through merging any number of tables together. I hope this documentation is helpful!

As per ThingWorx Documentation: Updating Properties Automatically in a Mashup A mashup using the GetProperties service can be configured to use websockets and receive updates to properties automatically. When creating or editing a mashup, you can configure the GetProperties service so that the properties are automatically updated by selecting the Automatically update values when able checkbox in the service properties panel. So, the feature to update Properties Automatically in a Mashup is limited to GetProperties service only. Following are the steps to invoke our own custom Service automatically when a property change: 1. Find all the Properties in your Thing for which the DataChange should trigger the custom service. 2. In mashup; add value display widget (or some other widgets) for each property in Step1. 3. Bind the properties from GetProperties service to these widget. 4. Set visible property of these widgets to false so that they don't show up at the RunTime. 5. Now bind the ServiceInvokeCompleted event of GetProperties Service to your custom Service. 6. Save and view Mashup. Result: When any of the Property from Step1 is changed; Custom Service will be invoked in our Mashup.