IoT & Connectivity Tips

  Hi everyone,   In case you’re looking for more reasons to appreciate the power of Azure, today we’re answering 10 frequently asked questions around how and why to use Azure SQL with ThingWorx.   You likely already know that we support multiple persistence providers, like Azure SQL, InfluxDB, H2, MSSQL and PostgreSQL, for you to store and persist your ThingWorx data. Here’s an up-close-and-personal look into why we recommend Azure SQL.   1. What is Azure SQL?         Azure SQL is a relational database hosted in the Azure cloud and is a fully managed Platform as a Service (PaaS) Database Engine. Azure SQL Database engine is based on the Enterprise Edition of SQL Server. The Azure platform fully manages every Azure SQL Database with a high percentage of data availability and guarantees no data loss. Azure SQL Database comes with built-in high availability, disaster recovery, and upgrade for the database. Refer to Microsoft's Azure SQL Database - Platform as a Service documentation for more information on Azure SQL Database and its features.   2. What are the top 3 reasons to use Azure SQL with ThingWorx?   Ease of Use and Management: Azure SQL greatly reduces the need to manage database resources for ThingWorx. It helps to reduce your total cost of ownership for managing database resources for ThingWorx by managing virtual machines, operating system, database software, upgrades, high availability, and backups for you, so you can focus on building your IoT solution. It provides unmatched scale and high availability for compute and storage without sacrificing performance. With Azure SQL, you can scale your application on demand with up to 99.95% availability.   Hybrid Deployments: ThingWorx supports multiple persistence providers to store IoT data for different use cases. Please refer to the ThingWorx Model and Data Best Practices Guide to learn more. If you’re already using Microsoft SQL Server with ThingWorx on premise, then you can use Azure SQL for your cloud deployments of ThingWorx-based IIoT solutions in hybrid scenarios. This allows you to reduce development time—develop once and deploy anywhere through a common programming surface area across Azure SQL (on cloud) and SQL Server (on premise). You can leverage ThingWorx federation to run ThingWorx in different deployment topologies.   If needed, you can also accelerate your on-premise SQL Server migrations without changing the application code by leveraging Managed Instance. Use the Azure Hybrid Benefit Savings Calculator to calculate your TCO. Enjoy additional deployment flexibility with Single Database for SQL applications created in the cloud or Elastic Pool for multi-tenant applications.   Security and Compliance: Azure SQL Database meets the most stringent compliance standards with built-in auditing and information protection technology. With its availability in different regions, its best suited for Government cloud and sovereign cloud. Please see this link to check for the latest update on Azure product availability by region. You can also get multi-layered security provided by Microsoft across physical datacenters, infrastructure, and operations and will always have the latest SQL Server capabilities in the cloud, with no patching or upgrading. It also offers protection to your databases from malicious acts with fine-grained access controls, Always Encrypted technology, and advanced threat protection capabilities.   3. How do I configure ThingWorx for Azure SQL? From ThingWorx Foundation platform version 8.4 release onwards, ThingWorx provides you an option to choose Azure SQL as a persistence provider to store your value stream, stream, and data table data. This Help Center provides all the details and steps to help you set up Azure SQL with ThingWorx.    You can run ThingWorx with Azure SQL either by downloading the ThingWorx Azure SQL .WAR file or by running it as containerized ThingWorx Docker images by downloading ThingWorx Dockerfiles. For reference, see the below image to help you download ThingWorx 8.4 artifacts.   Here’s a video demonstrating how to install ThingWorx. (view in My Videos)   Here’s a second video that walks you through configuring ThingWorx with Azure SQL. (view in My Videos)   4. Which versions of Azure SQL does ThingWorx support? Consult the latest system requirements guide here to learn which versions of Azure SQL ThingWorx supports.   5. What database deployment options do I have? In Azure, you can have your SQL Server workloads running in a hosted infrastructure (IaaS) or running as a hosted service (PaaS). Within PaaS, you have multiple deployment options and service tiers within each deployment option, such as Single Database, Elastic Pool sets, and managed Instance. ThingWorx supports all the deployment options to setup Azure SQL as a persistence provider. You can refer to this link on Azure SQL Database versus SQL Server to help you choose an option that works best for your business needs.   6. Why would I want to use an PaaS database? Service tools and built-in features enable a more streamlined and automated means of controlling and operating your database. The need for constant manual control and tweaking of information, recovery tools, compliance and updates is now configured and built into Azure SQL for a more hands-off approach to your storage database. Here is a table to inform you on how Azure SQL PaaS helps.   7. Which features are new to Azure SQL 2019? Azure SQL now offers Always Encrypted data transfer through TLS and auto-failover for managed instance deployment to enable transparent and coordinated failover of multiple databases. Azure SQL also offers a data migration assistant, which detects compatibility issues that can impact functionality when upgrading your database. For more information on features and functionality, see Microsoft SQL documentation or Azure SQL’s latest release notes.   8. Is there any guidance available to help me migrate to Azure SQL? Yes! Microsoft’s Database Migration Service enables seamless migration to Managed Instance with downtime measured in minutes. The process is highly automated and risk-free while streamlining the transition of SQL Server and on-Microsoft database systems such as Oracle to Azure SQL Database. You can learn more about upgrading to Azure SQL here.   9. What purchasing models are available to me?   vCore based (recommended) - For customers that prioritize flexibility and control, this model offers scaling of compute, storage, and I/O resources independently to optimize price based on need. The customer chooses the hardware and service tier based on high-availability design, storage type, fault-isolation methods, and I/O ranges.   DTU based - Three distinct available tiers are differentiated based directly upon compute, memory, and I/O resources. This model bundles the measures together for customers who want pre-configured or simplified resource options. You can refer to more pricing and purchase options here.   10. What should I do if I need technical support? If you select Azure SQL as your persistence provider, then all support requests related to configuring Azure SQL can be logged through PTC Technical Support at https://support.ptc.com or by calling 1-800-477-6435.   You may also want to use the PTC Community to learn and collaborate with the growing PTC developer community. For all other requests related to database management, troubleshooting, monitoring, and administration, we encourage you to reach out to Microsoft directly.   Let me know what you think in the comments below.   Stay connected, Kaya

Please find here an Labview implementation to connect to Thingworx via RestCalls. Have Fun using it. Any Feedback is appreciated. https://github.com/Seppel1985/LabVIEW_TWX_RestAPI

Applicable releases: 8.3 and above   Description: In this video, we will see  Basic concepts of SVM Some scenarios for a better understanding

Database backups are vital when it comes to ensure data integrity and data safety. PostgreSQL offers simple solutions to generate backups of the existing ThingWorx database instance and recover them when needed.   Please note that this does not replace a proper and well-defined disaster recovery plan. Export and Import are part of this strategy, but are not reflecting the complete strategy. The commands used in this post are for Windows, but can be adjusted to work on Linux-based systems as well.   Backup   To create a Backup, the export / dump functionality of PostgreSQL can be used.     pg_dump -U postgres -C thingworx > thingworxDump.sql     The -C option will include the statement to create the database in the .sql file and map it to the existing tablespace and user (e.g. 'thingworx' and 'twadmin'). The tablespace and user can be seen in the .sql file in the line with "ALTER DATABASE <dbname> OWNER TO <user>;"   In the above example, we're backing up the thingworx database schema and dump it into the thingworxDump.sql file   Tablespace, username & password are also included in the platform_settings.json   Restore   To restore the database, we just assume an empty PostgreSQL installation. We need to create the DB schema user first via the following commandline:     psql -U postgres -c "CREATE USER twadmin WITH PASSWORD 'ts';"     With the user created, we can now re-generate the tablespace and grant permissions to the twadmin user:   psql -U postgres -c "CREATE TABLESPACE thingworx OWNER twadmin LOCATION 'C:\ThingWorx\ThingworxPostgresqlStorage';" psql -U postgres -c "GRANT ALL PRIVILEGES ON TABLESPACE thingworx TO twadmin;" Finally the database itself can be restored by using the following commandline:   psql -U postgres < thingworxDump.sql This will create the database and populate it with tables, functions and sequences and will also restore the data from the .sql file.   It is important to have database, username and tablespace match with the original system - otherwise granting permissions and re-creating data might fail on recovery. User and tablespace can also be reused, so only the database has to be deleted before restoring it.   Part of the strategy   Part of the backup and recovery strategy should be to actually test the backup as well as the recovery part of the procedure. It should be well tested on a test-environment before deployed to any production environments. Take backups on a regular basis and test for disaster recovery once or twice a year, to ensure the procedure is still valid.   Data is the most important source in your application - protect it!  

Help Center link on how to control file transfers from the edge client using the EdgeControlled ThingShape.   The EdgeControlled ThingShape is a default entity included with ThingWorx that allows you to manage the amount of egress being sent from the platform to the Edge.   At the time of writing this post, the available 'When Disconnected' settings for a remotely bound property in ThingWorx are 'Fold' and 'Ignore'. Setting a property to 'Fold' while using this EdgeControlled ThingShape is necessary whether the device is connected all the time or only for brief updates.   To use this ThingShape in a real world scenario you might code your edge client to invoke the DequeueEgress REST API function available through this ThingShape. The parameter you pass in is then the number of messages you would like the client to receive. The result of this function is how many messages the platform then actually sent.   A quick setup: 1. Create a RemoteThing entity in ThingWorx 2. Create an ApplicationKey entity in ThingWorx 3. Setup an edge client to bind to that RemoteThing using the specified ApplicationKey 4. Manage Bindings on the Properties page of the RemoteThing, and pull in a few properties you would like to send property updates to 5. Set the 'When Disconnected' value to 'Fold' for each property you want to queue messages for   5a. Set any other settings on the properties you'd like; ie. persistence, logged 6. Save the Thing 7. Add the EdgeControlled ThingShape to the Thing 8. Save the Thing 9. Update property values, see exceptions thrown, but the value will be queued 10. Invoke DequeueEgress on the RemoteThing, with the number of messages to send to the edge client passed in as the parameter value   10a. Notice 'Fold' means only the last value set for a property will be sent to the edge client. There is currently no retention available for any values previously set to the property and stored as the message to be sent. Those values are lost upon a new value coming in before it's dequeued. 11. Verify the edge client has received the expected egress, and the return result of the DequeueEgress function was the expected # of messages sent.

Continuing our series of Troubleshooting ThingWorx Analytics installations, in this IoT Tech Tip we will cover two items have been appearing for many users.   Error 1069 Encountered with Native Windows Installation of ThingWorx Analytics 8.2   In some instances, when a user successfully installs ThingWorx Analytics (TWAS) to a Windows Server operating system, they will encounter an error where TWAS will report an Error 1069: The Service did not start due to logon failure.   This can occur with any individual Service that is created by the installation, the following fix should work in addressing the issue.   Primary Reason This Happens:   This error can be encountered when the user provides incorrect credentials for associating the Services to an account during installation. In TWAS 8.2, there is a utility that will enable to the user to change the associated user on the Services. It is important the user provides the password for the User Account on Windows, and not the user/password combination for ThingWorx Foundation Platform Server.   Steps to Fix Issue   Solution 1:   Open a Command Prompt as Administrator, via Start Menu à Run à type CMD. Then right click on cmd.exe and Run As Administrator.   In the elevated command prompt, change your directory to the ThingWorxAnalyticsServer/bin directory, for example in the default installation path would be: cd C:\Program Files (x86)/ThingWorxAnalyticsServer/bin Then execute the changeServiceUserAccount.bat <username>, for example: changeServiceUserAccount.bat user1   You will be prompted to change the password for the user.   Solution 2:   If Solution 1 does not resolve the issue, alternately you can manually change the Log On properties for each of the services. The changeServiceUserAccount.bat would do this via script, but on occasion this may work. Open the Control Panel and navigate to Services, for example: Control Panel à All Control Panel Items à Administrative Tools   You will have to right click each individual service and go to Properties à Log On tab and enter the account name and password for the local account. Note: Local System account will not resolve this issue.   This issue was resolved in the ThingWorx Analytics Server 8.3 release, where all Services are associated with the Network Service account.     More information can be found in this Knowledge Article   Uploading of a Dataset hangs or does not complete in ThingWorx Analytics 8.3   On occasion, after a fresh installation of ThingWorx Analytics Server 8.3 on a Windows Server operating system, a dataset will not complete its upload. Typically no error message is displayed, and the upload wizard UI will just hang on the upload progress after:   Creating copy of Configuration File... Submitting Create Dataset request... Creating copy of Data File...   Primary Reason This Happens:   This is caused by twas-zookeeper service being stuck in a PAUSED state. This means that in the post installation, twas-zookeeper did not start.   Steps to Fix Issue   You will have to double check that the JAVA_HOME variable was defined as a System Variable. In the ThingWorx Analytics Installation guide, pages 12-14 outline the steps required as pre-requisites. You can change this in Control Panel > System > Advanced Settings > Environment Variables, and ass a new variable named JAVA_HOME under System Variables. The value location should be the location of your deployment of JAVA software.   Typically this is located in C:\Program Files\Java\<jre or jdk>_<version number>     More information can be found in this Knowledge Article

This is a follow-up post on my initial document about Edge Microserver (EMS) and Lua Script Resource (LSR) security. While the first part deals with fundamentals on secure configurations, this second part will give some more practical tips and tricks on how to implement these security measurements.   For more information it's also recommended to read through the Setting Up Secure Communications for WS EMS and LSR chapter in the ThingWorx Help Center. See also Trust & Encryption Theory and Hands On for more information and examples - especially around the concept of the Chain of Trust, which will be an important factor for this post as well.   In this post I will only reference the High Security options for both, the EMS and the LSR. Note that all commands and directories are Linux based - Windows equivalents might slightly differ.   Note - some of the configuration options are color coded for easy recognition: LSR resources / EMS resources   Password Encryption   It's recommended to encrypt all passwords and keys, so that they are not stored as cleartext in the config.lua / config.json files.   And of course it's also recommended, to use a more meaningful password than what I use as an example - which also means: do not use any password I mentioned here for your systems, they might too easy to guess now 🙂   The luaScriptResource script can be used for encryption:   ./luaScriptResource -encrypt "pword123" ############ Encrypted String AES:A26fBYKHJq+eMu0Fm2FlDw== ############   The wsems script can be used for encryption:   ./wsems -encrypt "pword123" ############ Encrypted String AES:A26fBYKHJq+eMu0Fm2FlDw== ############   Note that the encryption for both scripts will result in the same encrypted string. This means, either the wsems or luaScriptResource scripts can be used to retrieve the same results.   The string to encrypt can be provided with or without quotation marks. It is however recommended to quote the string, especially when the string contains blanks or spaces. Otherwise unexpected results might occur as blanks will be considered as delimiter symbols.   LSR Configuration   In the config.lua there are two sections to be configured:   scripts.script_resource which deals with the configuration of the LSR itself scripts.rap which deals with the connection to the EMS   HTTP Server Authentication   HTTP Server Authentication will require a username and password for accessing the LSR REST API.     scripts.script_resource_authenticate = true scripts.script_resource_userid = "luauser" scripts.script_resource_password = "pword123"     The password should be encrypted (see above) and the configuration should then be updated to   scripts.script_resource_password = "AES:A26fBYKHJq+eMu0Fm2FlDw=="   HTTP Server TLS Configuration   Configuration   HTTP Server TLS configuration will enable TLS and https for secure and encrypted communication channels from and to the LSR. To enable TLS and https, the following configuration is required:     scripts.script_resource_ssl = true scripts.script_resource_certificate_chain = "/pathToLSR/lsrcertificate.pem" scripts.script_resource_private_key = "/pathToLSR/key.pem" scripts.script_resource_passphrase = "keyForLSR"     It's also encouraged to not use the default certificate, but custom certificates instead. To explicitly set this, the following configuration can be added:     scripts.script_resource_use_default_certificate = false     Certificates, keys and encryption   The passphrase for the private key should be encrypted (see above) and the configuration should then be updated to     scripts.script_resource_passphrase = "AES:A+Uv/xvRWENWUzourErTZQ=="     The private_key should be available as .pem file and starts and ends with the following lines:     -----BEGIN ENCRYPTED PRIVATE KEY----- -----END ENCRYPTED PRIVATE KEY-----     As it's highly recommended to encrypt the private_key, the LSR needs to know the password for how to encrypt and use the key. This is done via the passphrase configuration. Naturally the passphrase should be encrypted in the config.lua to not allow spoofing the actual cleartext passphrase.   The certificate_chain holds the Chain of Trust of the LSR Server Certificate in a .pem file. It holds multiple entries for the the Root, Intermediate and Server Specific certificate starting and ending with the following line for each individual certificate and Certificate Authority (CA):     -----BEGIN CERTIFICATE----- -----END CERTIFICATE-----     After configuring TLS and https, the LSR REST API has to be called via https://lsrserver:8001 (instead of http).   Connection to the EMS   Authentication   To secure the connection to the EMS, the LSR must know the certificates and authentication details for the EMS:     scripts.rap_server_authenticate = true scripts.rap_userid = "emsuser" scripts.rap_password = "AES:A26fBYKHJq+eMu0Fm2FlDw=="     Supply the authentication credentials as defined in the EMS's config.json - as for any other configuration the password can be used in cleartext or encrypted. It's recommended to encrypt it here as well.   HTTPS and TLS   Use the following configuration establish the https connection and using certificates     scripts.rap_ssl = true scripts.rap_cert_file = "/pathToLSR/emscertificate.pem" scripts.rap_deny_selfsigned = true scripts.rap_validate = true     This forces the certificate to be validated and also denies selfsigned certificates. In case selfsigned certificates are used, you might want to adjust above values.   The cert_file is the full Chain of Trust as configured in the EMS' config.json http_server.certificate options. It needs to match exactly, so that the LSR can actually verify and trust the connections from and to the EMS.   EMS Configuration   In the config.lua there are two sections to be configured:   http_server which enables the HTTP Server capabilities for the EMS certificates which holds all certificates that the EMS must verify in order to communicate with other servers (ThingWorx Platform, LSR)   HTTP Server Authentication and TLS Configuration   HTTP Server Authentication will require a username and password for accessing the EMS REST API. HTTP Server TLS configuration will enable TLS and https for secure and encrypted communication channels from and to the EMS.   To enable both the following configuration can be used:   "http_server": { "host": "<emsHostName>", "port": 8000, "ssl": true, "certificate": "/pathToEMS/emscertificate.pem", "private_key": "/pathToEMS/key.pem", "passphrase": "keyForEMS", "authenticate": true, "user": "emsuser", "password": "pword123" }   The passphrase as well as the password should be encrypted (see above) and the configuration should then be updated to   "passphrase": "AES:D6sgxAEwWWdD5ZCcDwq4eg==", "password": "AES:A26fBYKHJq+eMu0Fm2FlDw=="   See LSR configuration for comments on the certificate and the private_key. The same principals apply here. Note that the certificate must hold the full Chain of Trust in a .pem file for the server hosting the EMS.   After configuring TLS and https, the EMS REST API has to be called via https://emsserver:8000 (instead of http).   Certificates Configuration   The certificates configuration hold all certificates that the EMS will need to validate. If ThingWorx is configured for HTTPS and the ws_connection.encryption is set to "ssl" the Chain of Trust for the ThingWorx Platform Server Certificate must be present in the .pem file. If the LSR is configured for HTTPS the Chain of Trust for the LSR Server Certificate must be present in the .pem file.   "certificates": { "validate": true, "allow_self_signed": false, "cert_chain" : "/pathToEMS/listOfCertificates.pem" } The listOfCertificates.pem is basicially a copy of the lsrcertificate.pem with the added ThingWorx certificates and CAs.   Note that all certificates to be validated as well as their full Chain of Trust must be present in this one .pem file. Multiple files cannot be configured.   Binding to the LSR   When binding to the LSR via the auto_bind configuration, the following settings must be configured:   "auto_bind": [{ "name": "<ThingName>", "host": "<lsrHostName>", "port": 8001, "protocol": "https", "user": "luauser", "password": "AES:A26fBYKHJq+eMu0Fm2FlDw==" }]   This will ensure that the EMS connects to the LSR via https and proper authentication.   Tips   Do not use quotation marks (") as part of the strings to be encrypted. This could result in unexpected behavior when running the encryption script. Do not use a semicolon (:) as part of any username. Authentication tokens are passed from browsers as "username:password" and a semicolon in a username could result in unexpected authentication behavior leading to failed authentication requests. In the Server Specific certificates, the CN must match the actual server name and also must match the name of the http_server.host (EMS) or script_resource_host (LSR) In the .pem files first store Server Specific certificates, then all required Intermediate CAs and finally all required Root CAs - any other order could affect the consistency of the files and the certificate might not be fully readable by the scripts and processes. If the EMS is configured with certifcates, the LSR must connect via a secure channel as well and needs to be configured to do so. If the LSR is configured with certifcates, the EMS must connect via a secure channel as well and needs to be configured to do so. For testing REST API calls with resources that require encryptions and authentcation, see also How to run REST API calls with Postman on the Edge Microserver (EMS) and Lua Script Resource (LSR)   Export PEM data from KeyStore Explorer   To generate a .pem file I usually use the KeyStore Explorer for Windows - in which I have created my certificates and manage my keystores. In the keystore, select a certificate and view its details Each certificate and CA in the chain can be viewed: Root, Intermediate and Server Specific Select each certificate and CA and use the "PEM" button on the bottom of the interface to view the actual PEM content Copy to clipboard and paste into .pem file To generate a .pem file for the private key, Right-click the certificate > Export > Export Private Key Choose "PKCS #8" Check "Encrypted" and use the default algorithm; define an "Encryption Password"; check the "PEM" checkbox and export it as .pkcs8 file The .pkcs8 file can then be renamed and used as .pem file The password set during the export process will be the scripts.script_resource_passphrase (LSR) or the http_server.passphrase (EMS) After generating the .pem files I copy them over to my Linux systems where they will need 644 permissions (-rw-r--r--)

Predictive models: ​ Predictive model is one of the best technique to perform predictive analytics. This is the development of models that are trained on historical data and make predictions on new data. These models are built in order to analyse the current data records in combination with some historical data.   Use of Predictive Analytics in Thingworx Analytics and How to Access Predictive Analysis Functionality via Thingworx Analytics   Bias and variance are the two components of imprecision in predictive models. Bias in predictive models is a measure of model rigidity and inflexibility, and means that your model is not capturing all the signal it could from the data. Bias is also known as under-fitting.  Variance on the other hand is a measure of model inconsistency, high variance models tend to perform very well on some data points and really bad on others. This is also known as over-fitting and means that your model is too flexible for the amount of training data you have and ends up picking up noise in addition to the signal.   If your model is performing really well on the training set, but much poorer on the hold-out set, then it’s suffering from high variance. On the other hand if your model is performing poorly on both training and test data sets, it is suffering from high bias.   Techniques to improve:   Add more data: Having more data is always a good idea. It allows the “data to tell for itself,” instead of relying on assumptions and weak correlations. Presence of more data results in better and accurate models. The question is when we should ask for more data? We cannot quantify more data. It depends on the problem you are working on and the algorithm you are implementing, example when we work with time series data, we should look for at least one-year data, And whenever you are dealing with neural network algorithms, you are advised to get more data for training otherwise model won’t generalize.  Feature Engineering: Adding new feature decreases bias on the expense of variance of the model. New features can help algorithms to explain variance of the model in more effective way. When we do hypothesis generation, there should be enough time spent on features required for the model. Then we should create those features from existing data sets. Feature Selection: This is one of the most important aspects of predictive modelling. It is always advisable to choose important features in the model and build the model again only with important and significant features. e. let’s say we have 100 variables. There will be variables which drive most of the variance of a model. If we just select the number of features only on p-value basis, then we may still have more than 50 variables. In that case, you should look for other measures like contribution of individual variable to the model. If 90% variance of the model is explained by only 15 variables then only choose those 15 variables in the final model. Multiple Algorithms: Hitting at the right machine learning algorithm is the ideal approach to achieve higher accuracy. Some algorithms are better suited to a particular type of data sets than others. Hence, we should apply all relevant models and check the performance. Algorithm Tuning: We know that machine learning algorithms are driven by parameters. These parameters majorly influence the outcome of learning process. The objective of parameter tuning is to find the optimum value for each parameter to improve the accuracy of the model. To tune these parameters, you must have a good understanding of these meaning and their individual impact on model. You can repeat this process with a number of well performing models. For example: In random forest, we have various parameters like max_features, number_trees, random_state, oob_score and others. Intuitive optimization of these parameter values will result in better and more accurate models. Cross Validation: Cross Validation is one of the most important concepts in data modeling. It says, try to leave a sample on which you do not train the model and test the model on this sample before finalizing the model. This method helps us to achieve more generalized relationships. Ensemble Methods: This is the most common approach found majorly in winning solutions of Data science competitions. This technique simply combines the result of multiple weak models and produce better results. This can be achieved through many ways.  Bagging: It uses several versions of the same model trained on slightly different samples of the training data to reduce variance without any noticeable effect on bias. Bagging could be computationally intensive esp. in terms of memory. Boosting: is a slightly more complicated concept and relies on training several models successively each trying to learn from the errors of the models preceding it. Boosting decreases bias and hardly affects variance.     

Pgweb is a light weight cross platform client for PostgreSQL DBs written in Go. It can connect to both local & remote PostgreSQL instances behind firewall using native SSH tunneling both with password or ssk keys Due to no dependency on any other external library/utility all that is needed to run pgweb is to download it and run to connect to a working PostgreSQL server. Also works on RaspberryPi.   Note: It can also be used to connect to the CockroachDB instances, detailed in post Cockroach DB : An open source distributed SQL Database as external data store for ThingWorx   Getting Access Download pre-compiled binaries here for different platforms.   Building pgweb from source Use https://github.com/sosedoff/pgweb.git to clone the source and build it.   Additional Reads Pgweb Wiki Features List Usage Installing & Testing it After downloading the pgweb_windows_amd64.exe for windows platform (used for testing in this blog), start the pgweb utility using: pgweb_windows_amd64.exe   Note starting the pgweb utility from the command prompt shows the port on which pgweb is accessible e.g. This is when no flag is used to start up the pgweb utility, all default options used     Accessing the pgweb over localhost:8081 via a web browser     Connecting Pgweb allows for connecting to the PostgreSQL DB in variety of ways - e.g. using something like JDBC connection string like postgres://user:password@host:port/db?sslmode=mode. Additionally users can connect using standard hostname, port , username & password connection w/o SSL or via SSH tunneling.   Navigating around pgweb UI UI design is pretty straight forward and clean highlighting database name, all the tables available under that database and selected table's information on the left. To switch to different database running under the connected server click on the database name to see a drop down menu with names listed.     Selected table gets queried for all its data automatically displaying all its rows together with additional tabs for quickly reviewing the metadata about the table     Query Tab This is particularly interesting esp. for getting quick info on how the SQL query is going to perform. This could be used to analyze and generate the query plan showing preliminary data on how is it going to cost to run a particular query, how many times its going to loop over the complete table, what's going to be the execution time, etc   To get this information simply type in the sql statement under the Query tab and hit Explain Query button to generate this data. For larger tables it could take a while to get this information generated     The queried data then can be exported in formats : JSON, CSV & XML   Exporting table data out of the tables This is pretty handy feature allowing quick export of entire table's data set by right clicking on the table name and selecting one of the desired formats which includes JSON, CSV, etc.    

The Squeal functionality has been discontinued with ThingWorx 8.1, see ThingWorx 8.1.0 Release Notes   There might be scenarios where it should be disabled in earlier versions as well. This can be achieved e.g. with Tomcat Security Constraints. To add such a constraint, open <Tomcat>\webapps\Thingworx\WEB-INF\web.xml At the end of the file add a new constraint just before closing the </web-app> tag:   <security-constraint> <web-resource-collection> <web-resource-name>Forbidden</web-resource-name> <url-pattern>/Squeal/*</url-pattern> </web-resource-collection> <auth-constraint/> <user-data-constraint> <transport-guarantee>CONFIDENTIAL</transport-guarantee> </user-data-constraint> </security-constraint> Save the file and restart Tomcat. Accessing the /Thingworx/Squeal resource now will result in an error message:   HTTP Status 403 - Access to the requested resource has been denied   One scenario to be aware of is when the web.xml changes, e.g. due to updating ThingWorx or other manual changes. In such a case, ensure that the filter is still present in the file and taken into account.

Alerts via Anomaly Detection This documents objective is to provide information and links about alerts used for anomaly detection. This document covers following topics: What Is Anomaly Detection Implementing Anomaly Detection Creating an Anomaly Alert and Prerequisites Anomaly Stats Certainty Parameter Video Example On How To Create An Alert for Anomaly Detection Tips and troubleshooting What Is Anomaly Detection Anomaly Detection in ThingWorx is implemented via built-in ThingWatcher functionality. ThingWatcher detects anomalies by monitoring a data stream from a device, calculating an expected distribution of data, and validating that the current data point is a member of the expected distribution.   Implementing Anomaly Detection Anomaly Detection is enabled by default in ThingWorx. However, several steps are required to configure the functionality for your specific environment, including the prerequisite activities below.   Creating an Anomaly Alert and Prerequisites   Configuring Anomaly Detection to monitor a stream of data. For information about setting up Anomaly Detection, view Preparing ThingWorx for Anomaly Detection. Anomaly Stats Anomaly Alert Statuses moves through several statuses as it works its way through the corresponding phases. Initialized Calibrating Training Buffering Monitoring Failed Certainty Parameter The Certainty Parameter when implementing anomaly detection requires a number of factors to consider. At its most basic, ThingWatcher functionality compares two sets of data, a validation set (collected during the Calibrating phase) and a test dataset (data streaming from a remote device). ThingWatcher tries to determine the likelihood that the distribution of values in the test dataset is from the same distribution of values contained in the validation dataset. The accuracy of the model plays a large role in this determination, but so does the Certainty parameter used for the statistical analysis of the two data sets.   Video Example On How To Create An Alert for Anomaly Detection Anomaly Detection Part 1. Create connectivity between KEPServer and ThingWorx Platform. Anomaly Detection Part 2. Configure Anomaly Alert to bind simulated data coming through KEPServer for Anomaly Detection. Anomaly Detection Part 3. Viewing data via Anomaly Mashup. Tips and troubleshooting Diagnose and fix the most common issues that may be encountered when working with ThingWatcher. It cannot be stressed strongly enough that you should be familiar with your data including the average time interval between data points, and the collection duration and certainty threshold you specified.

Overview: Navigate 1.6 uses SAP ODataConnector that enables the connection to the SAP Netweaver Gateway through the ODdata specification. It is a specialized implementation of the ODataConnector. See Integration Connectors for documentation We will use SAP Netweaver Gateway Demo system: 1. Create an account on the Gateway Demo system (credentials to be used on the connector are sent by email) 2. Verify that the account has access to the basic OData sample service : https://sapes4.sapdevcenter.com/sap/opu/odata/IWBEP/GWSAMPLE_BASIC/ Note: ThingWorx can use SSO to access SAP, but in this post we will use fixed authentication. 1. OpenSAPODataConnector and configure connection settings: 2. Save the connector and go to services and execute service ValidateConnection to validate that 200 OK status is returned:   3. Open SAPPArtResourceProvider Thing and then Services and Edit the Service and in the script Enter the var results=”” Note: This is the Prefix used to get data from SAP which is added before the Part Number   4. Open the Data shape SAP_DATA_SHAPE and create the three Data Fields as below and save the Data Shape SAPMaterialCost => Type => Number SAPQuantity => Type => String SAPCurrency => Type => String SupplierID => Type => String Price => Type => String With => Type => String Depth => Type => String Height => Type => String DimUnit=> Type => String   5. Open the New Composer by setting the below preference from Administrator:   6. In New Composer open Thing ptc-SAP-OData-Connector, Services and edit the service GetProductSet in the Route info Select the End Point getProductSet. Click Done and save the Service: Execute the service GetProductSet and check that data is fetched from SAP. 7. Open SAPPartResourceProvider and edit Attribute Mapping Configuration and ass attributes defined earlier on SAP_DATA_SHAPE: 8. Open CompositePartResourceProvider and edit Attribute Mapping Configuration and ass attributes defined earlier on SAP_DATA_SHAPE:         9. Open Navigate Tailoring Pages and add the SAP attributes and save the Tailoring pages. 10. Search for some Part and open information page and check that SAP attributes are populated:  

The Asset Simulator can simulate actual device behavior without having to connect to a physical asset. It does this by replaying data sequences derived from mathematical distributions or actual asset data imported as CSV files. Virtual assets can be configured to reference these data sequences and expose them as asset behavior.   The Asset Simulator communicates with KepServerEX in the same way that a real device does. The simulated asset behavior is controlled through an administration console. If you would like to test with the Asset Simulator 8.2.0, please find attached a guide and the actual files necessary.   Notes: The attached Asset Simulator applies to both Manufacturing and Service Apps If using ThingWorx Manufacturing Apps, import the Manufacturing Apps demo data If using ThingWorx Service Apps, import the Service Apps demo data

Connectors allow clients to establish a connection to Tomcat via the HTTP / HTTPS protocol. Tomcat allows for configuring multiple connectors so that users or devices can either connect via HTTP or HTTPS.   Usually users like you and me access websites by just typing the URL in the browser's address bar, e.g. "www.google.com". By default browsers assume that the connection should be established with the HTTP protocol. For HTTPS connections, the protocol has to be specified explictily, e.g. "https://www.google.com"   However - Google automatically forwards HTTP connections automatically as a HTTPS connection, so that all connections are using certificates and are via a secure channel and you will end up on "https://www.google.com" anyway.   To configure ThingWorx to only allow secure connections there are two options...   1) Remove HTTP access   If HTTP access is removed, users can no longer connect to the 80 or 8080 port. ThingWorx will only be accessible on port 443 (or 8443).   If connecting to port 8080 clients will not be redirected. The redirectPort in the Connector is only forwarding requests internally in Tomcat, not switching protocols and ports and not requiring a certificate when being used. The redirected port does not reflect in the client's connection but only manages internal port-forwarding in Tomcat.   By removing the HTTP ports for access any connection on port 80 (or 8080) will end up in an error message that the client cannot connect on this port.   To remove the HTTP ports, edit the <Tomcat>\conf\server.xml and comment out sections like       <!-- commented out to disallow connections on port 80 <Connector port="80" protocol="org.apache.coyote.http11.Http11NioProtocol" connectionTimeout="20000" redirectPort="443" /> -->     Save and restart Tomcat. If opening Tomcat (and ThingWorx) in a browser via http://myServer/ the connection will fail with a "This site can’t be reached", "ERR_CONNECTION_REFUSED" error.   2) Forcing insecure connections through secure ports   Alternatively, port 80 and 8080 can be kept open to still allow users and devices to connect. But instead of only internally forwarding the port, Tomcat can be setup to also forward the client to the new secure port. With this, users and devices can still use e.g. old bookmarks and do not have to explicitly set the HTTPS protocol in the address.   To configure this, edit the <Tomcat>\conf\web.xml and add the following section just before the closing </web-app> tag at the end of the file:     <security-constraint>        <web-resource-collection>              <web-resource-name>HTTPSOnly</web-resource-name>              <url-pattern>/*</url-pattern>        </web-resource-collection>        <user-data-constraint>              <transport-guarantee>CONFIDENTIAL</transport-guarantee>        </user-data-constraint> </security-constraint>     In <Tomcat>\conf\web.xml ensure that all HTTP Connectors (port 80 and 8080) have their redirect port set to the secure HTTPS Connector (usually port 443 or port 8443).   Save and restart Tomcat. If opening Tomcat (and ThingWorx) in a browser via http://myServer/ the connection will now be forwarded to the secure port. The browser will now show the connection as https://myServer/ instead and connections are secure and using certificates.   What next?   Configuring Tomcat to force insecure connection to actually secure HTTPS connection just requires a simple configuration change. If you want to read more about certificates, encryption and how to setup ThingWorx for HTTPS in the first place, be sure to also have a look at   Trust & Encryption - Theory Trust & Encryption - Hands On

Why we need improve ThingWorx Query Performance? ThingWorx is good at injesting data from systems and devices and persisting data in Value Stream. When users build mashups/services which pull anything more than small amounts of data back, or have many users making queries at the same time, they encounter slow performance and/or server failure. Let's take a typical example which may happen among many of our end users: User A is currently gathering data from their edge devices and placing them in a Value Stream. About 80 properties are sharing the same Value Stream and they are estimating roughly 10 million rows a month of data into the platform. Their application requires querying large sections of this data and displaying aggregate info to certain mashups. A ton of work has been done to optimize these queries to make them as efficient as possible. However, querying with results larger than a few thousand rows causes RAM usage to spike so high, that the JVM runs out of heap space and brings down the server. What we have done to improve Query Performance in ThingWorx 8.2? 1. Improvements to the implementation of QueryPropertyHistory and QueryNamedPropertyHistory services:      1) These two query services are optimized by:          (1) Before ThingWorx 8.2, we query property history from database for each properties(imagine 80 properties in our example in the beginning), and then combine the            dataset(80 datasets in our example!) and display them in one table.          (2) In ThingWorx 8.2, we have replaced the "query for each property" with "single query for all properties"      2) Moved most of the query work to database level which is formally done post processing after grabing all the dataset from database including:          (1) Moved Combiner logic to database          (2) Moved Filter functionality in database Note: This improvement is only implemented on MS SQL and PostgreSQL, other persistence providers are not considered in this version, and this may be the plan for future release. 2. Improved string handling for the Query<Basetype>PropertyHistory services Memory reduction for these services are implemented by storing only one copy of each distinct string How much memory reduction could the Query Service Improvement bring QueryPropertyHistory and QueryNamedPropertyHistory Services From our internal test scenario we have seen approximately a 20% reduction in Thingworx platform memory utilization for these services with no Filter applied. Filtering may reduce memory an additional amount. Besides, a approximately 10% execution time saved as a result of this new improvement! Query<Basetype>PropertyHistory services Memory reduction is highly dependent on the stored data. From our internal test scenario, this improvement is providing up to 10% memory reduction on the platform. Note: This improvement is only implemented on MS SQL and PostgreSQL, so the memory reduction is not applied to other persistence providers. Best practice to call Query Services to improve Performance Although there are some Query service improvements in 8.2, still by following some rules or choosing a suitable service would bring extra performance improvement according to different use scenarios. From this point of view, this secion applies to all ThingWorx versions not specially for 8.2. 1. Limit the number of calls to the Query services from a given mashup/app       By repeatedly calling QueryPropertyHistory to display data may cause severe performance problems. Try to clean up unnecessary service calls in mashup and there should be a significant       improvement to the system.       Note: That was without any of the above improvements. 2. Use Query<BASETYPE>PropertyHistory service as much as possible     Unless customer needs to create a normalized dataset for multiple properties, try using QueryIntegerPropertyHistory, QueryStringPropertyHistory, etc. as they return smaller datasets and do not     use the combiner to normalize the data across a large number of properties. 3. Use QueryNamedPropertyHistory as much as possible     Similarly, QueryNamedPropertyHistory can provide a smaller dataset if you don’t need the values from all the properties on the Thing. So for example if a Thing has 10 logged properties but     you only need 3 returned; using QueryNamedPropertyHistory and identifying the three properties specifically needed will return a significantly reduced data set. 4. Use a Query to narrow down the dataset Where/when it makes sense, use a Query to narrow down the dataset; similarly to option 3, the returned dataset will be smaller. Note: this will have limited use scenario though. What needs to be done for end user when upgrading or utilize the service in ThingWorx 8.2 All the changes are transparent to the user! We improved the underlying implementation of the existing services so users won’t need to do anything after upgrade to see the improvements. There are no changes to the databases or schema.  

ThingWorx 8.2 System Requirements ThingWorx 8.2 Helpcenter The following feature enhancements and bug fixes exist in ThingWorx 8.2.0: Due to security updates, a minimum version of Apache Tomcat 8.0.47 or 8.5.23 should be used with ThingWorx. Enhancements Platform • Included information about opting out of metrics reporting. For more information, see the ThingWorx Metrics Reporting Services Configuration section in the Platform Subsystem help topic. • The Script Log Error has been added to improve error logging for scripts. • Added support to allow mashups to be rendered using jQuery 3.x runtime. • Query service optimization. This includes improved performance for the QueryPropertyHistory and QueryPropertyNamedHistory services. Previously, a database call was made for every logged property. With this improvement, one database call is made for all logged properties, resulting in the following improvements: ▪ A ~20% decrease in memory usage for the QueryPropertyHistory and QueryNamedPropertyHistory service queries if no filters are applied (PostgreSQL and MSSQL). ▪ Decreased time to execute query (~10%) for the QueryPropertyHistory and QueryNamedPropertyHistory services. Depends on latency to the database (PostgreSQL and MSSQL). ▪ Additional decrease in memory, based on the filter supplied during the query for QueryPropertyHistory and QueryNamedPropertyHistory services. (PostgreSQL and MSSQL). If a filter is applied that reduces the record count by 50%, then there is an additional 50% decrease in memory usage on top of the other 50% described in the first point. This optimization also results in an approximate 10% decrease in memory for single property queries. The Audit Subsystem has been added. It supports the following capabilities: • Automatically add audit entries to online storage. • Search for audit entries (use the QueryAuditHistory service) stored online. • Archive online audit entries to offline storage (automatically performed daily by default). • Export audit data, using the language selected for the export. • Purge online audit data on the basis of a specified number of days for audit data to remain online and also on the specified number of rows to keep online. • Clean up archived audit data automatically, based on a configured schedule. • The security of the PASSWORD base type has been enhanced and is now encrypted. See Passwords for more information. • Added the Collection Widget, which allows you to replicate/repeat mashups and content by using infotables to dynamically supply visual content and data. Refer to the KCS article for additional information here • Additional capability has been added to New Composer. For more information, refer to the ThingWorx Community blog. • The licensing process has been improved. An activation ID is no longer required to obtain a license and a new license file is not required for minor or major release upgrades. ◦ For connected scenarios, activation IDs are no longer required in the platform-settings.json file. ◦ For disconnected scenarios, go to the enhanced PTC Support site pages, select the product, enter a Device ID, and retrieve a license. • You can enable the Application Key Authenticator when SSO is enabled by editing the sso-settings.json configuration. For more information, see Configure the sso-settings.json File. • The CSS Editor was added to Mashup Builder, which allows developers to create modern experiences with responsiveness, animations, and advanced styling and behaviors. Refer to the KCS article for additional information here. • Added support for "Store and Forward" functionality to the interface between KEPServerEX and the ThingWorx platform. KepServerEX can be configured to store updated property data to disk when disconnected from the ThingWorx platform and will send that data gracefully when the connection is re-established. • In mashups, row and column gadget sizes 1 to 8 are now available. TW-25477 Bug Fixes Platform Related JIRA • Fixed an issue with Thing Shapes when editing subscriptions twice before canceling or closing in which the second edit was not saved. TW-28718 • Fixed an issue that was causing SQL Server apparent deadlock exceptions. TW-28208 • Added useful log information for troubleshooting LDAP and Active Directory errors. TW-23873 • Fixed an issue with exception handling in DSLProcessor in which line numbers were not included in the log. TW-18042 TW-17255 • Fixed an issue in which opening/closing brackets are not highlighted if there were 100 or more lines of code in a JavaScript service. TW-12740 Mashup Builder • Service error notification messages were fixed to display on multiple lines based on line breaks in the message. TW-24738 • Fixed an issue in which a master mashup header image was not fully displayed. PSPT-3365 Extensions Related JIRA • The Google Maps JavaScript API was updated to prevent the use of the library without an api key. If you are using the Google Map extension in your application, verify that the extension's metadata.xml file is updated with the correct URL (https://maps.google.com/maps/api/js?sensor=false&key=YOUR_API_KEY). Re-zip the extension and reimport into ThingWorx after making this change.

With the release of ThingWorx 8.1.3, we’ve made a few changes to our licensing model in order to simplify the process.  If you are using any version of ThingWorx 8.1 or later, all that is required to license the platform is a device ID, which is a unique identifier that is requested during the licensing process.  The ID is generated upon deployment of the platform. For those working with ThingWorx 8.1.3 and later, it is possible for the platform to automatically obtain a license from PTC should the server be able to communicate successfully with our license portal.  If the connection is blocked for any reason, or you are using a version of 8.1 prior to 8.1.3, a manual approach is also available via the PTC website. For customers who have already licensed any previous release version of ThingWorx 8.1, there is no need to re-request a license as your current license is compatible with subsequent releases of the platform (e.g. 8.1.3, 8.2.0, 9.0.0, etc.).  You will only need to request a new license file if you purchase new features that aren’t included in your current license. The article at the following link contains all of the necessary details for licensing any version of ThingWorx 8.1 or later. Licensing ThingWorx 8.1 and Later Should you have any questions regarding this process, please feel free to reach out to the ThingWorx Support team for further assistance.  The support team can be reached by logging a case as described in the following link: How to Create a New Support Case and Track an Existing Case on PTC eSupport Portal

Video Author:                     Durgesh Patel Original Post Date:            June 12, 2017 Applicable Releases:        ThingWorx   Description: In this video we review the different features available in Grid Advanced Extension version 2.0.  

Original Post Date:            September 30, 2016   Description: This tutorial video will walk you through the installation process for the PostgreSQL-based version of the ThingWorx Platform (7.2) in a RHEL environment.  All required software components will be covered in this video.    

Original Post Date:     June 6, 2016 Description: This is a video tutorial on creating a Value Stream, adding the Value Stream to a Remote Thing, adding, binding and querying the (remote) properties.