IoT Tips

Disclaimer: This post does of course not express any political views.   Pie Chart Coloring   In ThingWorx Pie Charts use a default color schema based on the DefaultChartStyle Definitions. These schemas are using fixed numbering and coloring systems, e.g. 1 is blue, 2 is green, 3 is red and so on. All Pie Charts will be rendered with these colors in the same order, no matter which data the chart is using. Visualization of data with the default colors might not necessarily help in creating an easy to read chart.   Just take a look at the following example with the default color schema. Let's just take political parties - as they are usually associated with a distinct color - to illustrate how the default color schema will fail depending on the data displayed.         In the first example, just by sheer coincidence the colors are perfectly matching the parties. When introducing a new party to the pool suddenly the blues are rendered green and the yellows rendered light-blue etc. This can be quite confusing, especially on election night 😉   Custom Color Schema   PoliticalParties Thing   To test a custom color schema, we first need to create a new Thing: PoliticalParties as a GenericThing Add a dataset property with the following PoliticalParties DataShape.         Save the Thing and set the InfoTable to:   Key Value The Purples 20 The Blues 20 The Greens 20 The Reds 20 The Yellows 20 Others 20   Number values don't actually matter too much, as the Pie Chart will automatically distribute them according to their percentage.   PoliticalParties Mashup   Create a new Mashup and add a PieChart to the canvas. Bind the PoliticalParties > GetPropertyValues > dataset to the Data input of the Widget. Ensure to set the LabelField to key and the ValueField to value for a correct mapping.     Save the Mashup and preview it.   It should show a non-matching color for each party listed in the InfoTable.   Custom Styles and States   Create new custom Style Definitions for each political party. As the Pie Chart is only using the Background Color other properties can stay on the default. I chose to go with a more muted version of the colors to make the chart easier to look at.         With the newly defined colors we can now generate a new State Definition as follows:       The States allow to evaluate the key-Strings in the Thing's InfoTable and assign a Style Definition depending on the actual value. In this definition we map a color schema based on the InfoTable's key-value to create a 1:1 mapping for the Strings.   This means, no matter where a certain party is positioned in the chart it will be tinted with its associated color.   Refining the Mashup   Back in the Mashup, select the PieChart. In the ColorFormat property choose the newly created State Definition.     Save the Mashup and preview it. With the States and Styles applies, colors are now displayed correctly.       Even when changing positions and numbers in the original InfoTable of the PoliticalParties Thing, the chart now considers the mapping of Strings and still displays the colors correctly.  

Precision and Recall are the evaluation matrices that are used to evaluate the machine learning algorithm used. This post needs some prior understanding of the confusion matrix and would recommend you to go through it here.   Example of Animal Image Recognition Consider the below Confusion Matrix for the input of the animal images and algorithm trying to identify the animal correctly: ANIMALS Cat Dog Leopard Tiger Jaguar Puma Cat 62 2 0 0 1 0 Dog 1 50 1 0 4 0 Leopard 0 2 98 4 0 0 Tiger 0 0 10 78 2 0 Jaguar 0 1 8 0 46 0 Puma 2 0 0 1 1 42   Explaining Few Random Grids: [Cat, Cat]: The grid is having the value 62. It means the image of a cat was identified as a cat for 62 times. [Cat, Dog]: The grid is having the value 2. It means the image of a cat was identified as dog twice. [Leopard, Tiger]: The grid is having the value 4. It means the image of Leopard was identified as Tiger for 4 times.   Questions To Find Some Answers    Q.How many times our algorithm predicted the image to be Tiger? A. Looking at the Tiger column: 0+0+4+78+0+1 = 83   Q.What is the probability that Puma will be classified correctly? A. Looking at the matrix above, we can see that we 42 times Puma was classified correctly. But twice it was classified as Cat, once as Tiger and once as Jaguar. So the probability will come down as: 42/(42+2+1+1)= 42/46 = 0.91      This concept is called as RECALL. It is the fraction of correctly predicted positives out of all actual positives. So we can say that Recall = (True Positives) / (True Positives + False Negatives)   Q. What is the probability that when our algorithm is identifying the image as Cat, it is actually Cat? A. Looking at the matrix above, we can see that once our algorithm has identified a Dog as a Cat, twice Puma as Cat and 62 times Cat as Cat. So the probability will come down as: 62/(62+1+2) = 0.95      This concept is called as PRECISION. It is the fraction if correctly predicted positives out of all predicted positives. So we can say that Precision = (True Positives) / (True Positives + False Positives)

This is a follow-up post on my initial document about Edge Microserver (EMS) and Lua Script Resource (LSR) security. While the first part deals with fundamentals on secure configurations, this second part will give some more practical tips and tricks on how to implement these security measurements.   For more information it's also recommended to read through the Setting Up Secure Communications for WS EMS and LSR chapter in the ThingWorx Help Center. See also Trust & Encryption Theory and Hands On for more information and examples - especially around the concept of the Chain of Trust, which will be an important factor for this post as well.   In this post I will only reference the High Security options for both, the EMS and the LSR. Note that all commands and directories are Linux based - Windows equivalents might slightly differ.   Note - some of the configuration options are color coded for easy recognition: LSR resources / EMS resources   Password Encryption   It's recommended to encrypt all passwords and keys, so that they are not stored as cleartext in the config.lua / config.json files.   And of course it's also recommended, to use a more meaningful password than what I use as an example - which also means: do not use any password I mentioned here for your systems, they might too easy to guess now 🙂   The luaScriptResource script can be used for encryption:   ./luaScriptResource -encrypt "pword123" ############ Encrypted String AES:A26fBYKHJq+eMu0Fm2FlDw== ############   The wsems script can be used for encryption:   ./wsems -encrypt "pword123" ############ Encrypted String AES:A26fBYKHJq+eMu0Fm2FlDw== ############   Note that the encryption for both scripts will result in the same encrypted string. This means, either the wsems or luaScriptResource scripts can be used to retrieve the same results.   The string to encrypt can be provided with or without quotation marks. It is however recommended to quote the string, especially when the string contains blanks or spaces. Otherwise unexpected results might occur as blanks will be considered as delimiter symbols.   LSR Configuration   In the config.lua there are two sections to be configured:   scripts.script_resource which deals with the configuration of the LSR itself scripts.rap which deals with the connection to the EMS   HTTP Server Authentication   HTTP Server Authentication will require a username and password for accessing the LSR REST API.     scripts.script_resource_authenticate = true scripts.script_resource_userid = "luauser" scripts.script_resource_password = "pword123"     The password should be encrypted (see above) and the configuration should then be updated to   scripts.script_resource_password = "AES:A26fBYKHJq+eMu0Fm2FlDw=="   HTTP Server TLS Configuration   Configuration   HTTP Server TLS configuration will enable TLS and https for secure and encrypted communication channels from and to the LSR. To enable TLS and https, the following configuration is required:     scripts.script_resource_ssl = true scripts.script_resource_certificate_chain = "/pathToLSR/lsrcertificate.pem" scripts.script_resource_private_key = "/pathToLSR/key.pem" scripts.script_resource_passphrase = "keyForLSR"     It's also encouraged to not use the default certificate, but custom certificates instead. To explicitly set this, the following configuration can be added:     scripts.script_resource_use_default_certificate = false     Certificates, keys and encryption   The passphrase for the private key should be encrypted (see above) and the configuration should then be updated to     scripts.script_resource_passphrase = "AES:A+Uv/xvRWENWUzourErTZQ=="     The private_key should be available as .pem file and starts and ends with the following lines:     -----BEGIN ENCRYPTED PRIVATE KEY----- -----END ENCRYPTED PRIVATE KEY-----     As it's highly recommended to encrypt the private_key, the LSR needs to know the password for how to encrypt and use the key. This is done via the passphrase configuration. Naturally the passphrase should be encrypted in the config.lua to not allow spoofing the actual cleartext passphrase.   The certificate_chain holds the Chain of Trust of the LSR Server Certificate in a .pem file. It holds multiple entries for the the Root, Intermediate and Server Specific certificate starting and ending with the following line for each individual certificate and Certificate Authority (CA):     -----BEGIN CERTIFICATE----- -----END CERTIFICATE-----     After configuring TLS and https, the LSR REST API has to be called via https://lsrserver:8001 (instead of http).   Connection to the EMS   Authentication   To secure the connection to the EMS, the LSR must know the certificates and authentication details for the EMS:     scripts.rap_server_authenticate = true scripts.rap_userid = "emsuser" scripts.rap_password = "AES:A26fBYKHJq+eMu0Fm2FlDw=="     Supply the authentication credentials as defined in the EMS's config.json - as for any other configuration the password can be used in cleartext or encrypted. It's recommended to encrypt it here as well.   HTTPS and TLS   Use the following configuration establish the https connection and using certificates     scripts.rap_ssl = true scripts.rap_cert_file = "/pathToLSR/emscertificate.pem" scripts.rap_deny_selfsigned = true scripts.rap_validate = true     This forces the certificate to be validated and also denies selfsigned certificates. In case selfsigned certificates are used, you might want to adjust above values.   The cert_file is the full Chain of Trust as configured in the EMS' config.json http_server.certificate options. It needs to match exactly, so that the LSR can actually verify and trust the connections from and to the EMS.   EMS Configuration   In the config.lua there are two sections to be configured:   http_server which enables the HTTP Server capabilities for the EMS certificates which holds all certificates that the EMS must verify in order to communicate with other servers (ThingWorx Platform, LSR)   HTTP Server Authentication and TLS Configuration   HTTP Server Authentication will require a username and password for accessing the EMS REST API. HTTP Server TLS configuration will enable TLS and https for secure and encrypted communication channels from and to the EMS.   To enable both the following configuration can be used:   "http_server": { "host": "<emsHostName>", "port": 8000, "ssl": true, "certificate": "/pathToEMS/emscertificate.pem", "private_key": "/pathToEMS/key.pem", "passphrase": "keyForEMS", "authenticate": true, "user": "emsuser", "password": "pword123" }   The passphrase as well as the password should be encrypted (see above) and the configuration should then be updated to   "passphrase": "AES:D6sgxAEwWWdD5ZCcDwq4eg==", "password": "AES:A26fBYKHJq+eMu0Fm2FlDw=="   See LSR configuration for comments on the certificate and the private_key. The same principals apply here. Note that the certificate must hold the full Chain of Trust in a .pem file for the server hosting the EMS.   After configuring TLS and https, the EMS REST API has to be called via https://emsserver:8000 (instead of http).   Certificates Configuration   The certificates configuration hold all certificates that the EMS will need to validate. If ThingWorx is configured for HTTPS and the ws_connection.encryption is set to "ssl" the Chain of Trust for the ThingWorx Platform Server Certificate must be present in the .pem file. If the LSR is configured for HTTPS the Chain of Trust for the LSR Server Certificate must be present in the .pem file.   "certificates": { "validate": true, "allow_self_signed": false, "cert_chain" : "/pathToEMS/listOfCertificates.pem" } The listOfCertificates.pem is basicially a copy of the lsrcertificate.pem with the added ThingWorx certificates and CAs.   Note that all certificates to be validated as well as their full Chain of Trust must be present in this one .pem file. Multiple files cannot be configured.   Binding to the LSR   When binding to the LSR via the auto_bind configuration, the following settings must be configured:   "auto_bind": [{ "name": "<ThingName>", "host": "<lsrHostName>", "port": 8001, "protocol": "https", "user": "luauser", "password": "AES:A26fBYKHJq+eMu0Fm2FlDw==" }]   This will ensure that the EMS connects to the LSR via https and proper authentication.   Tips   Do not use quotation marks (") as part of the strings to be encrypted. This could result in unexpected behavior when running the encryption script. Do not use a semicolon (:) as part of any username. Authentication tokens are passed from browsers as "username:password" and a semicolon in a username could result in unexpected authentication behavior leading to failed authentication requests. In the Server Specific certificates, the CN must match the actual server name and also must match the name of the http_server.host (EMS) or script_resource_host (LSR) In the .pem files first store Server Specific certificates, then all required Intermediate CAs and finally all required Root CAs - any other order could affect the consistency of the files and the certificate might not be fully readable by the scripts and processes. If the EMS is configured with certifcates, the LSR must connect via a secure channel as well and needs to be configured to do so. If the LSR is configured with certifcates, the EMS must connect via a secure channel as well and needs to be configured to do so. For testing REST API calls with resources that require encryptions and authentcation, see also How to run REST API calls with Postman on the Edge Microserver (EMS) and Lua Script Resource (LSR)   Export PEM data from KeyStore Explorer   To generate a .pem file I usually use the KeyStore Explorer for Windows - in which I have created my certificates and manage my keystores. In the keystore, select a certificate and view its details Each certificate and CA in the chain can be viewed: Root, Intermediate and Server Specific Select each certificate and CA and use the "PEM" button on the bottom of the interface to view the actual PEM content Copy to clipboard and paste into .pem file To generate a .pem file for the private key, Right-click the certificate > Export > Export Private Key Choose "PKCS #8" Check "Encrypted" and use the default algorithm; define an "Encryption Password"; check the "PEM" checkbox and export it as .pkcs8 file The .pkcs8 file can then be renamed and used as .pem file The password set during the export process will be the scripts.script_resource_passphrase (LSR) or the http_server.passphrase (EMS) After generating the .pem files I copy them over to my Linux systems where they will need 644 permissions (-rw-r--r--)

This is going to cover one way of configuring an SSL passthrough using HAProxy.  This guide is intended to be a reference document, and administrators looking to configure an SSL passthrough should make sure the end solution meets both their company's business and security needs.   Why use SSL Passthrough instead of SSL Termination? The main reason for ThingWorx would be if a company requires encrypted communication internally, as well as externally.  With SSL Termination, the request between the load balancer and the client is encrypted.  But the load balancer takes on the role to decrypt and passes that back to the server.  With SSL Passthrough, the request goes through the load balancer as is, and the decryption happens on the ThingWorx Application server.   What you will need to continue with this guide:   HAProxy installed A working ThingWorx application server (Guide to getting one setup can be found here) Tomcat configured for ssl NOTE : Always contact your Security team and make sure you have a certificate that meets your business policy For this tutorial, I created a self-signed certificate following along with the below guide.  If you have already obtained a valid certificate, then you can just skip over the step of creating it, and follow along with the Tomcat portion https://www.ptc.com/en/support/article?n=CS193947 Once configured, restart Tomcat and verify it is working by navigating to https://<yourServer>:<port>/Thingworx   With ThingWorx running as SSL and HAProxy installed, we just need to make sure the HAProxy configuration is setup to allow SSL traffic through.  We use 'mode tcp' to accomplish this.   On your HAProxy machine, open /etc/haproxy/haproxy.cfg for editing.  While most of this can be customized to fit your business needs, some variation of the highlighted portions below need to be included in your final configuration:   global         log /dev/log    local0         log /dev/log    local1 notice         chroot /var/lib/haproxy         stats socket /run/haproxy/admin.sock mode 660 level admin         stats timeout 30s         user haproxy         group haproxy         daemon           # Default SSL material locations         ca-base /etc/ssl/certs         crt-base /etc/ssl/private           # Default ciphers to use on SSL-enabled listening sockets.         # For more information, see ciphers(1SSL). This list is from:         #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/         # An alternative list with additional directives can be obtained from         #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy         ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS         ssl-default-bind-options no-sslv3   defaults         log global          option tcplog          mode tcp          option http-server-close          timeout connect 1s          timeout client  20s          timeout server  20s          timeout client-fin 20s          timeout tunnel 1h          errorfile 400 /etc/haproxy/errors/400.http          errorfile 403 /etc/haproxy/errors/403.http          errorfile 408 /etc/haproxy/errors/408.http          errorfile 500 /etc/haproxy/errors/500.http          errorfile 502 /etc/haproxy/errors/502.http          errorfile 503 /etc/haproxy/errors/503.http          errorfile 504 /etc/haproxy/errors/504.http         frontend https          bind *:443          mode tcp          default_backend bk_app           backend bk_app          mode tcp          server TWXAPP01  <twxapp01IP>:<port>         In this example, the user would connect to https://<loadbalancer>/Thingworx and the load balancer would forward the requests to https://<twxapp01IP>:<port>/Thingworx   That’s it!   A couple of side notes:   The load balancer port the clients connect to does not need to be the same as the ThingWorx port the load balancer will forward to If working in a Highly Available configuration, each ThingWorx Application server needs to have its own certificate configured If HAProxy seems unstable, try updating to the latest release If it is on the latest release according to the Unix repository, check https://www.haproxy.org/ and see if there is a later stable release.  There have been some issues where Ubuntu's latest update in the repository is actually a few years old

ThingWorx 8.3 covers the following areas of the product portfolio:  ThingWorx Analytics, ThingWorx Utilities, and ThingWorx Foundation which includes Connection Server and Edge capabilities.   Highlights of the release include:   ThingWorx Foundation Next Generation Composer: Now default admin and developer interface Full Feature parity with legacy Composer New capability for User and Group administration, Authorization and permissions, Export, Monitoring and Logging. More in Helpcenter Localization support for German and French Mashup Builder: JQuery 3 upgrade Grid Advanced Extension now supports Cell Editing and Footers Platform: Active Directory (AD) Integration enhancements for larger AD forests and user extension field mapping Upgrade in-place enhancements for Java SDK developers Developer Enablement Capture the usage statics such as time taken to execute a ThingWorx service, # of times a service runs in ThingWorx using Service Utilization Statistics functionality powered by all new and efficient Utilization Subsystem. Collect ThingWorx system data such as ESAPI configuration, ThingworxStorage logs, licensing, and JVM information to better diagnose system issues Service Utilization Statistics: ThingWorx Support Package tool Administrator Password and Password Length New installations of ThingWorx will be required to supply the initial Administrator password of the installer’s choice. That password must be supplied via a new entry in the platform-settings.json file. After the initial installation, the Administrator password should then be changed to a strong password to be used going forward. Additional information. As a step toward industry best practices, the Administrator password and all new passwords will need to be at least 10 characters.  When upgrading to 8.3, passwords from older versions of the platform will not need to be modified, but any new passwords being created will need to be at least 10 characters long. See the installation instructions for complete details.   ThingWorx Analytics New Descriptive Services  Core statistics (min, max, deviation, etc.), data distribution (binning), confidence intervals, and other useful calculations. Frequency analysis and transformation (via fast Fourier transform) for troubleshooting use cases and predictive analytics applications Improves users’ ability to apply logic and derive the following insights from streaming data without constructing complex models or accessing machine learning: Enables platform developers to easily process platform data in their applications and prepare the data for predictions. Statistical Process Control (SPC) Services Provides industry-standard calculations that allow IoT developers to implement SPC “control chart rules” in their applications.  Useful in manufacturing and in monitoring equipment and processes. Supports a wide assortment of rules, including number of points continuously above / below a range, in and out of range, increasing or decreasing trends, or alternating directions. Analytics Workbench Bundles the two Analytics interfaces (Analytics Builder and Manager) into a new Analytics section in Composer. Predictive Analytics Improvements Reduces overall install and administration complexity. Improves handling of time dseries data when used in predictive scoring. Includes a new learner, Support Vector Machines, enhancing the platform’s utility in building Boolean predictions. Includes a new ensemble method, Majority Vote, that improves generated model accuracy. Provides redundancy filtering which can optionally remove redundant information to improve explanatory analytics (Signals) and predictive model training. Now supports time series lookahead configuration, simplifying this type of prediction. Replaces ThingPredictor predictive scoring in Analytics Manager with native Analytics Server scoring: Improves scalability of concurrent jobs. Axeda Compatibility Package IDM Connector Support o   ACP v1.1.0 introduces the IDM Connector which enables Axeda customers to connect their Axeda IDM agents to the ThingWorx platform.  The IDM Connector provides support for registration requests, property updates, faults, events, file uploads and downloads.  Axeda ThingWorx Entity Exporter Update o   ACP v1.1.0 also includes an updated version of Axeda-ThingWorx Entity Exporter (ATEE) which now supports exporting Axeda IDM assets from the Axeda application into a format that can be imported in the ThingWorx Platform.  eMessage Connector Improvements o   Additionally, ACP v1.1.0 includes support for instruction based Software Content Management packages for the eMessage Connector which allows you to download file(s), execute instruction(s) and optionally restart the agent.  The Axeda Compatibility Extension (ACE) has new entities to support the IDM Connector and SCM for the eMesssage Connector.  o   Finally, updated versions of the Axeda Compatibility Extensions (ACE) and the Connection Services Extension (CSE) are included in ACP v1.1.0 and provide an improved workflow for granting permissions to the eMessage and IDM Connectors. ThingWorx Extension Updates Websocket Tunnel Extension Update The Websocket Tunnel Extension was updated for 8.3 to support the upgrade to jQuery3 Grid Advanced 4.0.0 comes with 2 key features: Editing - we now have cell editing support for all basetypes. The previous version had boolean editing; 4.0.0 now includes support for all basetypes. Footers - A footer section can now be added to the Grid to display rolled-up Grid totals. You can perform client-side calculations like count, min, max and average, and it includes support for custom functions. Note - Grid Advanced 4.0.0 only supports ThingWorx 8.3 and above. Custom Charts 3.0.1 12 Bug Fixes Google Maps 3.0.1 General Bug Fixes ThingWorx Utilities With the 8.3 Release, ThingWorx Utilities functionality are being repackaged into ThingWorx Foundation and ThingWorx Asset Advisor.  ThingWorx Workflow will now be available with Foundation.  The functionality from the Asset and Alert Management Utilities will be delivered in ThingWorx Asset Advisor.  ThingWorx Software Content Management capabilities will continue to be available for customer to manage the delivery of Software to their Connected Products.  The naming of “Utilities” is being phased out of the ThingWorx Platform packaging but the key functionality formerly described as ThingWorx Utilities continues to be delivered with version 8.3.   ThingWorx 8.3 Reference Documents ThingWorx Analytics 8.3 Reference Documents ThingWorx Platform 8.3 Release Notes ThingWorx Platform Help Center ThingWorx Edge SDKs and WebSocket-based Edge MicroServer Help Center ThingWorx Connection Services Help Center ThingWorx Analytics Help Center ThingWorx Industrial Connectivity Help Center ThingWorx Utilities Help Center ThingWorx Utilities Installation Guide     ThingWorx eSupport Portal ThingWorx Developer Portal PTC Marketplace   The following items will be available for download from the PTC Software Download site on June 8, 2018. ThingWorx Platform – Select Release 8.3 ThingWorx Utilities – Select Release 8.3 ThingWorx Analytics – Select Release 8.3 ThingWorx Extensions – Select Individual Extensions for download.  Will be available with the next Marketplace refresh

In this IoT-AR Tech Tip, we will cover what the microservices do, and their functions overall in the ThingWorx Analytics Server Application.   With the new architecture changes introduced with ThingWorx Analytics 8.1, many users have asked for a more descriptive explanation of the purpose of the microservices that make up the Analytics Server application. In 8.3, Descriptive Microservice has been introduced, and changings in how Predictive Scoring were incorporated. ThingPredictor has been deprecated and its primary functions have been incorporated into ThingWorx Analytics Server’s Prediction microservice.   Many of these microservices can be installed as separate distinct utilities, though there some that are required for base functionality of Analytics Server. This allows custom installation by the end-user to tailor the TWA deployment to their needs.   ThingWorx Analytics Server Microservices   Analytics Microservice – Analytics Server Edge Agent acts as an integration point between the ThingWorx server and the ThingWorx Analytics server. The Edge Agent automatically registers with ThingWorx and instantiates the Things that represent the connected ThingWorx Analytics microservicers. This component is mandatory.   Clustering Microservice - The Clustering Microserver provides services that categorize dataset records into groups (clusters) based on their similarities. Clustering returns results in the form of a PMML model. This component is optional.   Data Microservice - The Data Microserver provides data-handling services. These services include creating a dataset, appending new data, viewing dataset metadata, querying and retrieving information about the distribution of data in the dataset. This component is mandatory.   Predictive Microservice - The Prediction Microserver provides both real time and asynchronous (batch) predictive scoring. The scoring process evaluates each record in a dataset against a prediction model. Each record is assigned a predictive score that reflects the model's predicted outcome for that record. This component is optional. This component replaces ThingPredictor in functionality.   Prescriptive Microservice - The Prescriptive Microserver provides real time prescriptive scoring, which examines how certain changes might affect future outcomes. Fields identified as levers can be varied to determine how specific changes might affect future outcomes. This component is optional.   Profiles Microservice - The Profiling Microserver provides services to identify distinct subpopulations (profiles) within a dataset that share similar characteristics and are different from other subpopulations in statistically significant ways.  Profiles are not required in order to make a prediction, but they contribute to a strategic understanding of the complex factors associated with specific outcomes. This component is optional.   Results Microservice – The Results Microserver provides services for working with results. It also provides services to query input and output fields for Training and Clustering results, which output in PMML format. This component is mandatory.   Training Microservice – The Training Microserver uses machine learning algorithms and techniques to identify meaningful patterns in a dataset and generate a predictive model. A number of parameters are available for tuning the type and combination of learning techniques used. The resulting model is output in PMML format. This component is optional, however, if the Validation Microserver is selected, the Training Microserver will be selected by default.   Validation Microservice – The Validation Microserver provides metrics to evaluate how well a model was able to predict outcomes for a specific goal variable in a dataset. Depending on the goal variable, validation metrics can include RMSE, Matthew's Correlation (MCC), Pearson Correlation, true and false positive rates, accuracy, or a confusion matrix. This component is optional, however if the Training Microserver is selected, the Validation Microserver will be delected by default.   Analytics Worker – The Analytics Worker does the processing work in the system. It picks up jobs from the Zookeeper system and performs the requested work. By default, only one worker is installed, but the number of workers can be scaled up after installation. Workers can process any type of submitted job request. This component is mandatory.   ThingWorx Analytics – Descriptive Analytics Microservice   Descriptive Analytics Microservice is an optional microservice included with the ThingWorx Analytics Server installation package, but can operate as a distinct and separate utility.   Descriptive Analytics provides a library of on-demand services that perform common statistical calculations and facilitate statistical monitoring on raw data. Descriptive Analytics is not required to generate prediction models, but output from these services can provide insights that improve your understanding of your data.   Update January 6, 2020   For information about ThingWorx Analytics 8.5 and newer, please refer to the HelpCenter - Analytics Server System Architecture for 8.5

Overview This document is targeted towards covering basic PostgreSQL monitoring and health check related system objects like tables, views, etc. This allows simple monitoring of PostgreSQL database via some custom services, which I'll attach at the end of this document, from the ThingWorx Composer itself. I'll also try to cover short detail on some of the services that are included with the Thing: PostgreSQLHealthCheck which implements Database ThingTemplate   Pre-requisite The document assumes that the user already has ThingWorx running with PostgreSQL as a Persistence Provider.   How to install Usage for this is fairly straight forward, import the Entities.twx and it will create required Thing which implements Database ThingTemplate and some DataShapes. Each Service under the Thing: PostgreSQLHealthCheck has its own DataShape. Feel free edit these services / DataShapes if you are looking to use output of these services  as part of your mashup(s).   Make sure to edit the PostgeSQL's JDBC Connection String, Username & password under the configuration section in order to connect to your PostgreSQL instance under the Thing PostgreSQLHealthCheck which will be created when Entities.twx is imported (attached with this blog)   Note : Users can use these services to query non-ThingWorx related database created with PostgreSQL as part of the external JDBC connection.   Reviewing Services from Thing: PostgreSQLHealthCheck   1. DescribeTableStructure - Takes two inputs **Table Name** and the **Schema Name** in which the ThingWorx database tables exists both inputs have default values that can be modified to match your PostgreSQL schema setup and required table name - It provides information on a Table's structure, see below     2. GetAllPSQLConfig - Provides runtime details on all the configurations done in the postgresql.conf which are in-effect - For detail on pg_settings see Postgresql 9.4 Doc     3. GetAllPSQLConfigLimited Similar to GetAllPSQLConfig, however with limited information   4. GetAllPSQLRoles - Lists all the database roles/users - Also lists their access rights permissions together with OID - Helpful in identifying if the role is active/inactive or carries any limitation on the DB connections     5. GetPG_Stat_Activity - Part of the Statistics Collector subsystem for the PostgreSQL DB - Shows current state of the schema e.g. connections, queries, etc. - For more detail on the output refer to the PostgreSQL 9.4 doc   6. GetPSQLDBLocksInformation - Shows the kind of locks in effect on which database and on which relation (table) - Particularly useful in identifying the relations and what lock mode is enabled on them     7. GetPSQLDBStat - Shows database wide statistics - Like Commits, reads, block reads, tupples (rows) fetched, inserted, deadlocks, etc - For more detail refer to PostgreSQL 9.4 doc   8. GetPSQLLogDesitnation - Checks where the PostgreSQL server logs are directed to - I.e. stderr, csvlog or syslog - Default is stderr   9. GetPSQLLogFileName - Fetches the log PostgreSQL log file name and the filename format - E.g. postgresql-%Y-%m-%d_%H%M%S.log    10. GetPSQLLoggingLocation - Fetches the location where the logs are stored for PostgreSQL - e.g. pg_log, which is also the default location - Desired location for the logs can be done in the postgresql.conf file   11. GetPSQLRelationIndexes - Gets information on the Indexes - Information like index size, number of rows, table names on which the index is created   12. GetPSQLReplicationStat - Shows information related to the Replication on PostgreSQL DB - Applicable to the PostgreSQL DBs where replication is enabled   13. GetPSQLTablespaceInfo - Takes tablespace name as input (String DataType), service defaults to 'thingworx' - modify if needed - Fetches information like owner oid, tablespace ACL     14. GetPSQLUserIndexIO - Fetches index that are created only on the User created DB objects - Shows relations (table) vs index relations ids (index on table), together with their names - Also shows additional info like number of disk blocks read from this index & number of buffer hits in this index     15. GetPSQLUserSequencesIOStats - Fetches informtion on Sequence objects used on user defined relations (tables) - Number of disk blocks read from this sequence & buffer hits in this sequence     16. GetPSQLUserTableIOStat - Fetches disk I/O information on the user created tables     17. GetPSQLUserTables - Fetches all the user created tables, together with their name, OID Disk I/O Last auto vacuum , vacuum Also lists the amount of rows each relation (table) has   Finally The attached entity has some additional service not yet covered in this blog, as they are minor services. Therefore for brevity of this blog I've left them out for now, feel free to explore or enhance this. I will continue to look for any additional services and will enhance this document and the entities belong to this.    If you are looking to enhance this feel free to fork from twxPostgreSqlHealthCheck over Github.

This video walks through the dataset requirements when working with time series in ThingWorx Analytics Server.  Starting release 52.2 - ending release 8.2.   Also view: - Written version of those steps mentioned in the video - Help Center  

I always find it difficult to remember which version of software is supported with which version of ThingWorx, so I created a table for my reference. I hope this is also helpful to other people.   　 Oracle JDK Tomcat Database Options Memo PostgreSQL Neo4J H2 Microsoft SQLServer SAP HANA DetaStax Enterprise Edition ThingWorx 6.5 1.8.0(64-bit) 8.0.23(64-bit) 9.4.4 embedded N/A N/A N/A N/A IE 10 ThingWorx 6.6 1.8.0(64-bit) 8.0.23(64-bit) 9.4.4 embedded N/A N/A N/A N/A 　 ThingWorx 7.0 1.8.0(64-bit) 8.0.23(64-bit) 9.4.? embedded N/A N/A N/A N/A 　 ThingWorx 7.1 1.8.0_92-b14(64-bit) 8.0.33(64-bit) 9.4.5 embedded N/A N/A N/A 4.6.3 　 ThingWorx 7.2 1.8.0_92-b14(64-bit) 8.0.33(64-bit) 9.4.5 embedded embedded N/A N/A 4.6.3 IE 11 and later ThingWorx 7.3 1.8.0_92-b14(64-bit) 8.0.38(64-bit) 9.4.5 embedded embedded N/A SPS 11, 12 4.6.3, 5 　 ThingWorx 7.4 1.8.0_92-b14(64-bit) 8.0.38(64-bit) 9.4.5 embedded embedded 2014 and later SPS 11, 12 4.6.3, 5 　 ThingWorx 8.0 1.8.0_92-b14(64-bit) 8.0.44(64-bit), 8.5.13(64-bit) 9.4.5 embedded embedded 2014 and later SPS 11, 12 4.6.3, 5 　 ThingWorx 8.1 1.8.0_92-b14(64-bit) 8.0.44(64-bit), 8.5.13(64-bit) 9.4.5 embedded embedded 2014 and later SPS 11, 12 4.6.3, 5 　 ThingWorx 8.2 1.8.0_92-b14(64-bit) 8.0.47(64-bit), 8.5.23(64-bit) 9.4.5 embedded embedded 2014 and later SPS 11, 12 4.6.3, 5 　 ThingWorx 8.3 　 　 　 　 　 　 　 　 　

This video go through the steps required to use the Creo Insight extension: - Download and install the required extension - Set required config.pro options - Create provider in Analytics Manager - Publish sensor from Creo - Create analysis Event in Analysis Manager - Retrieve sensor values from ThingWorx in Creo     See also: - https://www.ptc.com/en/support/article?n=CS277514  for a  written version of those steps. - Creo Help Center  

Thing Subscription This post is intended for novice ThingWorx users who wants to understand what the definition of Thing Subscription is and the overall purpose of using Thing Subscriptions.   Definition of a Thing Subscription? A Thing subscription is a script(JavaScript) that is called each time an event occurs. Events are property states which are of end users interest (e.g. temperature) and therefore indicators to kick off some functionality in a Thing subscription when any action needed. Events can e.g. be triggered by an Alert that detects a change or an anomaly in property values. The Thing subscription is explicitly linked to an event and when the event is fired the data is being passed to the subscriber.    Why Use a Thing Subscription? Imagine your machine is running 24 hours 7 days a week with supervised human interaction. If a pump temperature exceeds accepted value it needs to be regulated by the manufacturing department. But no one in the department knows when the temperature will exceed accepted value or drop suddenly therefore, the machines is always sporadically physically supervised by humans which leads to heavy costs for the manufacture. With a Thing Subscription a notification alert email can be sent directly to the department manager who acts based on the email notification.   Thing Subscription must have A Thing subscription must have defined a rule which gets executed when an event occurs. The definition of the rule may accommodate any appropriate business logic.   Thing Subscription example process In this scenario Thing subscription is using a predictive analytics model to detect Data Change or any anomaly values going through a Thing Property. So, based on historical data including failure information, a predictive analytics model begins to analyze run-time values from individual Things/properties to the analytics server. The predictive analytics model detects a pattern which detects past failures, when the analytics model predicts a failure/event based on the analyzed patterns an action is being fired via a Thing subscription. That action could be for ThingWorx to create a service ticket or send a notification email to the service department.   Example of a simple Thing Subscription set-up without using Analytics model to analyze data but instead a build-in ThingWorx alert Below example of Thing Subscription will send a notification email when temperature exceeds defined values from ThingWorx alert configuration. Prerequisites; it is necessary to have a mail server extension imported into the ThingWorx Composer this enables the service department to receive the email notification when an event have occurred. The extension can be downloaded from the marketplace. 1. Create a Thing with the MailServer[i] as the Base Thing template.     2. Create a new Thing and add Properties together with an alert that is triggered when the value exeeds user defined temerature.   3. Enable the Thing Subcriptions by Select Subscription and click +Add Make sure to mark the checkbox Enabled Selecting your Event name and your Property name In the right side of the screen you can enter your script/function that will notify ThingWorx email service to create the email notification Select Done and Save   4. Enable Email notification by selecting Services Provide an name Select Me/Entities Mark Other entity Find your Thing where the MailServer is the Thing Template   5. Then find the SendMessage snippet/script and fill out the snippet with your personal information.   [i] View this blog for more information on how to install the MailServer

    Go beyond functional application development and learn how you can dress up your UI to enhance the user experience. In this webinar, we'll show you how to implement common design elements throughout your UI including custom logos, color schemes, and styles.   In this video, UI expert Tsveta Saul will demonstrate valuable tips and tricks for making a sophisticated IoT application with ThingWorx, including: Masters and Menus - enhance your app framework with personalized menu titles, backgrounds, and custom headers Layout and Labels - create consistency in your application by combining commonly used widgets State and Style Definitions - define widget styles to illustrate your brand Images - integrate visuals to describe thousands of words' worth of design and development specifications Watch the recording above, and download this sample Mashup containing all the data and entities shared in the video.   Q&A   We didn’t have time to get to all of the questions during the live webcast, but we’ve answered them here on our blog. Have any additional questions? Please leave us a comment. WHERE ELSE CAN STATE DEFINITIONS BE APPLIED? State Definitions can be applied to the Map Widget. Not only can you add color-coded pins, but also images that describe certain types of assets, for example. You can also add State Definitions to your Time Series Chart. For instance, you can color-code for values that exceed a certain threshold.   ARE THERE ANY IMPORTANT OR HIDDEN PROPERTIES THAT I SHOULD KNOW ABOUT? One feature to take note of is the Z-index, which helps you control the position of a widget in relation to another widget. This option can control whether users see the specific widget or not. ThingWorx has a robust a system that handles permissions – from users and groups to organizations. There’s a lot of control as to who can see what either through Model, or through specific services and conditions inside the Mashup.   HOW CAN I DISABLE A USER FROM SEEING SOMETHING OR DOING A CERTAIN ACTION? Again, that is handled through the security system of ThingWorx. There are multiple ways of authenticating users from other systems. You can create services that are related to your Things or user session parameters, and define whether a user can see those parameters or not.   HOW CAN WE CHANGE DATE FORMAT FOR X-AXIS IN TIME SERIES CHART? This property is available within the Time Series Chart Widget. It uses standard programming language – you can add or delete in the property.   CAN LAYOUT SPACING BE A PERCENTAGE? The spacing within the layout (none, 5px, 10px, 15px, or 20px).

  Based on real use cases and industry-leading solutions, this webinar looks at how developers can deliver valuable analytical outputs through experiences that ensure easy consumption of trusted analyses.   By taking a deep dive into a range of the analytics capabilities within ThingWorx, we will demonstrate how you can visualize complex analytic outputs to help your users understand what really matters in your data - and ultimately make quick, insightful business decisions.   Q&A   We didn’t have time to get to all of the questions during the live webcast, but we’ve answered them here on our blog. Have any additional questions? Please leave us a comment.   THIS SESSION TALKED ABOUT CONNECTIVITY WITH OTHER ANALYTICS PROGRAMS, SUCH AS R. HOW WOULD THINGWORX INTERACT WITH THE R PLATFORM? There are multiple reasons for using R and other analytics tools. Say you’re using R to build a predictive model, for instance—you can interact with data and load that information into our Analytics Server, even outside Analytics Builder. You can also interface in script within ThingWorx to run certain calculations driven by R. Moreover, even if you are building a model in R or any other tool, you can use Analytics Manager to operationalize data coming in from a Thing model that’s being scored against variables created elsewhere. Our ultimate goal is not to migrate you away from the tools you’re already using, but rather augment the development experience with ThingWorx integration.   HOW WOULD A DEVELOPER GO ABOUT CREATING & OBTAINING JSON AND CSV DATA FOR ANALYSIS? In terms of creating a historical data view, there are two separate methods. There’s creating a descriptive view for which you are using to create the model, and then there is operationalizing the model. On the operational side, it’s data coming from the Thing model being scored against the actual predictive model that’s created. For the descriptive view, the tool of choice ultimately boils down to organizational preference. How you load represented data into the Analytics Server is completely based on the tools with which you work.   CAN YOU EXPLAIN IN DETAIL THE STEPS FOR CONNECTING A MACHINE TO THE THINGWORX ANALYTICAL MODEL, INCLUDING HOW TO DEFINE THE DATA COMING FROM THE MACHINE TO CREATE THE MODEL? Absolutely. I’d encourage you to go check out our new Operationalize an Analytics Model developer guide available on the ThingWorx Developer Portal. In just 30 minutes, you’ll learn how to use Analytics Manager and ThingPredictor to automatically perform analytical calculations.   OUR ORGANIZATION SEES FEATURE ENGINEERING AS A KEY PART OF THE DATA ANALYSIS PROCESS. DO THINGWORX ALGORITHMS HANDLE FEATURE ENGINEERING INTERNALLY? Yes. There’s feature engineering in terms of getting a dataset ready for consumption. The technology ThingWorx provides is being able to automatically sift through the data and use various features to guide the selection of algorithms. Feature enrichment is what’s really powerful about our supervised machine learning capabilities.    HOW DO I INCORPORATE ADVANCED STYLING IN MY UI, LIKE ANIMATIONS AND RESPONSIVE BEHAVIORS? The standard way of achieving advanced styling in ThingWorx is to leverage the Media Entities, Style and State Definitions. Many widgets, such as the Value Display and the Shape Widget, have out-of-the-box ability to take a State Definition and apply advanced styling for things like severity of risks, etc. Watch our IoT Application Makeover webcast for more information about this topic.   DO YOU HAVE ANY RECOMMENDATIONS FOR GUARANTEEING DESIGN CONSISTENCY IN MY IOT APPLICATION? For non-designers: to keep your design clean and consistent, it is important to properly manage your Style Definitions. Define styles and stick to re-using them; don’t have five different styles for showing model accuracy, for instance. I would recommend creating 5-10 styles for text, and then from there choosing a color scheme for things like buttons, labels, charts, grid headers, and other elements where you need a vibrant color.

  Create compelling, modern application user interfaces in ThingWorx with the latest enhancements to our Mashup visualization platform - Collection and Custom CSS.   In this webinar with IoT application designer Gabriel Bucur, we'll show how the new Collection widget makes it easy to replicate visual content in your UI for menu systems, dashboards, tables, and more. You'll learn about several of the 60+ configuration properties available for collections, many of which offer input/output bindings for dynamic flexibility.   Gabriel will also demonstrate the styling and UX power of the latest feature in the Next Gen Composer, which allows you to write classes and CSS for your Mashups, masters, and widgets.   Watch the recording above, and download this sample Mashup containing all the data and entities shared in the video.   Q&A   We didn’t have time to get to all of the questions during the live webcast, but we’ve answered them here on our blog. Have any additional questions? Please leave us a comment.   WILL PTC CONTINUE SUPPORT FOR THE REPEATER WIDGET IN THINGWORX 8.2, OR WILL IT BE REMOVED? The Repeater Widget will not removed. However, due to limited performance in various browsers, switching to the Collection is highly recommended.   WHAT’S THE DIFFERENCE BETWEEN REPEATER AND COLLECTION, AND ARE THERE PROS AND CONS FOR EACH WIDGET? The Collection is an advanced widget that allows you to contain a series of repeated Mashups within a collection. Its functionality is similar to the Repeater Widget, but contains more properties that provide additional options and better performance, especially when handling large amounts of data.   IS IT POSSIBLE TO ADD A DRAG AND DROP ACTION TO LISTS OR REPEATERS, E.G. DRAGGING AN ELEMENT FROM ONE CONTAINER TO ANOTHER? Drag and drop functionality is not available in the Collection Widget at this time. It is, however, in consideration for future ThingWorx releases.   IN THE EVENT I HAVE MORE THAN ONE MASHUP (FOR EXAMPLE, MASHUP A AND MASHUP B), CAN I BIND DIFFERENT PROPERTIES TO THE SAME MASHUP PARAMETER ACCORDING TO THE MASHUP NAME? The MashupName row goes to the MashupNameField in the collection, where you’ll  have a dropdown after you populate it with the InfoTable that contains the MashupName. You can put all the bindings there, even if you don't use them in all the Mashups. For example: {"valueA":"MashupA","valueB":"MashupB"}   IS IT POSSIBLE TO ORDER SECTIONS HORIZONTALLY IN THE COLLECTION? Sections can only be ordered vertically at this time.   WHAT IS THE DIFFERENCE BETWEEN GLOBAL PROPERTIES AND SESSION VALUES? Global Properties are only available within the Collection. These properties offer a way to control Things from other widgets with which the Collection is displaying.   IF THERE ARE MULTIPLE COLLECTIONS, DO THEY SHARE THE SAME SET OF GLOBALPARAMETERS? No. If you defined a Boolean on your Collection, when you drag the Boolean output from a checkbox on the Collection you will see that you can bind it to that defined Boolean in the GlobalParameters.   WHEN USING CUSTOM CSS, DO YOU HAVE TO DEFINE STYLING FOR EACH ELEMENT, OR CAN YOU CREATE A STYLE THING WITH CSS? Widgets differ in functionality, so the same class might not apply to the same widget. However, if you define a styling in CSS for a button, for example, you can apply that class on any button you want.   DOES CUSTOM CSS ALWAYS OVERRIDE THE WIDGET STYLES? Yes. That is the essential purpose of custom CSS integration – to rewrite styles.   IF YOU HAVE TWO CONFLICTING STYLES – ONE IN CSS AND THE OTHER IN A STYLE DEFINITION – WHICH ONE TAKES PRECEDENCE? CSS will typically rewrite the ThingWorx styles; however, it depends on the specificity of the CSS target definition. For example: “.button-element" will be overwritten by ".default-button .button-element". Visit https://developer.mozilla.org/en-US/docs/Web/CSS/Specificity for more details regarding this topic.   CAN I RESIZE MY WIDGETS DURING RUNTIME? The size of the widget is determined by the CSS, and how it renders in ThingWorx. While you can bind different sizing classes to the CustomClass property of the widget, resizing with your mouse is not available at Runtime.

  The data in your industrial IoT application is only valuable if it tells a story. As a developer, you need to consider the logical connections between graphical elements and business data and determine how users want to consume the information. With the Mashup Builder in ThingWorx, you can rapidly create a custom visualization that displays data from your connected devices. These easy-to-configure widgets deliver real-time data functionality at your fingertips - streamlining, processing, and displaying valuable information for your application users.   In this video, you'll learn how to create immersive, interactive visualizations by utilizing dynamic charts and graphs in your GUI. ThingWorx technical engineer Jason Wyatt demonstrates how to: Create a UI with ThingWorx Composer and Mashup Builder Incorporate visual displays that highlight business data requirements Supply data to components in your Mashup leveraging pre-built widgets and services Implement the Time Series Chart, Pie Chart, Open Street Map, Location Picker, Auto Refresh, Textbox, and Button widgets Watch the recording above, and download this sample Mashup containing all the data and entities shared in the video.   Q&A   We didn’t have time to get to all of the questions during the live webcast, but we’ve answered them here on our blog. Have any additional questions? Please leave us a comment. THIS UI CAPABILITY CLEARLY IS USEFUL FOR PROTOTYPING. AT WHAT SCALE AND / OR COMPLEXITY DO YOU RECOMMEND USING TRADITIONAL PROGRAMMING METHODS? The ThingWorx platform does not require users to utilize the Mashup Builder to create a frontend for their application. Some PTC customers use ThingWorx for the backend and rely on a traditional HTML team to design a custom UI. In that scenario, ThingWorx provides the Edge connectivity and backend storage/organization/business-logic. But, that said, the Mashup Builder can be used to develop production-level sites, especially if you customize State and Style definitions and create a Master mashup. WHAT IS THE DIFFERENCE BETWEEN GETIMPLEMENTINGTHINGS VS. GETIMPLEMENTINGTHINGSWITHDATA VS. QUERYPROPERTYHISTORY? GetImplementingThings provides a list of all Things instantiated from a particular Thing Template. GetImplementingThingsWithData provides the current property values associated with the Things. QueryPropertyHistory provides historical Property values. WHAT ARE THE POSSIBLE SCENARIOS WHERE SERVICEINVOKECOMPLETED CAN BE TRIGGERED? ServiceInvokeCompleted is an Event that occurs when a Mashup Data Service completes. We recommend you use it in any situation where timing is important for proper application execution. For instance, in the application created for the webinar, I wanted QueryPropertyHistory to run after SetProperties was complete. Therefore, I used ServiceInvokeCompleted to trigger the second Service. IS THERE ANY WAY TO DISPLAY TWO INFO TABLE DATA IN SINGLE GRID? This isn't possible by default, but as a workaround you could make a custom Service to combine two InfoTables into one; and then you could apply the combined data to the Grid Widget. CAN I RESIZE MY WIDGETS DURING RUNTIME? You can change any Widget Property at runtime that accepts an incoming data-bind. You can tell which Widget Properties accept dynamic data by looking at the Property itself in the bottom-left section of the Mashup Builder. If the Property has a left arrow pointing at the Property name, then you can bind it to dynamic data and change it during runtime. For Widget Properties where this isn't possible (such as some Widgets' Width and Height), you can apply custom CSS. IS THERE AN UNDO FEATURE AVAILABLE ON MASHUP BUILDER? Currently, no, but I believe Undo is a feature request on the R&D radar. HOW DO YOU GET MASHUP INFORMATION UP TO THE MASTER MASHUP? You can pass information between two Mashups when one Mashup pushes a change down to a Thing, then the other Mashup may read that data from that same Thing. Additionally, Mashup Parameters and Session Variables can act like Global Variables that are accessible across multiple Mashups. DOES THE OPAQUE OR MAKING BG COLOR TRANSPARENT WORK? You may set certain Widgets' style in such a way that the Widget itself is visible, but the background of the Widget is transparent. CAN YOU SHOW AGAIN HOW YOU ADDED THE TEXTBOX TO THE PROPERTIES OF SERVICES? You can view the recording of the webinar to see how I made an invisible TextBox set the MaxItems Property of the QueryPropertyHistory Mashup Data Service. I used an invisible TextBox to get a static number, then applied that number to the MaxItems Parameter of QueryPropertyHistory in order to change the Service's functionality. WHAT IS THE PURPOSE FOR SETPROPERTIES? SetProperties is one of several Mashup Data Services that sends information from the Mashup to the backend. DO WE HAVE FILTERS ON THE PIE CHART FOR CHANGING THE VIEW WHEN A VALUE IS SELECTED IN THE FILTER? Yes, there are several operations you may perform when a section of a Pie Chart is selected. I had originally intended to show how you can change the displayed color when you select a particular section of the Pie Chart, but I unfortunately ran out of time. In addition, all three of the display Widgets were tied together: when I selected a section of the Pie Chart, the same piece of data was selected on the Map and Time Series (and vice versa for clicking on the other two). You can import the sample Mashup into your Composer to view the configuration settings. IS THERE WAY TO SET PROPERTIES AND GET DATA THROUGH EMAIL USING THINGWORX? Yes, there are several ways to push information to the ThingWorx backend. For instance, you could have an entirely external process which strips data from an e-mail and then makes a REST call to ThingWorx to archive the data or trigger some Service. And, yes, ThingWorx supports sending and receiving e-mail through an Extension which you may download for free from the ThingWorx Marketplace. CAN THE MAXITEMS OF THE QUERYPROPERTYHISTORY BE APPLIED TO A COMBO BOX / LIST, WHICH CAN BE SELECTED FROM THE MASHUP SCREEN ONLINE? If I'm understanding correctly, you're asking whether or not the MaxItems Parameter of the QueryPropertyHistory Mashup Data Service can be set dynamically. The answer is yes. For instance, that TextBox which I made invisible could have been left visible and given a Label of "Max Items to Display". It would still be tied to the QueryPropertyHistory Service in the same way. But when the TextBox has a new value entered, then that would change the Parameter configuration of QueryPropertyHistory to show whatever had been entered. You would still have to figure out how to call the QueryPropertyHistory Service again. Changing a Parameter just sets up how the Service will behave the next time it is called. DO YOU HAVE TIPS FOR MAKING A PRINT-FRIENDLY MASHUP? My only real recommendation would be to use a Static Mashup with a specific resolution (to ensure that everything fit on the page while still looking good), while also setting the Style of various Widgets such that everything was in gray-scale (or something similar) that would be easy on your printer's ink cartridge. HOW WELL DO THE STYLE DEFINITIONS AND CSS WORK WITH TWITTER'S BOOTSTRAP? I'm unfamiliar with Twitter's Bootstrap, but I do know that with Style Definitions and the new CSS functionality, you have a lot of control over exactly how your Mashup looks. So you should be able to configure your Mashup to comply if Twitter has some particular requirements. WHY DO WIDGETS NOT STICK TO THE MASHUP WHEN YOU DRAG THEM INTO A BUSY UI? I HAVE HAD WIDGETS 'FLY' BACK TO THE WIDGET LIST AND HAVE BEEN UNABLE TO GET THEM TO STICK. Unfortunately, that's a known issue. It has something to do with the fact that you're not allowed to drag-and-drop a new Widget on top of an existing Widget. Which is a little strange considering that you explicitly *ARE* allowed to stack Widgets on top of one another after they've been placed in the central Canvas areas. I believe that R&D is investigating. CAN YOU EXPLAIN THE DIFFERENT OPTIONS FOR MASHUP? I believe that this question has something to do with the options in the pop-up when you first create a new Mashup. A Responsive Mashup grows and shrinks to match the viewing-resolution, while Static stays at the resolution you specify. The other options, (Page vs. Template vs. Shape), have to do with a specific setup where you have a Mashup-in-Mashup design. For instance, you could subdivide a Responsive Mashup just as I did in the webinar, and then have one of those sub-sections be an entirely different Mashup. Template and Shape Mashups are used when you have the scenario I describe above with Mashup-in-Mashup, but you want the sub-Mashup to change based off some other metric, such as the selection of a Thing listed in a Grid Widget. You could use some Mashup Data Service like GetImplementingThings. That would return a list of every Thing instantiated from a Thing Template. You could then have a Grid which displays a list of every Thing returned by the GetImplementingThings Service. You could then have a Template Mashup stored within every Thing instantiated from that Template. Whenever a Thing is selected from the Grid, it displays the Template Mashup for that specific Thing. HOW DO YOU HIDE TOOLBAR WHICH ALLOW USER TO SELECT "SHOW/HIDE LOG", "SHOW/HIDE LOG", "RELOAD", ETC. ON YOUR MASHUP SCREEN? Many Widgets have a ""Visible"" Property which may be dynamically set via some other criteria. For instance, you could have a Checkbox Widget, which is great for Boolean values like the Visible Property. You could tie the State Property of the Checkbox Widget to a different Widget's Visible Property. When the Checkbox is checked, the other Widget is visible. When the Checkbox is unchecked, then the other Widget becomes invisible. I WOULD LIKE TO KNOW THE CHALLENGES IN RESPONSIVE MASHUPS VS. STATIC MASHUP DEVELOPMENT. ALSO NEED THE LIST OF WIDGETS NOT SUPPORTED BY THE RESPONSIVE MASHUPS. WHAT IS NEW IN THINGWORX 8.2 FOR RESPONSIVE MASHUPS? The main challenge of a Responsive Mashup is that it's almost necessary to test the Mashup at each resolution that you believe your users may be viewing the page. Responsive does a good job of stretching and shrinking… but this can also lead to undesirable situations where you have scroll bars because the viewing-resolution is too small for everything to fit. The main challenge of a Static Mashup is that it really only works at the specific resolution you set. If you have a 300x200 px Static Mashup, it will essentially be unreadable on a 4k display. As for a list, some Widgets *AREN'T* Responsive. The TextBox Widget will not grow and shrink as the viewing-resolution changes, for instance, but you can still use a TextBox in a Responsive Mashup. The big new item for Mashups in 8.2 was the inclusion of CSS and the Collections Widget. Either or both may be used in any Mashup, regardless of whether it's Responsive or Static.

There are many choices in life and ThingWorx offers some persistence provider options as well. As of ThingWorx release 8.2, five Database options are provided. 1 PostgreSQL  9.4.5 minimum 2 DataStax Enterprise Edition 4.6.3,5 3 SAP HANA  SPS 11, 12 4 Microsoft SQL Server 2014 and later 5 H2 (version info is not available, maybe because it's an embedded?) H2 is for small scale, mainly for testing purpose, PostgreSQL and Microsoft SQL Server are for middle scale and finally DataStax Enterprise Edition is for big scale. I don't have enough information about SAP HANA so would like to leave it untouched in my comment... I don't have a number as to how many customers are using which database but my gut feeling tells me that PostgreSQL is a popular option, especially cost-wise. PostgreSQL offers powerful tools, such as logging and utilities, to troubleshoot issues.   In this post I would like to cover some useful information you can retrieve by using pgstattuple and pgstatindex of contrib module. By default, PostgreSQL takes a good care of fragmentation and reindex by itself. But in some cases, there's a situation that you want to review status of the database to narrow down the cause of your troubleshooting issue. There are many ways to achieve it but contrib module is provided to review stats of tables and indexes. As explained in this article, it is recommended to keep the number of records in value_stream and stream less than 100,000. That means you'll insert and delete many records when running ThingWorx. What happens then? If you delete(/update) a record in a table, PostgreSQL keeps the previous record in a page but mark it as deleted(and inserts a new record when it's update operation) If the number of those logically deleted records increases, PostgreSQL needs to access many pages of the table to obtain records which meets the criteria user might experience slow performance because of this Those logically deleted records will be ultimately removed from pages when vacuum is run   If you have installed contrib module and enabled it, you can review stats of tables by command below; select * from pgstattuple('stream');                             //This returns the stats of stream table select * from pgstatindex('stream_id_time_index');    //This returns the stats of an index on stream table   pgstattuple returns information below (I modified the format to make it more readable in this post) and meaning of each items are explained in the document .   table_len tuple_count tuple_len tuple_percent dead_tuple_count dead_tuple_len dead_tuple_percent free_space free_percent  8192 1 33 0.4 3  97 1.18 8004 97.71   Before obtaining the stat, I Inserted 4 records and Deleted 3 records and therefore it shows that tuple_count (the active record is 1) and dead_tuple_count (the logically deleted records are 3) and dead_tuple_percent is 1.18. If dead_tuple_percent is high, that means the table is not vacuumed or many DML were executed after the last vacuum operation and this could be the cause of the slow ststem performance.   * IMPORTANT: pgstattuple, pgstatindex consumes resources so it's recommended to run them during the maintenance window.   Takaaki

ThingWorx® Service Apps are easy-to-deploy, pre-configured role-based apps that enhance visibility, productivity, and performance across your serviceable assets. The apps provide seamless connectivity and real-time data visualizations in addition to providing remote access to service your assets remotely. Use our guides to learn how to remotely monitor and troubleshoot machine connectivity, detect exception conditions across all assets, and improve the overall efficiency of your service organization.   To learn more and to download our free, fully functioning 30-day trial, login to the ThingWorx Developer portal.

Predictive models: ​ Predictive model is one of the best technique to perform predictive analytics. This is the development of models that are trained on historical data and make predictions on new data. These models are built in order to analyse the current data records in combination with some historical data.   Use of Predictive Analytics in Thingworx Analytics and How to Access Predictive Analysis Functionality via Thingworx Analytics   Bias and variance are the two components of imprecision in predictive models. Bias in predictive models is a measure of model rigidity and inflexibility, and means that your model is not capturing all the signal it could from the data. Bias is also known as under-fitting.  Variance on the other hand is a measure of model inconsistency, high variance models tend to perform very well on some data points and really bad on others. This is also known as over-fitting and means that your model is too flexible for the amount of training data you have and ends up picking up noise in addition to the signal.   If your model is performing really well on the training set, but much poorer on the hold-out set, then it’s suffering from high variance. On the other hand if your model is performing poorly on both training and test data sets, it is suffering from high bias.   Techniques to improve:   Add more data: Having more data is always a good idea. It allows the “data to tell for itself,” instead of relying on assumptions and weak correlations. Presence of more data results in better and accurate models. The question is when we should ask for more data? We cannot quantify more data. It depends on the problem you are working on and the algorithm you are implementing, example when we work with time series data, we should look for at least one-year data, And whenever you are dealing with neural network algorithms, you are advised to get more data for training otherwise model won’t generalize.  Feature Engineering: Adding new feature decreases bias on the expense of variance of the model. New features can help algorithms to explain variance of the model in more effective way. When we do hypothesis generation, there should be enough time spent on features required for the model. Then we should create those features from existing data sets. Feature Selection: This is one of the most important aspects of predictive modelling. It is always advisable to choose important features in the model and build the model again only with important and significant features. e. let’s say we have 100 variables. There will be variables which drive most of the variance of a model. If we just select the number of features only on p-value basis, then we may still have more than 50 variables. In that case, you should look for other measures like contribution of individual variable to the model. If 90% variance of the model is explained by only 15 variables then only choose those 15 variables in the final model. Multiple Algorithms: Hitting at the right machine learning algorithm is the ideal approach to achieve higher accuracy. Some algorithms are better suited to a particular type of data sets than others. Hence, we should apply all relevant models and check the performance. Algorithm Tuning: We know that machine learning algorithms are driven by parameters. These parameters majorly influence the outcome of learning process. The objective of parameter tuning is to find the optimum value for each parameter to improve the accuracy of the model. To tune these parameters, you must have a good understanding of these meaning and their individual impact on model. You can repeat this process with a number of well performing models. For example: In random forest, we have various parameters like max_features, number_trees, random_state, oob_score and others. Intuitive optimization of these parameter values will result in better and more accurate models. Cross Validation: Cross Validation is one of the most important concepts in data modeling. It says, try to leave a sample on which you do not train the model and test the model on this sample before finalizing the model. This method helps us to achieve more generalized relationships. Ensemble Methods: This is the most common approach found majorly in winning solutions of Data science competitions. This technique simply combines the result of multiple weak models and produce better results. This can be achieved through many ways.  Bagging: It uses several versions of the same model trained on slightly different samples of the training data to reduce variance without any noticeable effect on bias. Bagging could be computationally intensive esp. in terms of memory. Boosting: is a slightly more complicated concept and relies on training several models successively each trying to learn from the errors of the models preceding it. Boosting decreases bias and hardly affects variance.     

Pgweb is a light weight cross platform client for PostgreSQL DBs written in Go. It can connect to both local & remote PostgreSQL instances behind firewall using native SSH tunneling both with password or ssk keys Due to no dependency on any other external library/utility all that is needed to run pgweb is to download it and run to connect to a working PostgreSQL server. Also works on RaspberryPi.   Note: It can also be used to connect to the CockroachDB instances, detailed in post Cockroach DB : An open source distributed SQL Database as external data store for ThingWorx   Getting Access Download pre-compiled binaries here for different platforms.   Building pgweb from source Use https://github.com/sosedoff/pgweb.git to clone the source and build it.   Additional Reads Pgweb Wiki Features List Usage Installing & Testing it After downloading the pgweb_windows_amd64.exe for windows platform (used for testing in this blog), start the pgweb utility using: pgweb_windows_amd64.exe   Note starting the pgweb utility from the command prompt shows the port on which pgweb is accessible e.g. This is when no flag is used to start up the pgweb utility, all default options used     Accessing the pgweb over localhost:8081 via a web browser     Connecting Pgweb allows for connecting to the PostgreSQL DB in variety of ways - e.g. using something like JDBC connection string like postgres://user:password@host:port/db?sslmode=mode. Additionally users can connect using standard hostname, port , username & password connection w/o SSL or via SSH tunneling.   Navigating around pgweb UI UI design is pretty straight forward and clean highlighting database name, all the tables available under that database and selected table's information on the left. To switch to different database running under the connected server click on the database name to see a drop down menu with names listed.     Selected table gets queried for all its data automatically displaying all its rows together with additional tabs for quickly reviewing the metadata about the table     Query Tab This is particularly interesting esp. for getting quick info on how the SQL query is going to perform. This could be used to analyze and generate the query plan showing preliminary data on how is it going to cost to run a particular query, how many times its going to loop over the complete table, what's going to be the execution time, etc   To get this information simply type in the sql statement under the Query tab and hit Explain Query button to generate this data. For larger tables it could take a while to get this information generated     The queried data then can be exported in formats : JSON, CSV & XML   Exporting table data out of the tables This is pretty handy feature allowing quick export of entire table's data set by right clicking on the table name and selecting one of the desired formats which includes JSON, CSV, etc.    

The Squeal functionality has been discontinued with ThingWorx 8.1, see ThingWorx 8.1.0 Release Notes   There might be scenarios where it should be disabled in earlier versions as well. This can be achieved e.g. with Tomcat Security Constraints. To add such a constraint, open <Tomcat>\webapps\Thingworx\WEB-INF\web.xml At the end of the file add a new constraint just before closing the </web-app> tag:   <security-constraint> <web-resource-collection> <web-resource-name>Forbidden</web-resource-name> <url-pattern>/Squeal/*</url-pattern> </web-resource-collection> <auth-constraint/> <user-data-constraint> <transport-guarantee>CONFIDENTIAL</transport-guarantee> </user-data-constraint> </security-constraint> Save the file and restart Tomcat. Accessing the /Thingworx/Squeal resource now will result in an error message:   HTTP Status 403 - Access to the requested resource has been denied   One scenario to be aware of is when the web.xml changes, e.g. due to updating ThingWorx or other manual changes. In such a case, ensure that the filter is still present in the file and taken into account.

Alerts via Anomaly Detection This documents objective is to provide information and links about alerts used for anomaly detection. This document covers following topics: What Is Anomaly Detection Implementing Anomaly Detection Creating an Anomaly Alert and Prerequisites Anomaly Stats Certainty Parameter Video Example On How To Create An Alert for Anomaly Detection Tips and troubleshooting What Is Anomaly Detection Anomaly Detection in ThingWorx is implemented via built-in ThingWatcher functionality. ThingWatcher detects anomalies by monitoring a data stream from a device, calculating an expected distribution of data, and validating that the current data point is a member of the expected distribution.   Implementing Anomaly Detection Anomaly Detection is enabled by default in ThingWorx. However, several steps are required to configure the functionality for your specific environment, including the prerequisite activities below.   Creating an Anomaly Alert and Prerequisites   Configuring Anomaly Detection to monitor a stream of data. For information about setting up Anomaly Detection, view Preparing ThingWorx for Anomaly Detection. Anomaly Stats Anomaly Alert Statuses moves through several statuses as it works its way through the corresponding phases. Initialized Calibrating Training Buffering Monitoring Failed Certainty Parameter The Certainty Parameter when implementing anomaly detection requires a number of factors to consider. At its most basic, ThingWatcher functionality compares two sets of data, a validation set (collected during the Calibrating phase) and a test dataset (data streaming from a remote device). ThingWatcher tries to determine the likelihood that the distribution of values in the test dataset is from the same distribution of values contained in the validation dataset. The accuracy of the model plays a large role in this determination, but so does the Certainty parameter used for the statistical analysis of the two data sets.   Video Example On How To Create An Alert for Anomaly Detection Anomaly Detection Part 1. Create connectivity between KEPServer and ThingWorx Platform. Anomaly Detection Part 2. Configure Anomaly Alert to bind simulated data coming through KEPServer for Anomaly Detection. Anomaly Detection Part 3. Viewing data via Anomaly Mashup. Tips and troubleshooting Diagnose and fix the most common issues that may be encountered when working with ThingWatcher. It cannot be stressed strongly enough that you should be familiar with your data including the average time interval between data points, and the collection duration and certainty threshold you specified.