IoT & Connectivity Tips

ThingWorx Analytics Sample Data for the 8.1 Release

The following Expert Session videos are now available for viewing within the ThingWorx Community: ThingWorx Analytics Installation - This Expert Session will walk you through the complete installation of ThingWorx Analytics from the Prerequisites to Confirming the Installation is successful and all steps in between. The first half of the video gives a breakdown of the components and the process of the installation with the second half being an actual Demo of the Installation.     ThingWorx Analytics API Overview - This Expert Session is designed to help beginners get up and running with ThingWorx Analytics. It covers basic concepts like: What are APIs, how to configure the metadata file, and a live Demo that shows you how to interact and use ThingWorx Analytics in real time. This Expert Session would also be useful for experienced users who need a refresher course.   Decision Tree, ThingWorx Analytics Builder - This Expert Session reviews the concept of “Decision Trees” and the functionality that is available in ThingWorx Analytics Builder. First, you will learn how to create and upload a dataset in ThingWorx Analytics Builder.  After that, it shows you how to train a model and score on the model that was just generated. It then goes into detail on how the prediction learner "Decision Tree" operates and classifies inputs.   Use Case Identification - This Expert Session goes over ways to identify and develop a successful use case for ThingWorx Analytics. The example use case presented here is on employee retention in a fictional company with the goal of maximizing employee retention . This presentation will provide you with all the fundamentals you need to develop your own ThingWorx Analytics use cases from the ground up.   ThingWorx Analytics Signals - This Expert Session will provide you with an in depth explanation behind how Signals are calculated in ThingWorx Analytics, what purpose they serve, and why we use them.  Some basic mathematical concepts are discussed so viewers will have a better idea of how ThingWorx Analytics operates behind the scenes.   Related Links For more information, you can visit a new space dedicated to these helpful technical videos.   Additional Expert Sessions will be highlighted here in the ThingWorx Community every few weeks. Visit the Online Success Guide to access additional information about ThingWorx training and services.

Starting with the 8.1 release, the architecture of ThingWorx Analytics has changed from being a single sever to being split into several independent microservices.  This has been done to allow services to run concurrently. It also prevents issues with one microservice from affecting the others. Overview The new Analytics Server Architecture consists of a suite of 9 microservices: Data Clustering Profiling Signals Training Prediction Validation Presciptive Results All of the microservices work together to create a similar experience for users as it was in the past. The data that is uploaded and generated by the Analytics Server is stored directly in a file system, instead of a Postgres Database like it was in the past. Closer Integration with ThingWorx Please note that ThingWorx Foundation is required to be installed and operating before Installing Analytics.  During the install you will be asked to supply IP Address of the ThingWorx Instance that will be used for Analytics.  At this step, the AnalyticsServerThing is configured which allows the user to interact with Analytics Server through ThingWorx.  All of the configured microservices are represented as Things under the AnalyticsServerThing. This is because ThingWorx Analytics has become a native part of ThingWorx Foundation functionality and is dependent on ThingWorx for user interaction.  Because of these changes, there is no longer a direct ThingWorx Analytics Server REST API. Support for accessing the services via REST calls is now provided through the ThingWorx Core REST API layer.  Because of this, a new URI pattern is required moving forward. One other update from the older versions is that the requirement to use application keys and Application IDs are no longer necessary.  This should come as a welcome relief as the Application keys and IDs were the source of issues for users who may have misplaced them etc. Less Data-Centric In the old versions, jobs, models, signals, etc. were all tied to the dataset.  So there was no way to a model from one dataset to the other. With the new architecture, this is no longer the case you are able to move a model from one dataset to the other seamlessly.  Please note that when moving a model from one dataset to the other, it must have the same metadata between each of the datasets.  This is because a model created to increase efficiency in a factory would provide no insight on a dataset that monitors the soil moisture in a corn field. Updates to Metadata Although going over the exact changes to the Metadata is out of scope for this post, it is worth mentioning. For more details on the changes, please follow this link. Summary In conclusion, the new architecture of ThingWorx Analytics was done to increase scalability and to produce a more robust system.  The new release is much more integrated into the ThingWorx Platform to increase the ease of use from the previous releases.  It is much less data-centric than it was in the past and geared more to the solutions themselves. 

This video covers the new features of ThingWorx Analytics Builder 8.1   Updated Link for access to this video: What's New in ThingWorx Analytics Builder 8.1

In this video we are going through a few steps to ensure the installation process went fine.   Updated Link for access to this video:  Validating a ThingWorx Analytics Server 8.1 Installation

In this video we are walking through the installation steps of ThingWorx Analytics Server 8.1. This cover the Native Linux installation though the steps will be similar for a docker installation on Windows or Linux.   Updated Link for access to this video:  Installing ThingWorx Analytics Server 8.1 - Native Linux

This video shows the commands to execute to deploy the training and results microservices as docker container. This is based on Docker Toolbox to highlight the specific settings required on Toolbox.   Updated Link for access to this video:  Deploying Training & Result Microservices via Docker Containers for Anomaly Detection

Overview Time-series predictive models generated by ThingWorx Analytics Server in Analytics Manager will have additional columns in their dataShape generated by ThingPredictor. These columns are known as “transformation fields” and are used for internal processing but are not necessary for inclusion in the DataShape.  So there is no need to worry about mapping all these additional fields since it will be handled internally by ThingPredictor.  There is one addition step that the user must take which is detailed below. Step to Import: Edit the DataShape generated by ThingPredictor to match the format of the data that was provided during the model training process. In other words, remove all the transformation fields from the DataShape.

In this video we show: - how to deploy the microservices via jar files - how to setup ThingWorx to use those microservices for anomaly detection   Updated Link for access to this video:  ThingWorx Analytics: Deploying Training and Result Microservices via jar files for Anomaly Detection

This video is the 3 rd part of a series of 3 videos walking you through how to setup ThingWatcher for Anomaly Detection. In this video we will use Anomaly Mashup to visualize data received from my remote device.   Updated Link for access to this video:  Anomaly Detection 8.0:  Viewing Data via Anomaly Mashup:  Part 3 of 3

This video is the 2 nd part of a series of 3 videos walking you through how to setup ThingWatcher for Anomaly Detection. In this video you will learn how to use “Discover UI” from the “New Composer” to bind simulated data coming through KEPServer for Anomaly Detection.   Updated Link for access to this video:  Anomaly Detection 8.0: Configuring Anomaly Alerts:  Part 2 of 3

This video is the 1 st part of a series of 3 videos walking you through how to setup ThingWatcher for Anomaly Detection. In this first video you will learn the basics of how to create connectivity between KEPServer and ThingWorx Platform.   Updated Link for access to this video:  Anomaly Detection 8.0 - Part 1: Connecting KEPServer to ThingWorx: Part 1 of 3

In this video we show a simple mashup and services in order to display the ThingPredictor's real time scoring results. This video applies to ThingWorx Analytics 7.4 to 8.1   Updated Link for access to this video:  Showing ThingWorx Analytics Manager's results (ThingPredictor) in a Mashup

This is part 3 out of 3 videos on Getting Started with ThingWorx Analytics During this video you will learn:   Executing a “Signals” Job Getting the results of the “Signals” Job Executing a “Training Model” Job Getting the results of the “Training Model” Job   Updated Link for access to this video:  Getting Started with ThingWorx Analytics: Part 3 of 3

This video is the 2nd part, of a series of two videos, walking you through the configuration of Analysis Event which is applied for Real-Time Scoring. This part 2 video will walk you through the configuration of Analysis Event for Real-Time Scoring, and validate that a predictions job has been executed based on new input data.   Updated Link for access to this video:  Analytics Manger 7.4: Configure Analysis Event & Real Time Scoring Part 2 of 2

This video walks you through the use of Analysis Replay to execute analysis events on historic data. This video applies to ThingWorx Analytics 7.4 to 8.1   Updated Link for access to this video:  Analytics Manager : Using Analysis Replay

This video is the 1st part of a series of two videos walking you through the configuration of Analysis Event which is applied for Real-Time Scoring. This 1st video demonstrate how to create a Template and Thing which allows the prediction model to score in real-time. Note that this video is just for demo purposes, customers who have ThingWorx, they of course already have their properties set-up. They just need to configure Analysis Event which is demonstrated in the part 2 video.   Updated Link for access to this video:  Analytics Manger 7.4: Create a Template & Thing for Real-Time Scoring Part 1 of 2

This video will walk you through the first steps of how to set-up Analytics Manager for Real-Time Scoring. More specifically this video demonstrate how to share your predictive model from Analytics Builder into Analytics Manger -and test the shared model.   Updated Link for access to this video::  ThingWorx Analytics Manager: Publish & Test a Predictive Model

This video will walk you through the first steps of how to set-up Analytics Manager for Real-Time Scoring. More specifically this video demonstrates how to create an Analysis Provider and start ThingPredictor Agent. NOTE: For version 8.1 the startup command for the Agent has changed view the command in PTC Help center.   Updated Link for access to this video:  ThingWorx Analytics Manager: Create an Analysis Provider & Start the ThingPredictor Agent                                  

In this part of the Troubleshooting blog series, we will review the process on how to restart individual services essential to the ThingWorx Analytics Application within the Virtual Machine Appliance.   Services have stopped, and I cannot run my Analytics jobs! In some cases, we have had users encounter issues where a system or process has halted and they are unable to proceed with their tasks. This can be due to a myriad of reasons, ranging from OS hanging issues to memory issues with certain components.   As we covered earlier in Part II, the ThingWorx Analytics Application is installed in a CentOS (Linux) Operating System. As with most Linux Operating Systems, you have the ability to manually check and restart processes as needed.   Steps to Restart Services   With how the Application is installed and configured, the services for functionality should auto start when you boot up the VM. You will have to verify that the Appliance is functional by running your desired API call.   If a system is not functioning as expected, you will receive an error in your output when you POST an API call. Some errors are very specific and you can search the Knowledge Database for any existing Knowledge Articles that may solve the issue.   For error messages that do not have an existing article, you may want to attempted the following   Method 1:   If you are encountering issues, and are unsure what process is not working correctly, we would recommend a full Application restart. This involves restarting the Virtual Machine Appliance via the command line terminal.   We would recommend that you use the following command, as root user or using SUDO, as this is known as a “Graceful restart” ​sudo reboot -h now   This will restart the virtual machine safely, and once you are back up and running you can run your API calls to verify functionality. This should resolve any incremental issues you may have faced.   Method 2:   If you want to restart an individual service, there is a particular start order that needs to be followed to make sure the Application is operating as expected.   The first step is check what services are not running, you can use the following command to check what is running and its current status: service –status-all   The services you are looking for are the following: Zookeeper PostgreSQL Server GridWorker(s) Tomcat   If a particular service is not on the running list, you will have to manually start them by using the service start command. service [name of service] start e.g. service tomcat start You may be prompted for the root password   You can verify that the services are operating by running the status check command as described above.   If you need to restart all services, we have a specific start order, this is important to do as there are some dependencies such as Postgres for the GridWorker(s) and Tomcat to use.   The start order is as follows: Zookeeper PostgreSQL Server GridWorker(s) Tomcat   After completing the restart, and verifying that the services are running, run your desired API call.

Concepts of Anomaly Detection used in ThingWatcher ThingWatcher is based on anomaly detection with the normal distribution. What does that mean? Actually,  normally distributed metrics follow a set of probabilistic rules. Upcoming values who follow those rules are recognized as being “normal” or “usual”. Whereas value who break those rules are recognized as being unusual. What is a normal distribution? A normal distribution is a very common probability distribution. In real life, the normal distribution approximates many natural phenomena. A data set is known as “normally distributed” when most of the data aggregate around it's mean, in a symmetric way. Also, it's extreme values get less and less likely to appear. Example When a factory is making 1 kg sugar bags it doesn’t always produce exactly 1 kg. In reality, it is around 1 kg. Most of the time very close to 1 kg and very rarely far from 1 kg. Indeed, the production of 1 kg sugar bag follows a normal distribution. Mathematical rules When a metric appears to be normally distributed it follows some interesting law. As does the sugar bag example. The mean and the median are the same. Both are equal to 1000. It’s because of  the perfectly symmetric “bell-shape” It is the standard deviation called sigma σ that defines how the normal distribution is spread around the mean. In this example σ = 20 68% of all values fall between [mean-σ; mean+σ] For the sugar bag [980; 1020] 95% of all values fall between [mean-2*σ; mean+2*σ] For the sugar bag [960; 1040] 99,7% of all values fall between [mean-3*σ; mean+3*σ] For the sugar bag [940; 1060] The last 3 rules are also known as the 68–95–99.7 rule also called the three-sigma rule of thumb When the rules get broken: it’s an anomaly As previously stated, When a system has been proven normally distributed, it follows a set of rules. Those rules become the model representing the normal behavior of the metric. Under normal conditions, upcoming values will match the normal distribution and the model will be followed. But what happens when the rules get broken? This is when things turn different as something unusual is happening. In theory, in a normal distribution, no values are impossible. If the weights of the bags of sugar were really distributed, we would probably find a bag of sugar of 860 g every billion products. In reality, we approximate this sugar bag example as normally distributed. Also, almost impossible value are approximated as impossible Techniques of Anomaly Detection Technique n°1: outlier value An almost impossible value could be considered as an anomaly. When the value deviates too much from the mean, let’s say by ± 4σ, then we can consider this almost impossible value as an anomaly. (This limit can also be calculated using the percentile). Sugar bags who weigh less than 920 g or more than 1080 g are considered anomalous. Chances are, there is a problem in the production chain. This provides a simple way to define maximum and minimum thresholds. Technique 2: detecting change in the normal distribution Technique n°2 can detect unusual distribution fast, using only some points. But it can’t detect anomalies who move from one sigma σ to another in a usual manner. To detect this kind of anomaly we use a “window” of n last elements. If the mean and standard derivation of this window change too much from usual then we can deduce an anomaly. Using a big window with a lot of values is more stable, but it requires more time to detect the anomaly. The bigger the window is the more stable it becomes. But it would require more time to detect the anomaly as it needs to aggregate more values for the detection.