IoT & Connectivity Tips

Parquet Data Format used in ThingWorx Analytics   Starting ThingWorx Analytics Version 8.1 Data storage will no longer require the installation of a PostgreSQL database. Instead, uploaded CSV data is converted to the optimized  Apache Parquet format and stored directly in the file system. This Blog explains some the features of Apache Parquet justifying this transition in ThingWorx Analytics Data Storage. features What is Apache Parquet: Apache Parquet is a column-oriented data store of the Apache Hadoop ecosystem. It is compatible with most of the data processing frameworks in the Hadoop environment. It provides efficient data compression and encoding schemes with enhanced performance to handle complex data in bulk. Below is an illustration of the Columnar Storage model: Apache Parquet Features and Benefits: Apache Parquet is implemented using the record shredding and assembly algorithm taking into account the complex data structures that can be used to store the data. Apache Parquet stores data where the values in each column are physically stored in contiguous memory locations.  Due to the columnar storage, Apache Parquet provides the following benefits: Column-wise compression is efficient and saves storage space Compression techniques specific to a type can be applied as the column values tend to be of the same type Queries that fetch specific column values need not read the entire row data thus improving performance Different encoding techniques can be applied to different columns Some advantages of using Parquet for ThingWorx Analytics: Apart from the above benefits of using Parquet which amount to higher efficiency and increased performance, below are some advantages that apply specifically to ThingWorx Analytics This change in ThingWorx Analytics from using a Database to using Parquet removes the limitations on the number of data columns the system can handle. It also allows for streamlining the dataset creation process. Since the data is converted to a Parquet format, there is no need to separately optimize the dataset. Even when new data is appended to an existing dataset, a new partition is added and re-optimization is optional but not required. Data could be appended easily so there is no longer a need to re-load the full Dataset when new Data values are added The illustration below shows the transition from Row-based Data Storage model VS the columnar based Storage of Parquet

  Operationalize an Analytics Model Guide Part 1   Overview   This project will introduce ThingWorx Analytics Manager. Following the steps in this guide, you will learn how to deploy the model which you created in the earlier Builder guide. We will teach you how to utilize this deployed model to investigate whether or not live data indicates a potential engine failure. NOTE: This guide’s content aligns with ThingWorx 9.3. The estimated time to complete ALL 2 parts of this guide is 60 minutes.    Step 1: Analytics Architecture   You can leverage product-based analysis Models developed using PTC and third-party tools while building solutions on the ThingWorx platform. Use simulation as historical basis for predictive Models Create a virtual sensor from simulation Design-time versus operational-time intelligence It is important to understand how Analytics Manager interacts with the ThingWorx platform.   Build Model   In an IoT implementation, multiple remote Edge devices feed information into the ThingWorx Foundation platform. That information is stored, organized, and operated-upon in accordance with the application's Data Model. Through Foundation, you will upload your dataset to Analytics Builder. Builder will then create an Analytics Model.     Operationalize Model   Analytics Manager tests new data through the use of a Provider, which applies the Model to the data to make a prediction. The Provider generates a predictive result, which is passed back through Manager to ThingWorx Foundation. Once Foundation has the result, you can perform a variety of actions based on the outcome. For instance, you could set up Events/Subscriptions to take immediate and automatic action, as well as alerting stakeholders of an important situation.       Step 2: Simulate Data Source   For any ThingWorx IoT implementation, you must first connect remote devices via one of the supported connectivity options, including Edge MicroServer (EMS), REST, or Kepware Server. Edge Connectivity is outside the scope of this guide, so we'll use a data simulator instead. This simulator will act like an Engine with a Vibration Sensor, as described in Build a Predictive Analytics Model. This data is subdivided into five frequency bands, s1_fb1 through s1_fb5. From this data, we will attempt to predict (through the engine's vibrations) when a low grease emergency condition is occuring.   Import Entities   Import the engine simulator into your Analytics Trial Edition server. Download and unzip the attached amqs_entities.zip file. At the bottom-left of ThingWorx Composer, click Import/Export > Import.     Keep the default options of From File and Entity, click Browse, and select the amqs_entities.twx file you just downloaded.   Click Import, wait for the Import Successful message, and click Close.   From Browse > All, select AMQS_Thing from the list.   At the top, click Properties and Alerts to see the core functionality of the simulator. NOTE: The InfoTable Property is used to store data corresponding to the s1_fb1 through s1_fb5 frequency bands of the vibration sensor on our engine. The values in this Property change every ten seconds through a Subscription to the separate AMQS_Timer Thing. The first set of values are good, in that they do NOT correspond to a low grease condition. The second set of values are bad, in that they DO correspond to a low grease condition. These values will change whenever the ten-second timer fires.   View Mashup   We have created a sample Mashup to make it easier to visualize the data, since analyzing data values in the Thing Properties is cumbersome. Follow these steps to access the Mashup. On the ThingWorx Composer Browse > All tab, click AMQS_Mashup.   At the top, click View Mashup.    Observe the Mashup for at least ten-seconds. You'll see the values in the Grid Advanced Widget change from one set to another at each ten-second interval.     NOTE: These values correspond to data entries from the vibration dataset we utilized in the pre-requisite Analytics Builder guide. Specifically, the good entry is number 20,040... while the bad entry is number 20,600. You can see in the dataset that 20,400 corresponds to a no low grease condition, while 20,600 corresponds to a yes, low grease condition.   Step 3: Configure Provider   In ThingWorx terminology, an Analysis Provider is a mathematical analysis engine. Analytics Manager can use a variety of Providers, such as Excel or Mathcad. In this quickstart, we use the built-in AnalyticsServerConnector, an Analysis Provider that has been specifically created to work seamlessly in Analytics Manager and to use Builder Models. From the ThingWorx Composer Analytics tab, click ANALYTICS MANAGER > Analysis Providers, New....   In the Provider Name field, type Vibration_Provider. In the Connector field, search for and select TW.AnalysisServices.AnalyticsServer.AnalyticsServerConnector.   4. Leave the rest of the options at default and click Save.   Step 4: Publish Analysis Model   Once you have configured an Analysis Provider, you can publish Models from Analytics Builder to Analytics Manager. On the ThingWorx Composer Analytics tab, click ANALYTICS BUILDER > Models.   Select vibration_model_s1_only and click Publish.   On the Publish Model pop-up, click Yes. A new browser tab will open with the Analytics Manager's Analysis Models menu.      4. Close that new browser tab, and instead click Analytics Manager > Analysis Models in the ThingWorx Composer Analytics navigation. This is the same interface as the auto-opened tab which you closed.   False Test   It is recommended to test the published Model with manually-entered sample data prior to enabling it for automatic analysis. For this example, we will use entry 20,400 from the vibration dataset. If the Model is working correctly, then it will return a no low grease condition. In Analysis Models, select the model you just published and click View.   Click Test.   In the causalTechnique field, type FULL_RANGE. In the goalField field, type low_grease. For _s1_fb1 through _s1_fb5, enter the following values: Data Row Name Data Row Value _s1_fb1 161 _s1_fb2 180 _s1_fb3 190 _s1_fb4 176 _s1_fb5 193 6. Click Add Row. 7. Select the newly-added row in the top-right, then click Set Parent Row. 8. Click Submit Job. 9. Select the top entry in the bottom-left Results Data Shape field. 10. Click Refresh Job. Note that _low_grease is false and and _low_grease_mo is well below 0.5 (the threashold for a true prediction).   You have now successfully submitted your first Analytics Manager job and received a result from ThingPredictor. ThingPredictor took the published Model, used the no low grease data as input, and provided a correct analysis of false to our prediction.   True Test Now, let's test a true condition where our engine grease IS LOW, and confirm that Analytics Manager returns a correct prediction. In the top-right, select the false data row we've already entered and click Delete Row. For _s1_fb1 through _s1_fb5, change to the following values: Data Row Name Data Row Value _s1_fb1 182 _s1_fb2 140 _s1_fb3 177 _s1_fb4 154 _s1_fb5 176 3. Select the top entry in the bottom-left Results Data Shape field. 4. Click Refresh Job. Note that _low_grease is true and and _low_grease_mo is above 0.5.                 5. Click Submit Job.         6. Select the top entry in the bottom-left Results Data Shape field.         7. Click Refresh Job. Note that _low_grease is true and and _low_grease_mo is above 0.5.          You've now manually submitted yet another job and received a predictive score. Just like in the dataset Entry 20,600, Analytics Manager predicts that the second s1_fb1 through s1_fb5 vibration frequencies correspond to a low grease condition which needs to be addressed before the engine suffers catastrophic failure.   Enable Model   Since both false and true predictions made by the Model seem to match our dataset, let's now enable the Model so that it can be used for automatic predictions in the future. In the top-left, expand the Actions... drop-down box.   Select Enable.     Click here to view Part 2 of this guide.   

There have been a number of questions from customers and partners on when they should use different tools for calculation of descriptive analytics within ThingWorx applications. The platform includes two different approaches for the implementation of many common statistical calculations on data for a property: descriptive services and property transforms. Both of these tools are easy to implement and orchestrate as part of a ThingWorx application. However, these tools are targeted for handling different scenarios and also differ in utilization of compute resources. When choosing between these two approaches it is important to consider the specific use case being implemented along with how the implemented approach will fit into the overall design and architecture of the ThingWorx environment. This article will provide some guidance on scenarios to use each of these approaches in ThingWorx applications and things to consider with each approach.   Let's look at the two different approaches and some guidelines for when they should be used.   Descriptive services (click for more details) provide a set of ThingWorx services to analyze a data set and perform many common data transformations.  These services are targeted for performing calculations and transformations on recent operating history of a single property.  Descriptive services are called on demand to perform batch calculations. Scenarios to use descriptive services: On demand calculations performed within a mashup, a service call or an event to determine action and calculation results are not (always) stored Regular occurring calculations on logged property values or generated datasets (batch calculations) Calculations are done regularly in minutes, hours or days on a discrete set of data.  Examples: average value in last hour, median value in last day, or max value in last half hour.  Time between data creation and analysis is minutes or hours.  Some latency in the calculation result is acceptable for the use case. Input data set has 10s to 100s to 1000s of values.  Keep the size of the input data at 10,800 values or less.  If larger data sizes are required, then break them into micro batches if possible or use other tools to handle the processing. Multiple calculations need to be done from the same set of input data.  Examples: average value in last hour, max value in the last hour and standard deviation value in the last hour are all required to be calculated. Things to consider when using descriptive services Requires input dataset to be in the specific datashape format that is used by descriptive services.  If property values are logged in a value stream, there is a service to query the values and prepare the dataset for processing.  If scenarios where the data is not for a logged property, then another service or sql query can be used to prepare the dataset for processing. Requires javascript development work to implement.   This includes creation of a service to execute the descriptive services and usage of subscriptions and events to orchestrate calculations. An example of the javascript to execute descriptive services is available in the help center (here) Typically retrieval of the input data from value stream (QueryTimedValuesForProperty) is slowest part of the process. The input data is sent to an out of process platform analytics service for all calculations. Broader set of calculation services available (see table at the end of this article) Remember that these services are not meant to be used for big data calculations or big data preparation.  Look for other approaches if the input data sets grow larger than 10,800 values Property Transforms (click for more details) provide a set of transformation services for streaming data as it enters ThingWorx.   Property transforms are targeted for performing continuous calculations on recent values in the stream of a single property and delivering results in (near) real-time.  Since property transforms are continuous calculations, they are always running and using compute resources. Before implementing property transforms review the information in the property transform sizing guide to better understand factors that impact the scaling of property transforms. Scenarios to use: Continuous calculations on a stream for a single property as new data comes into ThingWorx New values enter the stream faster than one value per minute (as a general guideline) Calculations required to be done in seconds or minutes.  Examples: average electrical current in last 10 seconds, median pressure in the last 10 readings,  or max torque in last minute Time between data creation and analysis is small (in seconds).  Results of property transform is required for rapid decisions and action so reducing latency is critical Data sets used for calculation are small and contain 10s to 100s of values.  Calculated results are stored in a new property in the ThingModel Things to consider when using property transforms Codeless process to create new property transforms on a single property in the ThingModel Does not require input property values to be logged as calculations are performed on streaming data as it enters ThingWorx Unlike descriptive services which only execute when called, each property transform creates a continuously running job that will always be using compute resources.  Resource allocations for property transforms must be included in the overall system architecture.  Before selecting the property transform approach, refer to the Property Transform Sizing Guide for more information about how different parameters affect the performance of Property Transforms and results of performance load test scenarios. Let’s apply these guidelines to a few different use cases to determine which approach to select. 1. Mashup application that allows users to calculate and view median temperature over a selected time window In this scenario, the calculation will be executed on-demand with a user defined time window. Descriptive services are the only option here since there is not a pre-defined schedule and the user can select which data to use for the calculation.   2. Calculate the max torque (readings arriving one per second) on a press over each minute without storing all of the individual readings. In this scenario, the calculation will be executed without storing the individual readings coming from the machine. The transformation is made to the data on its way into ThingWorx and continuously calculating based on new values. Property transforms are the only option here since the individual values are not being stored.   3. Calculation of average pressure value (readings arriving one per second) over a five minute window to monitor conditions and raise an alert when the median value is more than two standard deviations from expected. In this scenario, both descriptive services and property transforms can perform the calculation required. The calculation is going to occur every 5 minutes and each data set will have about 300 values. The selection of batch (descriptive services) or streaming (property transforms) will likely be determined by the usage of the result. In this case, the calculation result will be used to raise an alert for a specific five minute window which likely will require immediate action. Since the alert needs to be raised as soon as possible, property transforms are the best option (although descriptive services will handle this case also with less compute resource requirements).   4, Calculation of median temperature (readings each 20 seconds) over 48 hour period to use as input to predict error conditions in the next week. In this scenario, the calculation will be performed relatively infrequently (once every 48 hours) over a larger data set (about 8,640 values). Descriptive services are the best option in this case due to the data size and calculation frequency. If property transforms were used, then compute resources would be tied up holding all of the incoming values in memory for an extended period before performing a calculation. With descriptive services, the calculation will only consume resource when needed, or once every 48 hours.   Hopefully this information above provides some more insight and guidelines to help choose between property transforms and descriptive services. The table below provides some additional comparisons between the two approaches.     Descriptive Services Property Transforms Purpose Provide a set of ThingWorx services to analyze a data set and perform many common data transformations. Provide a set of prescribed transformation services for streaming data as it enters ThingWorx. Processing Mode Batch Streaming / Continuous Delivery API / Service Composer interface API / Service Input Data Discrete data set Must be logged Single property Configurable by time or lookback Rolling data set on property X Persistence is optional Single property Configurable by time or lookback Output Data Return object handled programmatically Single output for discrete data set New property f_X in the input model Continuous output at configurable frequency Output time aligned with input data Available Services Statistics (min, max, mean, median, mode, std deviation) SPC calculations (# continuous data points: above threshold, in / out of range, increasing / decreasing, alternating) Data distribution: count by bins (histogram) Five numbers (min, lower quartile, median, upper quartile, max) Confidence interval Sampling frequency Frequency transform (FFT) Statistics (min, max, mean, median, mode, std deviation) SPC calculations (# continuous data points: above threshold, in / out of range, increasing / decreasing, alternating)

This is part of the continuing series of Blog posts regarding Troubleshooting the Application, this article will discuss more advance issues that some clients and customer have encountered while building or using ThingWorx Analytics. Packer Script Error – Unable to Download CentOS Image As the application is developed and built inside a CentOS image, the ThingWorx Analytics Packer Script tool for Virtual Machine Appliance creation utilizes the CentOS mirror repository in the creation process. When the end user is attempting to build the Virtual Machine Appliance with the Packer Script media creation tool, part of the process is to download the CentOS 7 ISO image file as the basis for the operating system that the ThingWorx Analytics Server software will be installed to. If CentOS updates or changes their mirror links for the source file ISO, you may encounter the following error: ==> virtualbox-iso: Downloading or copying Guest additions virtualbox-iso: Downloading or copying: file:///C:/Program%20Files/Oracle/VirtualBox/VBoxGuestAdditions.iso ==> virtualbox-iso: Downloading or copying ISO virtualbox-iso: Downloading or copying: file:///local-file-repo/CentOS-7-x86_64-Minimal-1511.iso virtualbox-iso: Error downloading: open local-file-repo/CentOS-7-x86_64-Minimal-1511.iso: The system cannot find the path specified. virtualbox-iso: Downloading or copying: http://mirror.spro.net/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-1511.iso virtualbox-iso: Error downloading: checksums didn't match expected: 88c0437f0a14c6e2c94426df9d43cd67 ==> virtualbox-iso: ISO download failed. Build 'virtualbox-iso' errored: ISO download failed. ==> Some builds didn't complete successfully and had errors: --> virtualbox-iso: ISO download failed. ==> Builds finished but no artifacts were created. Solution Method 1: Configuration File Replacement We have created a custom JSON configuration file that resolves the mirror issue for CentOS 7 v1611. You can download the JSON file here; you may have to right-click and “save link as” a JSON extension file. Also note, you will have to save/rename this JSON file as neuron-solo-variables.json. Using this file, navigate to your Packer Script builder directory, usually this is found in the following path: <PATH>\ThingWorx-Analytics-Server-Standalone\components\vm-builder\neuron-vm-builder Copy the new JSON file into this directory, and replace the current existing copy. You can now re-run the Packer Script for your desired Virtual Machine Appliance output. Method 2: Manual Configuration File Adjustment You will have to locate an active mirror for CentOS 7. A list of current active mirrors can be found here. When selecting a mirror, you will need to select the Minimal ISO install, as this is the base image that is used for the VM creation. Next, you will have to open the current neuron-solo-variables.json configuration file located in the <PATH>\ThingWorx-Analytics-Server-Standalone\components\vm-builder\neuron-vm-builder directory. You will have to replace the os_image_download_url value with an active Mirror URL from the list above. Next, for the os_iso_md5_checksum variable, you will need to replace the entry with the new SHA256 checksum from CentOS, which can be located here. Default Settings: New Settings: Save changes and close the neuron-solo-variables.json configuration file. CentOS has switched over from MD5 to SHA256 checksums. Even though in the following the variable name has “MD5” in the string, we will be modifying a second JSON configuration file to address this. In the same directory that we are currently working in, open the neuron-solo.json configuration file. You will need to modify the attribute iso_checksum_type to sha256 Default Settings: New Settings: Save changes and close the neuron-solo.json configuration file. You can now re-run the Packer Script for your desired Virtual Machine Appliance output.

The accuracy of a predictive model can be boosted in two ways: Either by embracing Feature engineering or by applying boosting algorithms straight away. There are multiple boosting algorithms like Gradient Boosting, XGBoost, AdaBoost, Gentle Boost etc. Every algorithm has its own underlying mathematics and a slight variation is observed while applying them. While working with boosting algorithms, we have come across two frequently occurring buzzwords: Bagging and Boosting. Bagging: It is an approach where you take random samples of data, build learning algorithms and take simple means to find bagging probabilities. Boosting: Boosting is similar, however the selection of sample is made more intelligently. We subsequently give more and more weight to hard to classify observations. Below are Default Algorithms used in Predictive Models generated in ThingWorx Analytics: Decision Tree Gradient Boost Linear regression Neural Net Random Forrest Logistic Regression Gradient boosting is a machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. It builds the model in a stage-wise fashion like other boosting methods do, and it generalizes them by allowing optimization of an arbitrary differential loss function. Let’s begin with an easy example: Assume, you are given a previous model M to improve on. Currently you observe that the model has an accuracy of 80% (any metric). How do you go further about it? One simple way is to build an entirely different model using new set of input variables and trying better ensemble learners. On the contrary, we have a much simpler way to suggest. It goes like this: Y = M(x) + error What if we are able to see that error is not a white noise but have same correlation with outcome(Y) value. What if we can develop a model on this error term? Like:error = G(x) + error2 Probably, we will see error rate will improve to a higher number, say 84%. Let’s take another step and regress against error2: error2 = H(x) + error3 Now we combine all these together: Y = M(x) + G(x) + H(x) + error3 This probably will have a accuracy of even more than 84%. What if we can find an optimal weights for each of the three learners: Y = alpha * M(x) + beta * G(x) + gamma * H(x) + error4 How Gradient Boosting Works: 1. Loss Function: The loss function used depends on the type of problem being solved. It must be differential, but many standard loss functions are supported and you can define your own. A benefit of the gradient boosting framework is that a new boosting algorithm does not have to be derived for each loss function that may want to be used, instead, it is a generic enough framework that any differential loss function can be used. 2. Weak Learner: Decision trees are used as the weak learner in gradient boosting. Specifically regression trees are used that output real values for splits and whose output can be added together, allowing subsequent models outputs to be added and “correct” the residuals in the predictions. Trees are constructed in a greedy manner, choosing the best split points based on purity scores like Gini or to minimize the loss. 3. Additive Model: Trees are added one at a time, and existing trees in the model are not changed. A gradient descent procedure is used to minimize the loss when adding trees. we have weak learner sub-models or more specifically decision trees. After calculating the loss, to perform the gradient descent procedure, we must add a tree to the model that reduces the loss. Improvements to Basic Gradient Boosting: 1. Tree Constraints: It is important that the weak learners have skill but remain weak. Below are some constraints that can be imposed on the construction of decision trees: Number of trees: ​Generally adding more trees to the model can be very slow to over fit. The advice is to keep adding trees until no further improvement is observed. Tree depth: Deeper trees are more complex trees and shorter trees are preferred. Generally, better results are seen with 4-8 levels. Number of nodes or number of leaves: like depth, this can constrain the size of the tree, but is not constrained to a symmetrical structure if other constraints are used. Number of observations per split: Imposes a minimum constraint on the amount of training data at a training node before a split can be considered Minimum improvement to loss: Is a constraint on the improvement of any split added to a tree. 2. Weighted Updates: The contribution of each tree to this sum can be weighted to slow down the learning by the algorithm. This weighting is called a shrinkage or a learning rate. "Each update is simply scaled by the value of the “learning rate parameter v". 3. Stochastic Gradient Boosting: At each iteration a sub sample of the training data is drawn at random (without replacement) from the full training data set. The randomly selected sub sample is then used, instead of the full sample, to fit the base learner. 4. Penalized Gradient Boosting: The additional regularization term helps to smooth the final learnt weights to avoid over-fitting. Intuitively, the regularized objective will tend to select a model employing simple and predictive functions.

One of the interesting features of ThingWorx Analytics Manager is its ability to run distributed models created in Excel (and more of course).  Most people having been tasked with understanding data have built models in Excel and have sometimes built quite complex models (or even applications) with it.   The ability to tie these models to real data coming from various systems connected through ThingWorx and operationalise their execution is a really simple way for people to leverage their existing work and I.P. on a connected analytics journey.   To demonstrate this power and ease of implementation, I created a sample data set with historical data, traffic profile, and a simple anomaly detection model to execute with Analytics Manager.  (files are attached)   The online help center was quite helpful in explaining the process of Creating the Excel Workbook, however I got stuck at the XML mapping stage.  The Analytics and Excel documentation both neglect to mention one important detail -- you must be using the Windows version of Excel in order to get the XML Source functionality (and I use Mac).  Once using Windows, it was easy to do - here is a video of the XML mapping part of the process (for the inputs and results).   

In this video we cover: a short introduction of Thingworx Analytics Builder The import of the Thingworx Analytics Builder extension   This video applies to ThingWorx Analytics 52.1 till 8.1   Updated Link for access to this video:  Installing Thingworx Analytics Builder:  Part 1 of 3

In our interactions with PTC customers we often learn they have previously performed Analytics modeling in Python, Matlab, R, or even built home grown analyses in languages such as Java or C++. As expected, when adopting an Industrial Innovation Platform such as ThingWorx that also has its own ThingWorx Analytics module, customers do not want to reimplement everything from scratch and would rather integrate their previous work in the Smart Applications built in ThingWorx, leveraging a combination of their existing toolset together with ThingWorx Analytics modeling. That is certainly possible and there are multiple ways to do that. In this article we will focus on several general ways to make that happen, but it is important to keep in mind that language specific approaches are also possible and we are happy to discuss those in the specific context of the customer.   Here are five different ways to bring existing Analytics into ThingWorx: If the task is to reuse an existing predictive model developed in a language such as Python/R/Matlab, typically one can export that model in PMML (Predictive Model Markup Language), an xml format, and import it in ThingWorx Analytics using the AnalyticsServer_ResultsThing -> UploadModel service. Libraries such as sklearn2pmml & r2pmml can be utilized towards that goal. The imported model can then be used in the same fashion as a ThingWorx Analytics developed model to power smart applications built in ThingWorx. If the Analysis involves more complex tasks than Predictive Modeling, such as custom data normalizations or non-standard Machine Learning models or home grown algorithms, one can use the options below. Call the ThingWorx exposed REST Web API from Python/Matlab/R/Java/Javascript. Every service from ThingWorx can be called that way, and the API can also be used to push analyses results into ThingWorx for further consumption, perhaps together with other sources of data such as sensor readings, in the smart applications built there. The documentation for the ThingWorx REST API can be found here.  Expose the existing Analytics via using a thin layer of REST Web Services. For example, in Python, this can be done using Flask, with few lines of code. Then, the orchestration can happen from ThingWorx by calling the exposed Web Service and weaving the results back into smart applications. Often our customers' current architecture involves a relational database (e.g. SQL Server, Oracle, etc) that is powering the existing Analytics, and stores the end results (predictions, correlations, etc). In this scenario, we can connect ThingWorx directly to that database to read these results.  Finally, in the case of complex Analytics, where a tighter integration with ThingWorx is desired, existing Analytics / algorithms can be wrapped into a ThingWorx Extension or an Analytics Provider using the corresponding PTC SDKs.  When choosing an integration option, customers need to carefully balance complexity of integration, constraints of their architecture, Analytics modeling complexity, as well as end user consumption requirements.

Video Author:                    Christophe Morfin Original Post Date:            June 9, 2017 Applicable Releases:        ThingWorx Analytics 8.0   Description: In this video we go through the steps to install ThingWorx Analytics Server 8.0.    

In this video we cover the process of installing ThingWorx Analytics Server 52.1. Make sure to have reviewed the part 1 video about pre requisite   Updated Link for access to this video:  Installing ThingWorx Analytics Server: Part 2 of 2

Build a Predictive Analytics Model Guide Part 2   Step 5: Profiles   The Profiles section of ThingWorx Analytics looks for combinations of data which are highly correlated with your desired goal. On the left, click ANALYTICS BUILDER > Profiles. Click New....The New Profile pop-up will open. NOTE: Notice the Text Data Only section which is new in ThingWorx 9.3.         3. In the Profile Name field, enter vibration_profile. 4. In the Dataset field, select vibration_dataset. 5. Leave the Goal field set to the default of low_grease. 6. Leave the Filter field set to the default of all_data. 7. Leave the Excluded Fields from Profile field set to the default of empty. 8. Click Submit. 9. After ~30 seconds, the Signal State will change to COMPLETED. The results will be displayed at the bottom.                 The results show several Profiles (combinations of data) that appear to be statistically significant. Only the first few Profiles, however, have a significant percentage of the total number of records. The later Profiles can largely be ignored. Of those first Profiles, both Frequency Bands from Sensor 1 and Sensor 2 appear. But in combination with the result from Signals (where Sensor 1 was always more important), this could possibly indicate that Sensor 1 is still the most important overall. In other words, since Sensor 1 is statistically significant both by itself and in combination (but Sensor 2 is only significant in combation with Sensor 1), then Sensor 2 may not be necessary.     Step 6: Create Model   Models are primarily used by Analytics Manager (which is beyond the scope of this guide), but they can still be used to measure the accuracy of predictions. When Models are calculated, they inherently withhold a certain amount of data. The prediction model is then run against the withheld data. This provides a form of "accuracy measure", which we'll use to determine whether Sensor 2 is necessary to the detection of a low grease condition by creating two different Models. The first Model (which you will create below) will contain all the data, while the second Model (in the next step) will exclude Sensor 2. On the left, click ANALYTICS BUILDER > Models.   Click New….The New Predictive Model pop-up will open.   3. In the Model Name field, enter vibration_model. 4. In the Dataset field, select vibration_dataset. 5. Leave the Goal field set to the default of low_grease. 6. Leave the Filter field set to the default of all_data.         7. Leave the Excluded Fields from Model section at its default of empty.       8. Click Submit. 9. After ~60 seconds, the Model Status will change to COMPLETED.   View Model   Now that the prediction model is COMPLETED, you can view the results. Select the model that was created in the previous step, i.e. vibration_model. Click View… to open the Model Information page.   Review the visualization of the validation results. Note that your results may differ slightly from the picture, as the automatically-withheld "test" portion of the dataset is randomly chosen. Click on the ? icon to the right of the chart for details on the information displayed.   The desired outcome is for the model to have a relatively high level of accuracy. The True Positive Rate shown on the Receiver Operating Characteristic (ROC) chart are much higher than the False Positives. The curve is relatively high and to the left, which indicates a high accuracy level. You may also click on the Confusion Matrix tab in the top-left, which will show you the number of True Positive and True Negatives in comparison to False Positives and False Negatives.     Note that the number of correct predictions is much higher than the number of incorrect predictions.     As such, we now know that our Sensors have a relatively good chance at predicting an impending failure by detecting low grease conditions before they cause catastrophic engine failure.     Step 7: Refine Model   We will now try comparing this first Model that includes both Sensors to a simpler Model using only Sensor 1. We do this because we suspect that Sensor 2 may not be necessary to achieve our goal. On the left, click ANALYTICS BUILDER > Models.   Click New…. In the Model Name field, enter vibration_model_s1_only. In the Dataset field, select vibration_dataset. Leave the Goal field set to the default of low_grease. Leave the Filter field set to the default of all_data.   On the right beside Excluded Fields from Model, click the Excluded Fields button. The Fields To Be Excluded From Job pop-up will open. 8. Click s2_fb1 to select the first Sensor 2 Frequency Band. 9. Select the rest of the Frequency Bands through s2_fb5 to choose all of the Sensor 2 frequencies. 10. While all the s2 values are selected, click the green "right arrow", i.e. the > button in the middle. 11. At the bottom-left, click Save. The Fields To Be Excluded From Job pop-up will close.           12. Click Submit. 13. After ~60 seconds, the Model State will change to COMPLETED. 14. With vibration_model_s1_only selected, click View....   The ROC chart is comparable to the original model (including Sensor 2). Likewise, the Confusion Matrix (on the other tab) indicates a good ratio of correct predictions versus incorrect predictions.     NOTE: These Models may vary slightly from your own final scores, as what data is used for the prediction versus for evaluation is random. ThingWorx Analytics's Models have indicated that you are likely to receive roughly the same accuracy of predicting a low-grease condition whether you use one sensor or two! If we can get an accurate early-warning of the low grease condition with just one sensor, it then becomes a business decision as to whether the extra cost of Sensor 2 is necessary.   Step 8: Next Steps   Congratulations! You've successfully completed the Build a Predictive Analytics Model guide, and learned how to:   Load an IoT dataset Generate machine learning predictions Evaluate the analytics output to gain insight    This is the last guide in the Getting Started on the ThingWorx Platform learning path.   This is the last guide in the Monitor Factory Supplies and Consumables learning path.   The next guide in the Design and Implement Data Models to Enable Predictive Analytics learning path is Operationalize an Analytics Model.     Additional Resources   If you have questions, issues, or need additional information, refer to:   Resource Link Support Analytics Builder Help Center    

This video is the 3 rd part of a series of 3 videos walking you through how to setup ThingWatcher for Anomaly Detection. In this video we will use Anomaly Mashup to visualize data received from my remote device.   Updated Link for access to this video:  Anomaly Detection 8.0:  Viewing Data via Anomaly Mashup:  Part 3 of 3

Users of ThingWorx Analytics (TWA) may choose to create a predictive model using TWA or import a predictive model that was created using other software. When importing into or exporting out of TWA, this predictive model must be in a PMML (Predictive Model Markup Language) version 4.3+ format. This post describes how to complete the import and export processes. Exporting: The user may create a model in two main ways inside of TWA: using the Builder user interface, or by using ‘Create Job’ service that exists the Training Thing. Whichever method is used, a model Job Id is created automatically by TWA for that model. It is this model Job Id that is used to identify the model inside of TWA, regardless of what is being done with that model.   If a model is trained using Builder, the user may highlight that model, click ‘Job Details’, and then copy the Job ID. This is done as follows:   Next, the user will navigate to Browse --> Things --> …TrainingThing. This is the Training Microservice inside of TWA where all the functionality involved with training a model exists. Within the …TrainingThing, the user will execute the ‘RetrieveModel’ service under Services. When executing the service, the user will paste the model Job ID (ex. 49704f1a-7fcd-4e38-ab53-84ef46517d0a) they copied earlier, and press ‘Execute’. The resulting text can then be highlighted and copied to Notepad or some other text editor, and saved as .pmml format (ex. ‘ModelExport.pmml’).   Importing Through Results Microservice: To import a model that has been saved in PMML 4.3+ format into TWA using the Results Microservice, the user will navigate to Manage --> Repositories (ex. AnalyticsUploadStorage) --> Actions --> Upload, and choose the PMML file. The user will then navigate to Browse --> Things --> …ResultsThing. This is the Results Microservice inside of TWA where all the functionality exists related to previously trained models. Within the …ResultsThing, the user will execute the ‘UploadModel’ service under Services. Alternatively, the user can upload the model from any repository using ‘UploadModelFromRepository” service.   To create a model from the uploaded PMML inside of TWA, the user will fill out the filePath and name then execute the service. Note: This model will not show up in Builder, as that would require model validation information that is not part of the imported PMML file.   The resulting Job Id can be used to make predictions, such as by using the …PredictionThing’s BatchScore or RealtimeScore services. At this point, the uploaded model acts the same way as if the model were created inside of that TWA environment.       Importing Through Analytics Manager: To import a model that has been saved in PMML 4.3+ format into TWA using the Analytics Manager, the user will navigate to Analytics --> Analytics Manager --> Analysis Models, and click the green “New” button. Next the user will choose the provider name (or create a new one by navigating to Analytics --> Analytics Manager --> Analysis Providers). The user will also check the box to “Upload Model”, and click the grey “Choose File” button to find the PMML file. Finally, the user will click the black “Upload” button, then the green “Save” button.     At this point, the model is uploaded into ThingWorx Analytics, and the user may progress through the subsequent steps to set up “Analysis Events” and “Analysis Jobs” that will be powered by the imported model.

Underneath video walks through how to Publish a Model from Analytics Builder into Analytics Manager using the connector named TW.AnalysisServices.AnalyticsServer.AnalyticsServerConnector.

    Configure streaming asset properties for SPC monitoring.   Guide Concept   Note: This guide is intended only as a starting point and is not a fully developed or supported solution. This accelerator has been developed using ThingWorx 8.5 and should not be used with any previous versions of the software.   This project introduces you to configuring Properties from your connected streaming assets for Statistical Process Control (SPC) monitoring.   Following the steps in this guide, you will learn how multiple connected assets and their Properties can be displayed in a hierarchy tree.   You will then configure these Properties using predetermined set points, and upper and lower control points for the assets.   Finally, you will learn to navigate the monitoring of the Properties.   We introduce some of the basic building blocks of an SPC accelerator, including important Things and Mashups. You will also use ThingWorx Timers to simulate streaming data.     You'll learn how to   Configure multiple properties for SPC monitoring Identify abnormalities in streaming property values   NOTE: The estimated time to complete this guide is 30 minutes     Step 1: Import SPC Accelerator   Before exploring Statistical Process Control (SCP) within ThingWorx Foundation, you must first import some Entities via the top-right Import / Export button.   Download and unzip PTC_StatisticalCalculations_PJ.zip and PTC_SPC_PJ.zip. These two files each contain a ThingWorx project of a similar name. Import PTC_StatisticalCalculations_PJ.twx first. Import PTC_SPC_PJ.twx once the other import has completed. Explore the imported entities.   Each of the projects contain multiple entities of various types. The most important entities you will use in this guide are as follows:    Entity Name                                   Description Motor_Pump1 Timer to simulate streaming data Motor_Blower1 Timer to simulate streaming data PTC.SPC.ConfigurationHelper Thing that manages the accelerator PTC.SPC.Configuration_MU Mashup for configuring SPC properties PTC.SPC.Monitoring_MU Mashup for monitoring SPC properties   Step 2: Configure Properties for SPC monitoring   You may configure SPC monitoring for multiple production lines, connected assets on those lines, and time-series Properties on those assets using the SPC accelerator.   This is done by viewing the PTC.SPC.Configuration_MU Mashup.   Follow these steps to configure an SPC Property.   Create a new production line   In the Enter New Production Line Name text field, type Line100. Click Add New Line.   Now you will see the new production line added to the Asset Hierarchy tree along the left. All production lines you’ve created (as well as their assets and the assets’ Properties) will be shown here.   In the lower-right, the SPC Property Configuration area has disappeared because the item selected in the Asset Hierarchy tree is the new line; only assets within lines can have streaming Properties.   Add a streaming asset to Line100   Within the Select Asset Thing entity picker, type Motor_Blower1. Click Add Asset for Line.   This asset has Properties that are streaming into ThingWorx. Multiple assets can be added to the same production line by selecting the line from the Asset Hierarchy tree and following the steps above.   If you select the new asset from the Asset Hierarchy tree, you will see that the list of Properties Eligible for SPC Monitor is shown in the SPC Property Configuration area. All Properties have the same default configuration associated with them.   Configure a property for SPC Monitoring   Below is a brief description of each of the configuration parameters:    Parameter           Description Sample Size Number of consecutive property values to average together. For example, a Sample Size of 5 will tell the accelerator to group every 5 property values together and calculate their average value. XBar Points Number of the most recent sample aggregations to display in the SPC Monitoring Mashup. This also affects SPC calculations. Capability Points Number of the most recent sample aggregations to use when populating the Capability Histogram in the SPC Monitoring Mashup. Set Point Value determined to be the set point for that particular property on the asset. Lower Design Spec Value determined to be the lower design spec for that particular property on the asset. This is used for capability calculations. Upper Design Spec Value determined to be the upper design spec for that particular property on the asset. This is used for capability calculations.   Select Pressure1 from the list of eligible properties. Enter the following values:  Properties                      Values Sample Size 5 Xbar Points 30 Capability Points 60 Set Point 73 Lower Design Spec 68 Uppder Design Spec 78   3, Select Xbar-R Chart. 4. Click Add or Update SPC Monitoring. 5. Pressure1 is added to the Asset Hierarchy tree underneath the Motor_Blower1 asset. 6. If you select this Property, you can modify the configuration and save it by clicking Add or Update SPC Monitoring.   Configure assets and Properties   Follow these steps using the following parameters:                                     Line Asset  Property  Sample Size Xbar Point Capability Points Set Points Lower Design Spec Upper Design Spec Chart Type 100 Motor_Blower1 Pressure1 5 30 60 73 68 78 Xbar-R 100 Motor_Blower1 Pressure2 5 30 60 78 68 89 Xbar-R 100 Motor_Blower1 Temperature1 5 30 60 50 44 56 Xbar-s 100 Motor_Blower1 Temperature2 5 30 60 85 73 97 Xbar-s 100 Motor_Pump1 Vibration10 5 30 60 150 108 190 Xbar-s 100 Motor_Pump1 Vibration11 8 60 100 200 168 220 Xbar-s 100 Motor_Pump1 Pressure100 5 30 60 100 84 118 Xbar-R 100 Motor_Pump1 Pressure200 5 30 60 90 84 97 Xbar-R   As you add assets to Line100 and configure their Properties, you will see the Asset Hierarchy tree grow. If you need to remove an asset or its associated Properties from the Asset Hierarchy tree, you may select that item, and click Remove Selected. For any item you remove, its child-items will also be removed.   Click here to view Part 2 of this guide.

In this video you would see how to start to use your already created Virtual Image of ThingWorx Analytics using Oracle Virtual Box. This Video is Part-1 of the Series Getting Started with ThingWorx Analytics.   Updated Link for access to this video:  Getting Started with ThingWorx Analytics: Part 1 of 2

This video is the 2 nd part of a series of 3 videos walking you through how to setup ThingWatcher for Anomaly Detection. In this video you will learn how to use “Discover UI” from the “New Composer” to bind simulated data coming through KEPServer for Anomaly Detection.   Updated Link for access to this video:  Anomaly Detection 8.0: Configuring Anomaly Alerts:  Part 2 of 3

Video Author:                     Christophe Morfin Original Post Date:            October 2, 2017 Applicable Releases:        ThingWorx Analytics 8.1   Description:​ In this video we will walk thru the installation steps of ThingWorx Analytics Server 8.1.  This covers the Native Linux installation though the steps will be similar for a docker installation on Windows or Linux.    

In this video we show the setup for anomaly detection (ThingWatcher) in release 8.4. We also show how to create an anomaly alert.  

Preface   In this blog post, we will discuss how to Start and Stop ThingWorx Analytics, as well as some other useful triaging/troubleshooting commands. This applies to all flavors of the native Linux installation of the Application.   In order to perform these steps, you will have to have sudo or ROOT access on the host machine; as you will have to execute a shell script and be able to view the outputs.   The example screenshots below were taken on a virtual CentOS 7 Server with a GUI as ROOT user.     Checking ThingWorx Analytics Server Application Status   1. Change directory to the installation destination of the ThingWorx Analytics (TWA) Application. In the screenshot below, the application is installed to the /opt/ThingWorxAnalyticsServer directory   2. In the install directory, there are a series of folders and files. You can use the ​ls​ command to see a list of files and folders in the installation directory.     a. You will need to go navigate one more level down into the ./ThingWorxAnalyticsServer/bin​ ​directory by using command ​cd ./bin​     b. As you can see above, we used the ​pwd​ command to verify that we are in the correct directory.   3. In the ./ThingWorxAnalyticsServer/bin directory, there should be three shell files: configure-apirouter.sh, configure-user.sh, and twas.sh     a. To run a status check of the application, use the command ./twas.sh status           i. This will provide a list of outputs, and a few warning messages. This is normal, see screenshot below:      b. You will have a series of services, which will have a green active (running) or red not active (stopped).           i. List of services: twas-results-ms.service - ThingWorx Analytics - Results Microservice twas-data-ms.service - ThingWorx Analytics - Data Microservice twas-analytics-ms.service - ThingWorx Analytics - Analytics Microservice twas-profiling-ms.service - ThingWorx Analytics - Profiling Microservice twas-clustering-ms.service - ThingWorx Analytics - Clustering Microservice twas-prediction-ms.service - ThingWorx Analytics - PredictionMicroservice twas-training-ms.service - ThingWorx Analytics - Training Microservice twas-validation-ms.service - ThingWorx Analytics - Validation Microservice twas-apirouter.service - ThingWorx Analytics - API Router twas-edge-ms.service - ThingWorx Analytics - Edge Microservice   Starting and Stopping ThingWorx Analytics   If you encounter any errors or stopped services in the above, a good solution would be to restart the TWA Server application.   There are two methods to restart the application, one being the restart ​command, the other would be using the stop​ and ​start​ commands.   Method 1 - Restart Command:   1. In the same ./ThingWorxAnalyticsServer/bin​ ​directory, run the following command: ./twas.sh restart     a. The output of a successful restart will look like the following: 2. The restart should only take a few seconds to complete   Method 2 - Stop / Start Commands:   1. In the same ./ThingWorxAnalyticsServer/bin​ ​directory, run the following command: ./twas.sh stop 2. After the application stops, run the following command: ./twas.sh start   Note: You can confirm the status of the TWA Server application by following the steps in the "Checking ThingWorx Analytics Server Application Status" section above.

Connecting Existing Things to ThingWorx Industrial Gateway for Anomaly Detection   In this Video you will learn how to :   - To bind a property of an existing entity to the KEPSserverEX Data Feed - To create an Alert on that property and monitor it's behavior   Updated Link for access to this video:  Connecting Existing Things to ThingWorx Industrial Gateway for Anomaly Detection