ThingWorx Analytics Training: Module 6 Part 3
This video continues Module 6: Predictive Models & Model Validation of the ThingWorx Analytics Training videos. It then begins to describe some of the performance metrics used to evaluate predictive models.
This video contains a hands-on exercise, and it is recommended to complete it before moving on to subsequent sections. This exercise of training a predictive model is continued from Part 2.
This video has chapters — use the chapter menu in the video player to jump to any section.
Chapter Summaries
0:00 – Advanced Model Configuration: Validation Holdout Percentage & Max Fields
Walks through the Advanced Model Configuration tab in ThingWorx Analytics, explaining the default 20% Validation Holdout (the 80/20 training-validation split) and lowering Max Fields from 25 to 15 of 34 fields to reduce overfitting while preserving predictive power.
0:58 – Redundancy Filter & Confidence Model Availability
Covers the Redundancy Filter option, which keeps only top signals with high mutual information to the goal variable yet unique from each other, and explains why a confidence model is unavailable for the Boolean Grinder Error Occurrence field in the BeanPro data set.
1:57 – Sampling Strategy: Up-Sampling & Down-Sampling
Explains the Sampling Strategy section, showing how choosing Strategy Up Sample with Value True and a factor of five creates additional copies of positive training records, and why this exercise leaves Strategy set to None.
2:47 – Confusion Matrix Weights, Cost Matrix & Grayed-Out Parameters
Details enabling custom Confusion Matrix Weights to act as a cost matrix—raising true positive weight for rare medical diagnoses or weighting false negatives ($90 downtime) versus false positives ($20 service charge) in machine-failure prediction—plus why Time Series and Text Parameters are grayed out.
4:31 – Learning Techniques: Add Learner, Random Forest & Ensemble Options
Introduces the Learning Techniques section and the Add Learner button, configuring a random forest by adjusting max depth and number of trees, building ensembles, and reviewing ensemble options Elite Average, Average, Best, and Majority Vote with a comparison metric.
5:45 – Soloist Neural Network Setup & Model Submission
Demonstrates removing Decision Tree, Gradient Boost, and Linear Regression learners so only a neural network remains, which switches the ensemble technique to Soloist, then clicking the green Submit button to train the model.
6:30 – Model Results: Precision, Recall & ROC Chart Thresholds
Reviews the completed model's details—name, dataset, and data filters—and opens performance metrics including precision, recall, ROC, the generated ROC chart, and the confusion matrix, noting how the chosen threshold and matrix weights affect results.
7:32 – Validation vs. Training Metrics & Overfitting Signs
Compares the Validation Metrics and Training Metrics tabs, explaining why training scores run higher, why validation metrics best predict production performance on unseen records, and how a large gap between them signals an overfitting model.
9:08 – Signals & Profiles Jobs on Training Records
Explains running signals and profiles jobs scoped to the records used to train the model, contrasting them with dataset-wide signals and profiles jobs launched from the left panel unless a data filter is applied.
9:36 – Model Validation Overview: RMSE & Pearson Correlation
Opens model validation, defining Root Mean Squared Error (RMSE)—lower is better—for continuous goal variables and Pearson correlation coefficient, which measures linear correlation between predicted and actual values on a scale from -1 to 1.
11:09 – Confusion Matrix Explained: TP/FN/FP/TN, Accuracy, TPR & FPR
Breaks down the confusion matrix into true positives, false negatives, false positives, and true negatives, then defines model accuracy, true positive rate, and false positive rate, working a numeric example yielding 0.78 accuracy, 0.71 TPR, and 0.16 FPR.
13:38 – Recall & Precision: Medical Diagnosis and Spam Detection
Defines recall (true positive rate) as ideal when false negatives are unacceptable, such as predicting diabetes, and precision as ideal when you must trust predicted positives, illustrated with spam-versus-inbox email classification.
15:03 – Specificity, R-Squared & Adjusted R-Squared
Explains specificity for avoiding false positives in asset-failure prediction, then R-squared for variance explained via residual and total sum of squares, and Adjusted R-squared, which penalizes irrelevant variables to compare models and curb overfitting.
17:38 – AUC / ROC Curve Interpretation & Classifier Comparison
Introduces Area Under the Curve (AUC) and the Receiver Operating Characteristic (ROC), plotting true positive rate against false positive rate across thresholds, and interprets sample values (0.5 random, 0.6 weak, 0.7 okay, 0.8 good, 0.9 excellent, 1.0 perfect).