ThingWorx Analytics Training: Module 6 Part 4
This video continues Module 6: Predictive Models & Model Validation of the ThingWorx Analytics Training videos. It covers some modeling techniques to help build better predictive models. It discusses the dangers of models that overfit data, and how to avoid overfitting.
This video does not contain any hands-on exercises, and you don't need access to a running ThingWorx Analytics environment.
This video has chapters — use the chapter menu in the player to jump to any section.
Chapter Summaries
0:00 – Accuracy vs. Recall: Metrics for Imbalanced Classes
Explains why accuracy misleads on imbalanced data using a healthcare example where only 0.02% of patients develop kidney disease, showing a 99.9%-accurate model can be useless and recall is the better performance metric.
1:17 – Upsampling / Oversampling: ThingWorx Analytics for Rare Events
Covers upsampling (oversampling) to duplicate and saturate positive observations so machine learning algorithms learn rare events, and how ThingWorx Analytics automatically copies positive observations to improve detection during scoring.
2:27 – Downsampling & Holdout Samples
Describes downsampling as removing or desaturating negative observations to raise the positive class occurrence rate, and why a holdout sample is set aside before saturation to avoid an overly optimistic model — handled automatically by ThingWorx Analytics.
2:58 – Upsampling vs. Downsampling: Tradeoffs
Compares the two approaches: upsampling adds record copies that increase processing power and time, while downsampling risks losing information through random sampling bias — noting upsampling is far more common in practice.
3:28 – K-Fold Cross-Validation & ThingWorx Workaround
Explains K-fold cross-validation, dividing the dataset into equal folds and averaging performance across iterations for a robust, unbiased assessment, plus a manual step-by-step workaround since ThingWorx Analytics doesn't support it out of the box.
5:28 – Bagging / Bootstrapping Resampling
Details bagging (bootstrapping) as sampling with replacement into bags, building and averaging a model per bag to reduce variance, typically drawing 50–100 samples — powerful for small datasets and achievable in ThingWorx via a modified column approach.
6:39 – Overfitting: Noise vs. Underlying Relationships
Introduces overfitting, where an excessively complex model with too many parameters relative to observations captures random noise instead of the true relationship, exaggerating minor fluctuations and producing poor predictive performance.
7:15 – Occam's Razor & Model Generalization
Applies Occam's razor (the law of parsimony) to predictive modeling, contrasting lines that overfit by connecting every point against well-generalized models that follow the overall trend and score new records more reliably; bootstrapping, cross-validation, and feature engineering help reduce overfitting.
8:39 – Detecting Overfitting: Training vs. Validation
Shows how to spot overfitting by validating twice — once on training data and once on validation data — where strong training metrics paired with weak validation metrics reveal an overconfident, overfit model.
9:05 – Data Traits That Cause Overfitting: Discrete Values & Boolean Pivoting
Explains how features with many discrete values (zip codes, city names, product IDs, diagnostic codes) trigger overfitting, since ThingWorx Analytics pivots each value into new Boolean columns — a 10,000-value zip code column becomes 10,000 columns, easily exceeding the record count.
10:15 – Feature Engineering: Binning, Transformation, Text Extraction & Logit Features
Walks through feature engineering fixes: binning (grouping US states into nine census regions), feature transformation (text to an ordinal 1–4 scale), text-extraction flags from free-form service comments, and logit-transformed numerical features for high-cardinality columns.
11:49 – Performance Metrics by Goal Variable Type
Summarizes metric selection by goal variable: R squared, adjusted R squared, Pearson correlation, and RMSE for continuous targets; model accuracy, ROC, confusion matrix, and Matthews correlation coefficient for Boolean and categorical targets.
12:20 – Choosing an ML Algorithm: Regression, Neural Nets, Trees & Ensembles
Offers algorithm-selection tips: linear regression for continuous values, logistic regression for categorical outcomes, neural networks for both (with overfitting and speed caveats), decision trees for explainable if-then rules, and ensembles for populations with distinct subgroup behavior.