ThingWorx Analytics Training: Module 3 Part 1
- December 23, 2022
- 0 replies
- 1005 views
This video begins Module 3: Data Profiling of the ThingWorx Analytics Training videos. It describes the process of examining your data to make sure that it is suitable for the use case you would like to explore.
This video does not contain any hands-on exercises, and you don't need access to a running ThingWorx Analytics environment.
This video has chapters — use the chapter menu in the player to jump to any section.
Chapter Summaries
0:00 – Module 3 Intro: Data Profiling, Metadata & BeanPro Dataset
This module introduces data profiling in ThingWorx Analytics, explaining how it assesses a data source's suitability for an analytics use case, what metadata is, and how to profile the sample BeanPro dataset used throughout the course.
0:30 – ThingWorx Analytics Data Flow: Raw Data to Champion Model
Walks through the end-to-end ThingWorx Analytics pipeline: raw CSV, streaming IoT, or data-warehouse sources are structured into a contextual repository and dimensional model, then combined with feature engineering and metadata into an analytics-ready view feeding signals, profiles, predictive modeling, and a champion model.
2:10 – What Is Data Profiling: First Step in the Analytics Workflow
Defines data profiling as examining data before it enters a use case to confirm it fits the selected goal; it is the non-destructive first workflow step that guides how to handle data issues and helps reach a first model quickly for iteration.
2:48 – Data Usability & Consistency: Date Formats & Decimal Separators
Covers checking whether data is usable, consistent, and complete, flagging idiosyncrasies like mixed US/Canada versus European or Australian date formats (MM-DD-YYYY vs DD-MM-YYYY) and comma-versus-period decimal separators across records.
3:37 – Data Quality & Fitness of Purpose
Explains assessing whether data provides evidence for the intended goal, since data quality directly impacts model success; a dataset may be large yet missing key features, or high quality but insufficient in volume, and may suit one purpose but not another.
4:23 – Data Types & Statistical Analysis Overview
Discusses how data types such as numbers and strings shape an analytics-ready view, determine acceptable feature values, and dictate correct statistical measurements, with statistical analysis surfacing issues like missing values and outliers early in a use case.
4:56 – Why Profile Data: Business Rules & Daylight Saving Pitfalls
Outlines the benefits of profiling: better source-data understanding, exposing assumptions and pitfalls (like 23- or 25-hour daylight saving days), identifying needs for additional data, and discovering or validating undocumented, inherent business rules within the data.
6:19 – Suitability & Feature Selection: Hospital Readmission Example
Frames suitability as whether data is fit for its intended purpose, recommends consulting subject matter experts, previews feature engineering, and illustrates useful predictive features (lab results, prior length of stay) for a hospital readmission model.
7:26 – Completeness: Nulls, Interpolation & Hidden Missing Values
Introduces statistical analysis elements and metadata discovery, then details completeness: how often an attribute is populated versus null, its importance for time-series models, median interpolation for small gaps, and hidden missing values disguised as 0, 999, or NA.
8:55 – Uniqueness: Distinct Values, Primary Keys & Binning
Explains uniqueness as counting distinct values per attribute, favoring a unique identifier such as a product serial number or auto-incrementing primary key, spotting malformed data and duplicates, and supporting binning for numeric-to-categorical feature engineering.
9:40 – Value Distribution & Outliers: Histograms
Describes value distribution as record counts across feature values, visualized with a histogram, and outliers as extreme values from bad sensor readings or genuine points that can bias machine learning algorithms and must be identified during profiling.
10:29 – Range Analysis: Min, Max, Mean, Median & Imputation
Covers range analysis of minimum, maximum, mean, and median values, using mean or median to impute small amounts of missing data, and noting that outliers can skew the mean so the median is often the more robust imputation choice.
11:01 – Data Profiling Tools: Pandas, SSIS, Talend, Informatica, Pentaho, FactoryTalk
Surveys free and paid profiling tools: Python Pandas describe/info and Pandas Profiling HTML reports, Microsoft SSIS ETL data profiling task, Talend Data Quality with Open Studio, Informatica Data Explorer scorecards, Pentaho drag-and-drop rules, Rockwell Automation FactoryTalk outlier filtering, and custom database queries.
12:44 – Recap: Data Profiling Summary
Recaps data profiling as actions that reveal potential dataset issues, confirming usability, goal support, sufficient quality and fitness of purpose, data types, and statistical conclusions needed to build accurate predictive models that enable decision making.