cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

ThingWorx Navigate is now Windchill Navigate Learn More

IoT & Connectivity Tips

Sort by:
First we need to Understand below terms: Quantitative Variable: A quantitative variable is naturally measured as a number for which meaningful arithmetic operations make sense. Examples: Height, age, crop yield, GPA, salary, temperature, area, air pollution index (measured in parts per million), etc. Categorical variable: Any variable that is not quantitative is categorical. Categorical variables take a value that is one of several possible categories. As naturally measured, categorical variables have no numerical meaning. Examples: Hair color, gender, field of study, college attended, political affiliation, status of disease infection. Ordinal Variables: An ordinal variable is a categorical variable for which the possible values are ordered. Ordinal variables can be considered “in between” categorical and quantitative variables. Example: Educational level might be categorized as     1: Elementary school education     2: High school graduate     3: Some college     4: College graduate     5: Graduate degree •    In this example (and for many ordinal variables), the quantitative differences between the categories are uneven, even though the differences between the labels are the same. (e.g., the difference between 1 and 2 is four years, whereas the difference between 2 and 3 could be anything from part of a year to several years) •    Thus it does not make sense to take a mean of the values. •    Common mistake: Treating ordinal variables like quantitative variables without thinking about whether this is appropriate in the particular situation at hand. Ordinal regression: In statistics, ordinal regression (also called "ordinal classification") is a type of regression analysis used for predicting an ordinal variable. The Ordinal Regression procedure allows you to build models, generate predictions, and evaluate the importance of various predictor variables in cases where the dependent (target) variable is ordinal in nature. Ordinal dependents and linear regression: When you are trying to predict ordinal responses, the usual linear regression models don't work very well. Those methods can work only by assuming that the outcome (dependent) variable is measured on an interval scale. Because this is not true for ordinal outcome variables, the simplifying assumptions on which linear regression relies are not satisfied, and thus the regression model may not accurately reflect the relationships in the data. In particular, linear regression is sensitive to the way you define categories of the target variable. With an ordinal variable, the important thing is the ordering of categories. So, if you collapse two adjacent categories into one larger category, you are making only a small change, and models built using the old and new categorizations should be very similar. Unfortunately, because linear regression is sensitive to the categorization used, a model built before merging categories could be quite different from one built after. Below are some examples pf ordered logistic regression: Example 1: A marketing research firm wants to investigate what factors influence the size of soda (small, medium, large or extra large) that people order at a fast-food chain. These factors may include what type of sandwich is ordered (burger or chicken), whether or not fries are also ordered, and age of the consumer. While the outcome variable, size of soda, is obviously ordered, the difference between the various sizes is not consistent. The difference between small and medium is 10 ounces, between medium and large 8, and between large and extra large 12. Example 2: A researcher is interested in what factors influence modaling in Olympic swimming. Relevant predictors include at training hours, diet, age, and popularity of swimming in the athlete’s home country. The researcher believes that the distance between gold and silver is larger than the distance between silver and bronze. Example 3: A study looks at factors that influence the decision of whether to apply to graduate school. College juniors are asked if they are unlikely, somewhat likely, or very likely to apply to graduate school. Hence, our outcome variable has three categories. Data on parental educational status, whether the undergraduate institution is public or private, and current GPA is also collected. The researchers have reason to believe that the “distances” between these three points are not equal. For example, the “distance” between “unlikely” and “somewhat likely” may be shorter than the distance between “somewhat likely” and “very likely”. How to use and get result by Ordinal Regression: Clink this link for PDF                                                                                                                                                                                                                                                                                                                        PDF source: http://www.norusis.com
View full tip
In this video we are going through a few steps to ensure the installation process went fine.   Updated Link for access to this video:  Validating a ThingWorx Analytics Server 8.1 Installation
View full tip
In this video we are walking through the installation steps of ThingWorx Analytics Server 8.1. This cover the Native Linux installation though the steps will be similar for a docker installation on Windows or Linux.   Updated Link for access to this video:  Installing ThingWorx Analytics Server 8.1 - Native Linux
View full tip
Updates: App Keys defaults - Now stored in secure keystore - Newly created app keys stored automatically - On upgrated existing app keys are migrated to secure keystore Change the app key default expiration time to 1 day - Changed from 100 years - UI date picker - If date not picked now defaults to 1 day Best Practice: - Carefully consider expiration - Set to desired value at time of creation - Scripts should carefully choose time -Knowledge base article in the works Edge SSL updates C SDK TLS/SSL: C-SDK support for OpenSSL: - Version 1.0.2 that supports tls  1.2. - Tomcat 8 compatible ciphers - EMS will follow soon BYO SSL - Abstraction layer &Documentation - Path to building any SSL for supported environments - Porting - Different version of open ssl: straight forward - Other SSL: some expertise required - Enables other SSL providers: - Burden to validate on SDK developer Possibilities: -AxTLS -WolfSSL -Mocana EMS improvements SafeInt Library -C++ library -Helps prevent integer overflows Better certificate loading support -EMS and LUA script resource can authenticate -Bidirectional EMS's HTTP server now defaults to requiring authentication for LSR Overall theme: secure by default Q: If appkey expired in 1 day, does a new one get automatically created? A: Automatic one is not created, change the expiration date when creating the app key. When it expires - have to create a new one.
View full tip
Licensing summary, installing 8.1: - Now instance specific license used which prevents license sharing and protects our intellectual property further - 2 paths for getting up and running with a license:          -Connected: platform-settings configuration only          -Disconnected: text file generated, self-serve creation from PTC support portal, license_capability_response.bin generated to be placed in platform folder   Connected / Disconnected mode LicensingSubsystem::GetLicenseState returns : UNINITIALIZED in disconnected mode LICENSE_EXISTS in connected mode   User Journey Refer to Licensing ThingWorx 8.1 and Later for specific instructions for generating a license for your instance   NOTE: Each instance needs a valid license file to be running.   Troubleshooting: - If any misconfiguration or connection failures occur, error messages will be thrown to Application log Example: unable to fetch license file with device id; Unable to connect to the PTC license server. Please make sure the LicensingConnectionSettings settings... - Connection attempt will occur upon ThingWorx startup. If connection can't be made, instance will be following disconnected scenario  -Make sure Pop Up Blockers are turned off     Platform Settings - Licensing License Connection Settings (in platform-settings.json - plain text sample) -new section for licensing connection strings: -user name - used to connect to ptc support portal -password - encrypted (recommended) or plain text password used to connect to ptc support portal   "LicensingConnectionSettings": { "username":"<PTC_support_portal_username>", "password":"<PTC_support_portal_password>" }    New Services on Licensing Subsystem - GetInstanceID - returns instance/device ID which is created at startup - WriteLicenseUsageData - writes encrypted license usage (same as system data table) to ThingworxStorage\reports\LicenseUsageReport folder   Instance ID The license file is now bound to the platform Instance ID aka Device ID - (unlike most other PTC products where the license is bound to the hardware : CPUID, MAC, ...) This Instance ID is generated during first startup and stored in the database Instance ID is accessible in composer with LicensingSubsystem::GetInstanceID   Q: What happens when license files are bad or missing? A: If there is an invalid license file, with 8.0 valid license.bin needs to be in the folder before starting up; tomcat will crash. In 8.1 no need for license file as long as connected (platform-settings.json), no need to take an extra step, dynamic connection.  In disconnected scenario, ThingWorx will run for 30 days in limited mode with monitoring mashup accessible to check logs.   Q: Where is the InstanceID stored ? A: It's stored in the database   Q: Will the instanceID change during updates (minor 8.1.0 to 8.1.1) A: Device id's don't change.   Q: What will happen in disconnected scenario if there is no valid license after 30 days? The application will not start anymore or user is not able to login? A:  It shuts down, so there is no more "limited" mode. However, a user can come along on day 55 (for example) and can drop in a valid license and start the web app to get it fully running.   Q: What happens if the customer has to reinstall the platform after the license has been fetched ? Is it possible to "return" the license ? A: Reinstalling the platform results in the generation of a new Device ID, therefore a new license file will need to be generated.  
View full tip
Excited to announce ThingWorx 8.1 is officially available in our Support Portal. Please find the release notes below. The following feature enhancements and bug fixes exist in ThingWorx 8.1.0: Enhancements Platform: • Metrics Reporting is enabled by default, which allows usage, performance, and diagnostics data to be sent to a PTC server daily. For more information about this setting, see Platform Subsystem. • You can add and configure Notifications in New Composer. For more information, see Adding Notifications. • License files are now instance specific.. • Security for application keys has been enhanced. The defualt expiration date has been changed to 24 hours if it is not explictly set. • Additional capability has been added to New Composer. • Improvements to anomaly detection accuracy have been added. As a result, data collection restart is no longer necessary after a long gap and the H2 database that installs with the Training Microservice is stored in memory, not as a persisted file. For more information, see Anomaly Detection. • You can now load configuration/project files from KEPServerEX instances Bug Fixes Platform • Fixed an issue where Tomcat failed to start when using SAP HANA. TW-22191 • Fixed an issue that was preventing ThingWorx from starting after the File Transfer Subsystem was disabled. TW-22177 • Fixed an issue where the change history of a Mashup was automatically updated even if no changes were made. TW-22114 • Fixed an issue that was preventing the ServiceInvokeCompleted event from working after performing an in-place upgrade. TW-21784 • Fixed an issue where alert notifications were not being sent to recipients after removing a recipient. TW-21585 • Fixed an issue where the Add button in the Services page did not display after creating a Data Table. TW-21518 • Fixed an issue with alert notifications for entities containing periods in the name. TW-21347 • Fixed an issue that was causing connected assets to display as disconnected in ThingWorx Utilities. UTL-4698 • Fixed an issue where data bind was lost after changing Read-Only settings to Read/Write in Composer. TW-23506 • Fixed an issue that was causing a MetricsReportingTask error after enabling ThingWorx Performance Advisor. TW-21141 • Fixed an issue with the ThingWorx authentication window when specifying the site while using FF and IE. TW-21271 Mashup Builder • Fixed an issue with the List widget that was causing incorrect tooltips to display. TW-24012 TW-23961 TW-23038 • Fixed an issue where Chrome was automatically retrying Remote Service calls when a timeout occurred. TW-23828 • Fixed an issue after restarting the ThingWorx web app where the Runtime or Composer’s index.html were missing. TW-23984 • Fixed an issue where closing a modal dialogue did not remove the disabled state from an element. TW-11217 • Fixed an issue when creating a popup with the Navigation widget. The tab sequence of the popup was dependent on the original mashup. TW-11151 • Fixed an issue with localized values of data columns when using the Data Filter widget. TW-11059 Extensions  • Fixed an issue where CSV parser extension import failed if the text file that was being imported did not include a new line character at the end of the last line of text. TW-21863 • Fixed an issue with the Advanced Grid widget where the Reset button was not localized. TW-21457 • Fixed an issue with the jQuery library used by the WebSocketTunnel_ExtensionPackage widget. Note If you are using the WebSocketTunnel_ ExtensionPackage, you will need to upgrade to version 3.0.2 if you are upgrading to ThingWorx 8.1.0. To upgrade the extension, go to the Web Sockets Tunnel Widget and Library page of the ThingWorx Marketplace. TW-24465 End of Life Information SQUEAL functionality has been discontinued in 8.1. System requirements: http://support.ptc.com/WCMS/files/173583/en/ThingWorx_Core_8.1_System_Requirements_1.0.pdf Installation guide: http://support.ptc.com/WCMS/files/173600/en/Installing_ThingWorx_8.1_1.0_.pdf ThingWorx 8.1 Cross Platform Highlights: Security ThingWorx 8.1 Cross Platform Highlights and Q&amp;A: Licensing
View full tip
If you've installed and used version 8.0.0 of the Manufacturing Apps and would like to extend your usage beyond the trial expiration time, we invite you to use the new version of the Apps (8.0.1), which now includes Production Advisor.   You will need to uninstall and reinstall the application to begin this process.  You will be able to preserve your configuration data by exporting it, but you will only be able to import it in the commercial version of the Application.  To export your configuration: navigate to the hamburger menu at the top right of the screen and select "Export Configuration and Data" To move to the latest release of the free edition of the application (8.0.1): 1. Uninstall the applications.  Navigate to the Start Menu > ThingWorx Manufacturing Apps > Uninstall. 2. Download the latest version: https://www.ptc.com/en/thingworx/manufacturing-apps/Dashboard/Download-Apps 3. Run the installer.   If you have any questions or concerns, feel free to post a question in the manufacturing apps community, or in response to this post; we will be happy to assist you!   - The PTC Manufacturing Team
View full tip
I had tested this on a Raspberry Pi 3 which was working fine. I'm not sure if there's any resource limitation on the Pi 2 that would cause potential issues. I however had some performance issues when using the H2 version, therefore I stuck to the Neo4J version. As there were some changes introduced in ThingWorx 7.4 I would also stick to 7.2 or 7.3. If the Tomcat folder exists in the webapps, that means that initial deployment has been successful. I'd recommend checking the Tomcat logs and the ThingWorx ApplicationLog / ConfigurationLog for any potential issues.
View full tip
    About   This is part of a ThingBerry related blog post series.         ThingBerry is ThingWorx installed on a RaspBerry Pi, which can be used for portable demonstrations without the need of utilizing e.g. customer networks. Instead the ThingBerry provides its own custom WIFI hotspot and allows Things to connect and send / receive demo data on a small scale.   In this particual blog post we'll discuss on how to connect a ESP8266 module to the ThingBerry WIFI hotspot and send data from a DHT-11 sensor via the MQTT protocol.   As the ThingBerry is a highly unsupported environment for ThingWorx, please see this blog post for all related warnings.   Install MQTT broker on the ThingBerry     To install mosquitto as a MQTT broker, log in to the ThingBerry and run     sudo apt-get install mosquitto   This will provide a basic broker installation, which is good enough for this example. MQTT clients (including ThingWorx) will connect to this broker to exchange messages. There will be no added security like encrypted traffic shown in this example, it's however good practise to secure MQTT broker / client connections.   While the ESP8266 module is publishing information, ThingWorx will subscribe to the corresponding topics to update its internal property values with what is sent by the ESP8266 module.   For more information on MQTT, how to configure it for ThingWorx or more security relevant information also see   https://community.thingworx.com/message/5063#5063 https://community.thingworx.com/community/developers/blog/2016/08/08/securing-mqtt-connection-to-thingworx-platform?sr=tcontent   Configure the ESP8266     There are too many instructions on the web already on how to initially setup the ESP8266 and use it with the Arduino IDE. I'll therefore just refer to Google which covers the topic more extensively than I ever could.   All coding in this example is done in the Arduino IDE and is pushed to the ESP8266 (NodeMCU) via USB. For this you might need to install a CH340g USB driver for the NodeMCU.   In the Arduino IDE under Tools, I have set my environment to   Board: NodeMCU 1.0 (ESP-12E Module) CPU Frequency: 80 MHz Flash Size: 4M (3M SPIFFS) Upload Speed: 115200 Port: COM3   Under Sketch > Include Library > Manage Libraries add / install the following libraries:   DHT sensor library by Adafruit Adafruit Unified Sensor by Adafruit PubSubClient by Nick O'Leary   These bring the libraries necessary to read data from the DHT-11 sensor and to configure the ESP8266 as MQTT client.     Wiring the DHT-11 sensor     The following image shows the PINs on the ESP8266     I'm using a DHT-11 sensor with cables included and already fixed to a board with 3 PINs. In case you're using a different version, there might be additional components and wiring required, like a resistor etc. Google might help here as well.     Ensure that neither board nor sensor are plugged in, and the ESP8266 is powered off.   To hook the sensor up to the ESP8266, join   ( - ) to GND ( + ) to 3.3V (out) to D3   After all the connections are made, connect the ESP8266 via USB to a computer / laptop with the Arudino IDE configured.   Coding   In the Arduino IDE use the following code - adjust the WIFI settings and the MQTT broker configuration. Ensure to rename the ESP_xx name / topic to something more meaningful, e.g. a specific device name (or just leave it as is if in doubt).   Use the ssid and wpa_passphrase from the hostapd.conf used to configure the ThingBerry as WIFI hotspot.   Copy&paste the code below into the Arduino IDE, verify it and upload it to the ESP8266.     If searching for a WIFI connection, the device's blue LED will blink. A successful connection to the broker and publishing the values will result in a static blue LED. In case the LED is off, the connection to the broker is lost or messages cannot be published.   For troubleshooting, use the Serial Monitor function (at 115200 baud) in the Arduino IDE. In case sensor data cannot be read but the wiring is correct and the code addressing the correct PIN verify the sensor is indeed working. It took me a long time to figure out that the first sensor I used was a defective device.   The current configuration sends updates every 10 seconds - longer intervals might make more sense, but can trigger a timeout for the MQTT broker. In this case the program will re-connect automatically and log corresponding messages in the Serial Monitor. This might seem like an error, but is indeed intended behavior by the code and the MQTT broker.     Configure MQTT Thing in ThingWorx     Create a new Thing in ThingWorx based on the MQTT Template. Add two properties:   temperature humidity   Both set to persistent and logged and Data Change Type to ALWAYS. Also configure a Value Stream to log a history of values.   In the configuration, add two more subscriptions. Activate the "subscribe" checkbox and map name (local property) to topic (MQTT topic), e.g.   name = temperature; topic = ESP_xx/temp name = humidity; topic = ESP_xx/hum   Ensure the correct servernames, ports etc. are configured (an empty servername will use the localhost).   Save the configuration. Property values should now be updated from the MQTT broker, depending on what the device is sending.   Code #include "DHT.h" #include "PubSubClient.h" #include "ESP8266WiFi.h" /* * * Configure parameters for sensor and network / MQTT connections * */ // setup DHT 11 pin and sensor #define DHTPin D3 #define DHTTYPE DHT11 // setup WiFi credentials #define WLAN_SSID "mySSID" #define WLAN_PASS "WIFIpassword" // setup MQTT #define MQTTBROKER "mqttbrokerhostname" #define MQTTPORT 1883 // setup built-in blue LED #define LED 2 /* * ============================================================ * * DO NOT CHANGE ANYTHING BELOW * (unless you know what you're doing) * */ // initiate DHT DHT dht(DHTPin, DHTTYPE); // initiate MQTT client WiFiClient wifiClient; PubSubClient client(MQTTBROKER, MQTTPORT, wifiClient); /* * setup */ void setup() { // switch off internal LED pinMode(LED, OUTPUT); digitalWrite(LED, HIGH); // start serial monitor Serial.begin(115200); // start DHT dht.begin(); // start WiFi WiFi.begin(WLAN_SSID, WLAN_PASS); } /* * the loop */ void loop() { // while not connected to WiFi, print "." // after connection exit the loop // blink LED while having no WiFi signal boolean wifiReconnect = false; while (WiFi.status() != WL_CONNECTED) { digitalWrite(LED, LOW); delay(200); Serial.print("."); digitalWrite(LED, HIGH); delay(300); wifiReconnect = true; } // if WiFi has reconnected, print new connection information and turn on LED if (wifiReconnect == true) { // print connection information and local IP address, mac address Serial.println(); Serial.println("WiFi connected"); Serial.println(WiFi.localIP()); Serial.println(WiFi.macAddress()); Serial.println(); // turn on built-in LED to indiciate successful WiFi connection digitalWrite(LED, LOW); } // if MQTT client is not connected, connect again // turn on built-in LED to indicate a successful connection if (!client.connected()) { Serial.println("Disconnected from MQTT server... trying to connect"); if (client.connect("ESP_xx")) { Serial.println("Connected to MQTT server"); Serial.println("Topic = ESP_xx"); digitalWrite(LED, LOW); } else { Serial.println("MQTT connection failed"); digitalWrite(LED, HIGH); } Serial.println(); } // read temperature and humidity from sensor float t = dht.readTemperature(); float h = dht.readHumidity(); if (isnan(t) || isnan(h)) { // if temperature or humidity is not a number, print error Serial.println("Failed retrieving data from DHT sensor"); } else { // print temperature and humidity Serial.print(t); Serial.print("° - "); Serial.print(h); Serial.print("%"); Serial.println(); // only send values to MQTT broker, if client is connected if (client.connected()) { // boolean to check for errors during payload transfer bool isError = false; // create payload and publish values via MQTT client // use buffer to convert float to char* char buffer[10]; dtostrf(t, 0, 0, buffer); if (client.publish("ESP_xx/temp", buffer)) { Serial.print(" published /temp "); } else { Serial.print(" failed /temp "); isError = true; } dtostrf(h, 0, 0, buffer); if (client.publish("ESP_xx/hum", buffer)) { Serial.print(" published /hum "); } else { Serial.print(" failed /hum "); isError = true; } Serial.println(); // on error, turn off LED if (isError == true) { digitalWrite(LED, HIGH); } else { digitalWrite(LED, LOW); } } } // sleep for 10 seconds // if sleep > default mosquitto timeout : a reconnect is forced for each update-cycle delay(10000); }
View full tip
How to enable ThingWorx Performance Advisor Applies To ThingWorx 7.2+ Description How to enable ThingWorx Performance Advisor Resolution In the ThingWorx Composer, go to Systems > Subsystems, select the PlatformSubsystem and choose Configuration In the Metrics Reporting Service Configuration Select the "Enable Metrics Reporting" checkbox to activate Performance Advisor reporting Enter your current PTC credentials (username and password) for either the customer support portal or the developer portal After providing those details, use the Request button to request an Authorization Key. Customer Number and Name will be filled automatically and an Authorization Key is generated which allows the server identifying itself to the PTC environment. Those fields are read-only. ThingWorx is now ready to send Performance Advisor data and metrics to PTC Related FAQ - Performance Advisor for ThingWorx | PTC https://community.thingworx.com/community/developers/blog/2017/05/22/performance-advisor-for-thingworx-explore-configure…
View full tip
The accuracy of a predictive model can be boosted in two ways: Either by embracing Feature engineering or by applying boosting algorithms straight away. There are multiple boosting algorithms like Gradient Boosting, XGBoost, AdaBoost, Gentle Boost etc. Every algorithm has its own underlying mathematics and a slight variation is observed while applying them. While working with boosting algorithms, we have come across two frequently occurring buzzwords: Bagging and Boosting. Bagging: It is an approach where you take random samples of data, build learning algorithms and take simple means to find bagging probabilities. Boosting: Boosting is similar, however the selection of sample is made more intelligently. We subsequently give more and more weight to hard to classify observations. Below are Default Algorithms used in Predictive Models generated in ThingWorx Analytics: Decision Tree Gradient Boost Linear regression Neural Net Random Forrest Logistic Regression Gradient boosting is a machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. It builds the model in a stage-wise fashion like other boosting methods do, and it generalizes them by allowing optimization of an arbitrary differential loss function. Let’s begin with an easy example: Assume, you are given a previous model M to improve on. Currently you observe that the model has an accuracy of 80% (any metric). How do you go further about it? One simple way is to build an entirely different model using new set of input variables and trying better ensemble learners. On the contrary, we have a much simpler way to suggest. It goes like this: Y = M(x) + error What if we are able to see that error is not a white noise but have same correlation with outcome(Y) value. What if we can develop a model on this error term? Like:error = G(x) + error2 Probably, we will see error rate will improve to a higher number, say 84%. Let’s take another step and regress against error2: error2 = H(x) + error3 Now we combine all these together: Y = M(x) + G(x) + H(x) + error3 This probably will have a accuracy of even more than 84%. What if we can find an optimal weights for each of the three learners: Y = alpha * M(x) + beta * G(x) + gamma * H(x) + error4 How Gradient Boosting Works: 1. Loss Function: The loss function used depends on the type of problem being solved. It must be differential, but many standard loss functions are supported and you can define your own. A benefit of the gradient boosting framework is that a new boosting algorithm does not have to be derived for each loss function that may want to be used, instead, it is a generic enough framework that any differential loss function can be used. 2. Weak Learner: Decision trees are used as the weak learner in gradient boosting. Specifically regression trees are used that output real values for splits and whose output can be added together, allowing subsequent models outputs to be added and “correct” the residuals in the predictions. Trees are constructed in a greedy manner, choosing the best split points based on purity scores like Gini or to minimize the loss. 3. Additive Model: Trees are added one at a time, and existing trees in the model are not changed. A gradient descent procedure is used to minimize the loss when adding trees. we have weak learner sub-models or more specifically decision trees. After calculating the loss, to perform the gradient descent procedure, we must add a tree to the model that reduces the loss. Improvements to Basic Gradient Boosting: 1. Tree Constraints: It is important that the weak learners have skill but remain weak. Below are some constraints that can be imposed on the construction of decision trees: Number of trees: ​Generally adding more trees to the model can be very slow to over fit. The advice is to keep adding trees until no further improvement is observed. Tree depth: Deeper trees are more complex trees and shorter trees are preferred. Generally, better results are seen with 4-8 levels. Number of nodes or number of leaves: like depth, this can constrain the size of the tree, but is not constrained to a symmetrical structure if other constraints are used. Number of observations per split: Imposes a minimum constraint on the amount of training data at a training node before a split can be considered Minimum improvement to loss: Is a constraint on the improvement of any split added to a tree. 2. Weighted Updates: The contribution of each tree to this sum can be weighted to slow down the learning by the algorithm. This weighting is called a shrinkage or a learning rate. "Each update is simply scaled by the value of the “learning rate parameter v". 3. Stochastic Gradient Boosting: At each iteration a sub sample of the training data is drawn at random (without replacement) from the full training data set. The randomly selected sub sample is then used, instead of the full sample, to fit the base learner. 4. Penalized Gradient Boosting: The additional regularization term helps to smooth the final learnt weights to avoid over-fitting. Intuitively, the regularized objective will tend to select a model employing simple and predictive functions.
View full tip
To maintain a cleaner look of your ThingWorx server access URL, whether for production or convenience reasons, you may look into setting up redirection. This is a quick example on how to redirect <your.main.url.com> to <your.main.url.com/Thingworx> Go to the /<apache-tomcat-directory>/webapps/ROOT/ and find the "index.jsp" file. Copy that file for backup purposes and replace with a new one, containing the following: <!DOCTYPE html> <html> <head> <title>Redirecting....</title>     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <script type="text/javascript">          function Redirect() {                window.location="/Thingworx/";         }         setTimeout('Redirect()', 0);  </script> </head> Please note that once the URL is hit, it will still append the rest of the /Thingworx query in the address bar (i.e keep the "redirected to" address). You may also utilize this to have your <main.url.com> redirect to one of the mashups. This was the user, with proper permissions in place, may access the mashup directly, bypassing the Composer.
View full tip
It’s critical for us to configure all correct parameters while running your application in Production environment or even in development env. While GUI makes it very user-friendly and easy to set up the right values in the right fields, it's useful to know how to do the same programmatically/without the "Configure Tomcat" utility. One way, if you're using Tomcat as a Windows service, you can adjust the JVM options by going to the bin dir and running: tomcat8 //US//MYSERVICENAME ++JvmOptions=-Dexample.license.directory="C:\Program Files\example" Turn the service off before you do this and restart it when you finish. cd $CATALINA_HOME .\bin\service.bat install tomcat .\bin\tomcat8.exe //US//tomcat8 --JvmMs=512 --JvmMx=1024 --JvmSs=1024 Setting the --JvmXX parameters may not be enough. You may also need to specify the JVM memory values explicitly. From the command line it may look like this: bin\tomcat8w.exe //US//tomcat8 --JavaOptions=-Xmx=1024;-Xms=512;.. Be careful not to override the other JavaOptions. But the best and recommended way is to use setenv.sh/setenv.bat (Linux/Windows respectively). It isn't in the as-downloaded Tomcat. But if you look in catalina.sh/catalina.bat, there's a check for a file called setenv. If it's there, it's run. That's where you set JAVA_OPTS, CATALINA_OPTS, etc. We use it to set JAVA_HOME, JAVA_OPTS, CATALINA_OPTS and JPDA_ADDR. Putting all your environment variables into this file is ideal because then you don't have to change the stock startup scripts. Then when monitoring the log we can see the parameters taken:
View full tip
A common issue that is seen when trying to deploy, design or scale up a ThingWorx application is performance.  Slow response, delayed data and the application stopping have all been seen when a performance problems either slowly grows or suddenly pops up.  There are some common themes that are seen when these occur typically around application model or design.  Here are a few of the common problems and some thoughts on what to do about them or how to avoid them. Service Execution This covers a wide range of possibilities and is most commonly seen when trying to scale an application.  Data access within a loop is one particular thing to avoid.  Accessing data from a Thing, other service or query may be fast when only testing it on 100 loops, but when the application grows and you have 1000 suddenly it's slow.  Access all data in one query and use that as an in memory reference.  Writing data to a data store (Stream, Datatable or ValueStream) then querying that same data in one service can cause problems as well.  Run the query first then use all the data you have in the service variables.   To troubleshoot service executions there are a few methods that can be used.  Some for will not be practical for a production system since it is not always advisable to change code without testing first. Used browser development tools to see the execution time of a service.  This is especially helpful when a mashup is slow to load or respond.  It will allow quickly identifying which of multiple services may be the issue. Addition of logging in a service.  Once a service is identified adding simple logging points in the service can narrow what code in the service cases the slow down (it may be another service call).  These logging statements show up in the script logs with time stamps ( you can also log the current time with the logging statements). Use the test button in Composer.  This is a simple on but if the service does not have many parameters (or has defaults) it's a fast and easy way to see how long a service takes to return,' When all else fails you can get thread dumps from the JVM.  ThingWorx Support created an extension that assists with this.  You can find it on the Marketplace with instructions on how to use it.  You can manually examine the output files or open a ticket with support to allow them to assist.  Just be careful of doing memory dumps, there are much larger, hard to analyse and take a lot of memory.  https://marketplace.thingworx.com/tools/thingworx-support-tools Queries ​These of course are services too but a specific type.  Accessing data in ThingWorx storage structures or from external sources seems fairly straight forward but can be tricky when dealing with large data sets.  When designing and dealing with internal platform storage refer to this guide as a baseline to decide where to store data...  Where Should I Store My Thingworx Data?   NEVER store historical data in infotable properties.  These are held in memory (even if they are persistent) and as they grow so will the JVM memory use until the application runs out of it.  We all know what happens then.  Finally one other note that has causes occasional confusion.  The setting on a query service or standard ThingWorx query service that limits the number of records returned.  This is how many records are returned to from the service at the end of processing, not how many are processed or loaded in memory.  That number may be much higher and could cause the same types of issues. Subscriptions and Events ​This is similar to service however there is an added element frequency.  Typical events are data change and timers/schedulers.  This again is often an issue only when scaling up the number of Things or amount of data that need to be referenced.  A general reference on timers and schedulers can be found here.  This also describes some of the event processing that takes place on the platform.  Timers and Schedulers - Best Practice For data change events be very cautions about adding these to very rapidly changing property values.  When a property is updating very quickly, for example two times each second, the subscription to that event must be able to complete in under 0.5 seconds to stay ahead of processing.  Again this may work for 5-10 Things with properties but will not work with 500 due to resources, speed and need to briefly lock the value to get an accurate current read.  In these cases any data processing should be done at the edge when possible (or in the originating system) and pushed to the platform in a separate property or service call.  This allows for more parallel processing since it is de-centralized. A good practice for allowing easier testing of these types of subscription code is to take all of the script/logic and move it to a service call.  Then pass any of the needed event data to parameters in the service.  This allows for easier debug since the event does not need to fire to make the logic execute.  In fact it can essentially be stand alone by the test button in Composer. Mashup Performance This​ one can be very tricky since additional browser elements and rendering can come into play. Sometimes service execution is the root of the issue and reviewed above, other times it is UI elements and design that cause slow down. The Repeater widget is a common culprit. The biggest thing to note here is that each repeater will need to render every element that is repeated and all of the data and formatting for each of those widgets in the repeated mashup. So any complex mashup that is repeated many times may become slow to load. You can minimize this to a degree based on the Load/Unload setting of the widget and when the slowness is more acceptable (when loading or when scrolling). When a mashup is launched from Composer it comes with some debugging tools built in to see errors and execution. Using these with browser debug tools can be very helpful. Scaling an Application When initially modeling an application scale must be considered from the start. It is a challenge (but not impossible) to modify an application after deployment or design to be very efficient. Many times new developers on the ThingWorx platform fall into what I call the .Net trap. Back when .Net was released one of the quote I recall hearing about it's inefficiencies was "memory is cheap". It was more cost efficient to purchase and install more memory than to take extra development time to optimize memory use. This was absolutely true for installed applications where all of the code was complied and stored on every system. Web based applications are not quite a forgiving since most processing and execution is done on the single central web server. Keep this in mind especially when creating Shapes, Templates and Subscriptions. While you may be writing one piece of code when this code is repeated on 1,000 Things they will all be in memory and all be executing this code in parallel. You can quickly see how competition for resources, locks on databases and clean access to in memory structures can slow everything down (and just think when there are 10,000 pieces of that same code!!). Two specific things around this must be stated again (though they were covered in the above sections). Data held in properties has fast access since it is in JVM memory. But this is held in memory for each individual Thing, so hold 5 MB of information in one Thing seems small, loading 10,000 Thing mean instant use of 50 GB of memory!! Next execution of a service. When 10 things are running a service execution takes 2 seconds. Slow but not too bad and may not be too noticeable in the UI. Now 10,000 Things competing for the same data structure and resources. I have seen execution time jump to 2 minutes or more. Aside from design the best thing you can do is TEST on a scaled up structure. If you will have 1,000 Things next year test your application early at that level of deployment to help identify any potential bottlenecks early. Never assume more memory will alleviate the issue. Also do NOT test scale on your development system. This introduces edits changes and other variables which can affect actual real world results. Have a QA system setup that mirrors a production environment and simulate data and execution load. Additional suggestions are welcome in comments and will likely update this as additional tool and platform updates change.
View full tip
This video shows the commands to execute to deploy the training and results microservices as docker container. This is based on Docker Toolbox to highlight the specific settings required on Toolbox.   Updated Link for access to this video:  Deploying Training & Result Microservices via Docker Containers for Anomaly Detection
View full tip
This Expert Session will walk you through the Components involved in the ThingWorx Studio Augmented Reality Environment, a detailed Architecture, supported devices, and exploring the resources. The session shall provide great insight into the working and the technicalities involved in the ThingWorx Studio.   For full-sized viewing, click on the YouTube link in the player controls.   Visit the Online Success Guide to access our Expert Session videos at any time as well as additional information about ThingWorx training and services.
View full tip
This Expert Session is designed to help beginners get up and running with ThingWorx Analytics. It covers basic concepts like: What are APIs, how to configure the metadata file, and a live Demo that shows you how to interact and use ThingWorx Analytics in real time. This Expert Session would also be useful for experienced users who need a refresher course.   For full-sized viewing, click on the YouTube link in the player controls.   Visit the Online Success Guide to access our Expert Session videos at any time as well as additional information about ThingWorx training and services.
View full tip
This Expert Session will walk you through the complete installation of ThingWorx Analytics from the Prerequisites to Confirming the Installation is successful and all steps in between. The first half of the video gives a breakdown of the components and the process of the installation with the second half being an actual Demo of the Installation.   For full-sized viewing, click on the YouTube link in the player controls.   Visit the Online Success Guide to access our Expert Session videos at any time as well as additional information about ThingWorx training and services.
View full tip
This Expert Session is about platform sizing with dependency on the type of environment and correlated scalability options. It talks about federation and high availability as well as provides visual diagrams to understand the architecture of different ThingWorx solutions.   For full-sized viewing, click on the YouTube link in the player controls.   Visit the Online Success Guide to access our Expert Session videos at any time as well as additional information about ThingWorx training and services.
View full tip
​​​There are four types of Analytics:                                                                 Prescriptive analytics: What should I do about it? Prescriptive analytics is about using data and analytics to improve decisions and therefore the effectiveness of actions.Prescriptive analytics is related to both Descriptive and Predictive analytics. While Descriptive analytics aims to provide insight into what has happened and Predictive analytics helps model and forecast what might happen, Prescriptive analytics seeks to determine the best solution or outcome among various choices, given the known parameters. “Any combination of analytics, math, experiments, simulation, and/or artificial intelligence used to improve the effectiveness of decisions made by humans or by decision logic embedded in applications.”These analytics go beyond descriptive and predictive analytics by recommending one or more possible courses of action. Essentially they predict multiple futures and allow companies to assess a number of possible outcomes based upon their actions. Prescriptive analytics use a combination of techniques and tools such as business rules, algorithms, machine learning and computational modelling procedures. Prescriptive analytics can also suggest decision options for how to take advantage of a future opportunity or mitigate a future risk, and illustrate the implications of each decision option. In practice, prescriptive analytics can continually and automatically process new data to improve the accuracy of predictions and provide better decision options. Prescriptive analytics can be used in two ways: Inform decision logic with analytics: Decision logic needs data as an input to make the decision. The veracity and timeliness of data will insure that the decision logic will operate as expected. It doesn’t matter if the decision logic is that of a person or embedded in an application — in both cases, prescriptive analytics provides the input to the process. Prescriptive analytics can be as simple as aggregate analytics about how much a customer spent on products last month or as sophisticated as a predictive model that predicts the next best offer to a customer. The decision logic may even include an optimization model to determine how much, if any, discount to offer to the customer. Evolve decision logic: Decision logic must evolve to improve or maintain its effectiveness. In some cases, decision logic itself may be flawed or degrade over time. Measuring and analyzing the effectiveness or ineffectiveness of enterprises decisions allows developers to refine or redo decision logic to make it even better. It can be as simple as marketing managers reviewing email conversion rates and adjusting the decision logic to target an additional audience. Alternatively, it can be as sophisticated as embedding a machine learning model in the decision logic for an email marketing campaign to automatically adjust what content is sent to target audiences. Different technologies of Prescriptive analytics to create action: Search and knowledge discovery: Information leads to insights, and insights lead to knowledge. That knowledge enables employees to become smarter about the decisions they make for the benefit of the enterprise. But developers can embed search technology in decision logic to find knowledge used to make decisions in large pools of unstructured big data. Simulation: ​Simulation imitates a real-world process or system over time using a computer model. Because digital simulation relies on a model of the real world, the usefulness and accuracy of simulation to improve decisions depends a lot on the fidelity of the model. Simulation has long been used in multiple industries to test new ideas or how modifications will affect an existing process or system. Mathematical optimization: Mathematical optimization is the process of finding the optimal solution to a problem that has numerically expressed constraints. Machine learning: “Learning” means that the algorithms analyze sets of data to look for patterns and/or correlations that result in insights. Those insights can become deeper and more accurate as the algorithms analyze new data sets. The models created and continuously updated by machine learning can be used as input to decision logic or to improve the decision logic automatically. Paragmetic AI: ​Enterprises can use AI to program machines to continuously learn from new information, build knowledge, and then use that knowledge to make decisions and interact with people and/or other machines.                                               Use of Prescriptive Analytics in ThingWorx Analytics: Thing Optimizer: Thing Optimizer functionality provides the prescriptive scoring and optimization capabilities of ThingWorx Analytics. While predictive scoring allows you to make predictions about future outcomes, prescriptive scoring allows you to see how certain changes might affect future outcomes. After you have generated a prediction model (also called training a model), you can modify the prescriptive attributes in your data (those attributes marked as levers) to alter the predictions. The prescriptive scoring process evaluates each lever attribute, and returns an optimal value for that feature, depending on whether you want to minimize or maximize the goal variable. Prescriptive scoring results include both an original score (the score before any lever attributes are changed) and an optimized score (the score after optimal values are applied to the lever attributes). In addition, for each attribute identified in your data as a lever, original and optimal values are included in the prescriptive scoring results. How to Access Thing Optimizer Functionality: ThingWorx Analytics prescriptive scoring can only be accessed via the REST API Service. Using a REST client, you can access the Scoring service which includes a series of API endpoints to submit scoring requests, retrieve results, list jobs, and more. Requires installation of the ThingWorx Analytics Server. How to avoid mistakes - Below are some common mistakes while doing Prescriptive analytics: Starting digital analytics without a clear goal Ignoring core metrics Choosing overkill analytics tools Creating beautiful reports with little business value Failing to detect tracking errors                                                                                                                                 Image source: Wikipedia, Content: go.forrester.com(Partially)
View full tip
This blog is intended to help diagnose and fix the most common issues that may be encountered when working with ThingWatcher. It cannot be stressed strongly enough that you should be familiar with your data including the average time interval between data points, and the collection duration and certainty threshold you specified. Before you start troubleshooting ThingWatcher, check that result and training microservices is running. For testing result microservices open a web browser and paste result URL; http://<IP of microservices>:<Port of results microservices>/results/models (e.g., http://localhost:8096/results/models) For testing training microservices open a web browser and paste training URL; http://<IP of microservices>:<Port of training microservices>/training (e.g., http://localhost:8091/training) If you see either: {"values":[],"total":0,"next":null,"previous":null} or a list of training jobs in JSON format, this means the result and training microservice service is available. 1. Question. I haven't seen an anomaly but I believe that my 'property' is anomalous?         This can be caused by different reasons, here are the most common causes: The certainty is too high. If the certainty is too high ThingWatcher is conservative in its categorization of "true positives" and therefore may emit more "false negatives". Reducing the certainty will change this behavior but note that ThingWatcher may now categorize too many "false positives" as a result. In other words, ThingWatcher may detect the desired anomalies but also some non-anomalies. The 'property' is anomalous during training data collection. If ThingWatcher creates a predictive model from anomalous data, it may not be able to detect the desired anomalies during MONITORING because the data does not really appear to be anomalous. So ThingWatcher treats this pattern as 'normal'. Therefore, ensure that 'property' values are also non-anomalous during training. There are long time gaps during the monitoring state so ThingWatcher stays in Buffering and categorizes these data points as non-anomalous. 2. Question. ThingWatcher detects an anomaly but my 'property' is non-anomalous? The certainty might be too low. In this case, ThingWatcher reports anomalies when the incoming data pattern looks even slightly different from the expected data pattern. ThingWatcher might need more training data. If the 'property' data has a pattern that occurs over a long time span, ThingWatcher needs to collect multiple cycles of all these patterns in order to detect a true anomaly without emitting too many false positives. 3. Question. ThingWatcher is in FAILED State, why?     There are many possible reasons for a failed state, here are the most likely problems that can cause a failed state. ThingWatcher emits a FAILED ThingWatcher State because the training service has not been setup or is down. similarly, the result service is not available. NotemessageText=Unexpected exception. {Throwable=[ConnectException: Operation timed out}]]messageText=Unexpected exception. {Throwable=[ConnectException: Connection refused}]]. Note that ThingWatcher is still able to collect all training data and you will only begin to see these failed states after ThingWatcher tried to post the training request. ThingWatcher emits a FAILED ThingWatcher State because time gaps prevent the data collection for training.You will see this warning in the log messages : "A long time gap was detected in the data that is greater than the threshold of {n}". This means you have a long gap in the training data and ThingWatcher will recollect the data. If there are more than 3 recollections due to a long time gap, ThingWatcher transitions to a failed state and will not be able to recover. In this case you can either instruct ThingWatcher to retrain and try again or check the data source to make sure it does not have long gaps. 4. Question. Why does ThingWatcher remain in Buffering? There are many possible reasons for ThingWatcher to remain in Buffering, but the most likely issue is time gaps which cause ThingWatcher to remain stuck in Buffering. If the incoming data regularly contains long time gaps, you will notice that ThingWatcher keeps alternating between the monitoring and buffering states. You may need to provide better quality data i.e. more evenly spaced data. Source: Alex Meng, Specialist Software Engineer
View full tip
Announcements