IoT & Connectivity Tips

This is a lessons learned write up that I proposed to present at Liveworx but it didn't make the cut, but I did want to share it with all the developer folks. Please note that this is before we added Influx and Micro Services, which help improve the landscape. Oh and it's long 🙂 ------------------------------------------ This is written as of Thingworx 8.2   Different ways to scale Data and Processing with Thingworx Two main issues are targeted Data Storage Platform processing Data Storage in Thingworx Background Issues around storage is that due to the limited indexing in the Persistence Provider with then the actual values according to the datashape being in a JSON Blob So when you look in the Persistence Provider you’ll see Source sourceType Location entityID Datetime Tags ValueJSONBlob   The first six carry an index, the JSON Blob which holds the values according to the datashape is not, that can read something like {value1:firstvalue,value2:secondvalue,value3:[ …. ]} etc. This means that any queries beyond the standard keys – date/time, entityID (name of Stream or DataTable), source, sourcetype, tags, location become very inefficient because it will query the records and then apply the datashape query server side. Potentially this can cause you to pull way more records over from Persistence Provider to Platform than intended. Ie: a Query on Temperature in my data, that should return 25 records for a given month, will perhaps first return 250K records and then filter own to 25. The second issue with storage is that all Streams are stored in one table in the Persistence Provider using entityID as an additional key to figure out which stream the record is for. This means that your record count per table goes up much faster than you’d expect. Ie: If I have defined 5 ValueStreams for 5 different asset types, ultimately all that data is still in one table in the Persistence Provder. So if each has 250K records, a query against the valuestream will then in actuality be a query against 1.25 million records. I think both of these issues are well known and documented? By now and Dev is working on it. Solution approaches So if you are expecting to store a lot of records what can you do? Archive The easiest solution is to keep a limited set and archive off the rest of the data, preferably into a client’s datalake that is not part of the persistence provider, remember archiving from one stream to another stream is not a solution! Unless … you use Multiple Persistence Providers Multiple Persistence Providers Thingworx does support multiple persistence providers for storing data. So you can spin up extra schemas (potentially even in the same DataBase Server) to be the store for additional Persistence Providers which then are mapped to a specific Stream/ValueStream/DataTable/Blog/Wiki. You still have to deal with the query challenge, but you now have less records per data store to query through. Direct queries in the Persistence Provider If you have full access to your persistence provider (NOTE: PTC Cloud Services does NOT provide this right now). You can create an additional JDBC connection to the Persistence Provider and query the stream directly, this allows you to query on the indexed records with in addition a text search through the JSON Blob all server side. With this approach a query that took several minutes at times Platform side using QueryStreamEntries took only a few seconds. Biggest savings was the fact that you didn’t have to transfer so many records back to the Platform server. Additional Schemas You can create your own schema (either within the persistence provider DB – again not supported by PTC Cloud Services) in a Database Server of your choice and connect to it with JDBC/REST. (NOTE: I believe PTC Cloud Service may/might offer a standalone server with actual root access) This does mean you have to create your own Getter/Setter services to retrieve and store information, plus you’ll need some event to store (like DataChange). This approach right now is probably a common if not best practice recommendation if historical information is required for the solution and the record count looks to go over 1 million records and can’t just be queried based on timestamp. Thingworx Event Processing Background Thingworx will consistently deal with many Things that have many Properties, and often times there will be Alerts/Rules that need to run based on value changes. When you are using straight up Alerts based on a limit value, this isn’t such a challenge, but what if you need to add some latch/lock/debounce logic or need to check against historical values or check multiple conditions? How can you design something that can handle evaluating these complex rules, holds some historical or derived values and avoid race conditions and be responsive? Potential Problems Race conditions Multiple Events may need to update the same Permanent or Temporary store for the determination of a condition. Duplicates If you don’t have some ‘central’ tracker, you may possibly trigger the same rule multiple times. Slow response You are potentially triggering thousands or more events at the same time, depending on how you’ve set up your logic, your response could become so slow that the next event will be firing before finish and you’ll overload the system. System queue overrun If your events trigger faster than you can handle the events, you will slowly build up and finally overrun the event queue. System Thread count overrun Based on the number of cores in your system, you can overrun the number of threads that can be handled. Connection Pool overrun Each read/write to a stream/datatable but also Property Persist is a usage of the connection pool to your persistence provider. If you fire a lot at once, you can stack up requests and cause deadlocks System out of memory Potentially in handling the events you are depending on in memory information, if that is something that grows over time, you could hit an ‘Out of Memory’ issue. Solution Approaches Batch processing Especially with Agents/Sources that write a set of property updates, you potentially trigger multiple threads that all may need the same source information or update the same target information. If you are able to process this as a batch, you can take all values in account and only process this as a single event and have just a single read from source or single write to target. This will be difficult to achieve when using something like Kepserver, unless it is transferring as something non-standard like MQTT. But if you can have the data come in as a single REST POST this approach becomes possible. In Memory vs. Table/Stream Storage To speed up response time, you can put necessary information into Memory vs. in a DataTable or Stream. For example, if you need the most current received record together with some historical values, you could: Use a Stream but carry the current value because the stream updates async. (ie adding the current value to the stream doesn’t guarantee that when you read from the stream it has already been committed) Use a DataTable because they are synchronous but it can make the execution slow, especially if you are reaching 100K records or more Use an InfoTable or JSON Property, now this information is in memory and runs the fastest and is synchronous. Note that in some speed testing JSON object was faster than InfoTable and way faster than DataTable. One challenge is that you would have to do a full overwrite if you need to persist this information. Doing a full write does open up the danger of a race condition, if this information is being updated by multiple threads at the same time. If it is ok to keep the information in memory than an InfoTable is nice because you can just add/delete rows in memory. I sadly haven’t figured out yet how to directly do this to a JSON object property :(. It is important to consider disaster recovery scenarios if you are only using this in memory Centralized Processing vs. Distributed Processing Think about how you can possibly execute some logic within the context of the Entity itself (logic within the ThingShape/ThingTemplate) vs. having it fire into a centralized Service (sync or async) on a separate Entity. Scheduler or Timer As much as Schedulers and Timers are often the culprit of too many threads at the same time, a well setup piece of logic that is triggered by a Scheduler or Timer can be the solution to avoid race conditions If you are working with multiple timers, you may want to consider multiple schedulers which will trigger at a specific time, which means you can eliminate concurrence (several timers firing at the same time) Think about staggering execution if necessary, by using the hated, looked down upon … but oft necessary … pause() function !!!! Synchronous vs. Asynchronous Asynchronous execution can give great savings on the processing speed of a thread, since it will kick off the asynch parts and continue on. The terrible draw back, you can’t tell when it is finished nor what the resulting output is. As you mix and match synch/asynch vs processing speed, you may need to consider other ways to pick up when an asynch process finishes, some Property elsewhere that will trigger into a DataChange for example. Interesting examples Batch Process With one client there was a batch process that would post several hundred results at once that all had to be evaluated. The evaluation also relied on historical information. So with some logic these properties were processed as a batch, related to each other and also compared to information held in memory besides historically storing the information that came in. This utilized several in memory objects and ultimately also an eval() statement to have the greatest flexibility and performance. Mix and Match With another client, they had a requirement to have logic to do latch/lock and escalation. This means that some information needs to be persisted, however because all the several hundred properties per asset are coming in through Kepware once a second, it also had to be very fast. The approach here was to have the DataChange place information into an in memory infotable that then was picked up by a separate latch/lock/escalation timer to move it over to the persistent side. This allowed for the instantaneous processing of DataChange and Alerts, but also a more persistent processing of latch/lock/escalation logic. In Conclusion Remember that PTC created its software for specific purposes. I don’t think there ever will be a perfect magical platform that will do everything we need and want. Thingworx started out on a specific path which was very high speed data ingest and event platform with agnostic all around connectivity, that provided a very nice holistic modeling approach and a simple way to build UI/UX. Our use cases will sometimes go right past everything and at times to the final frontier aka the bleeding edge and few are a carbon copy of another. This means we need to be innovative and creative. Hopefully all of you can use the expert knowledge you have about our products to create those, but then also be proactive and please share with everyone else!  

An introduction to installing the ThingWorx platform. Information on the environment, prerequisites, and configuration steps when installing ThingWorx. Includes walkthroughs of installing with H2 and PostgreSQL databases, an introduction and demonstration of the Linux installation script, solutions to common installation problems and more.     For full-sized viewing, click on the YouTube link in the player controls.   Visit the Online Success Guide to access our Expert Session videos at any time as well as additional information about ThingWorx training and services.

Welcome to the Thingworx Community area for code examples and sharing.   We have a few how-to items and basic guidelines for posting content in this space.  The Jive platform the our community runs on provides some tools for posting and highlighting code in the document format that this area is based on.  Please try to follow these settings to make the area easy to use, read and follow.   At the top of your new document please give a brief description of the code sample that you are posting. Use the code formatting tool provided for all parts of code samples (including if there are multiple in one post). Try your best to put comments in the code to describe behavior where needed. You can edit documents, but note each time you save them a new version is created.  You can delete old versions if needed. You may add comments to others code documents and modify your own code samples based on comments. If you have alternative ways to accomplish the same as an existing code sample please post it to the comments. We encourage everyone to add alternatives noted in comments to the main post/document.   Format code: The double blue arrows allow you to select the type of code being inserted and will do key word highlighting as well as add line numbers for reference and discussions.

Database backups are vital when it comes to ensure data integrity and data safety. PostgreSQL offers simple solutions to generate backups of the existing ThingWorx database instance and recover them when needed.   Please note that this does not replace a proper and well-defined disaster recovery plan. Export and Import are part of this strategy, but are not reflecting the complete strategy. The commands used in this post are for Windows, but can be adjusted to work on Linux-based systems as well.   Backup   To create a Backup, the export / dump functionality of PostgreSQL can be used.     pg_dump -U postgres -C thingworx > thingworxDump.sql     The -C option will include the statement to create the database in the .sql file and map it to the existing tablespace and user (e.g. 'thingworx' and 'twadmin'). The tablespace and user can be seen in the .sql file in the line with "ALTER DATABASE <dbname> OWNER TO <user>;"   In the above example, we're backing up the thingworx database schema and dump it into the thingworxDump.sql file   Tablespace, username & password are also included in the platform_settings.json   Restore   To restore the database, we just assume an empty PostgreSQL installation. We need to create the DB schema user first via the following commandline:     psql -U postgres -c "CREATE USER twadmin WITH PASSWORD 'ts';"     With the user created, we can now re-generate the tablespace and grant permissions to the twadmin user:   psql -U postgres -c "CREATE TABLESPACE thingworx OWNER twadmin LOCATION 'C:\ThingWorx\ThingworxPostgresqlStorage';" psql -U postgres -c "GRANT ALL PRIVILEGES ON TABLESPACE thingworx TO twadmin;" Finally the database itself can be restored by using the following commandline:   psql -U postgres < thingworxDump.sql This will create the database and populate it with tables, functions and sequences and will also restore the data from the .sql file.   It is important to have database, username and tablespace match with the original system - otherwise granting permissions and re-creating data might fail on recovery. User and tablespace can also be reused, so only the database has to be deleted before restoring it.   Part of the strategy   Part of the backup and recovery strategy should be to actually test the backup as well as the recovery part of the procedure. It should be well tested on a test-environment before deployed to any production environments. Take backups on a regular basis and test for disaster recovery once or twice a year, to ensure the procedure is still valid.   Data is the most important source in your application - protect it!  

  Step 7: Add JAR Resources   You can add external JARs for use by your extension. Every third-party JAR that is not already included in either the ThingWorx Extension SDK or ThingWorx Core needs to be added to your extension project as a JAR Resource. These JAR resources are added to your metadata.xml as a tag and are used to reference the Java classes in the third-party JAR on which the extension depends. You can either use the Add button   or the ThingWorx Menu from the menu bar to add a new JAR resource. By doing so, the JAR is automatically updated in the metadata file and added to the build path. Although ThingWorx allows developers to include jar files for third-party libraries in their code, it is recommended that you avoid adding jar files for common libraries and instead use the jar files that are included with the SDK. Adding jar files to an extension could cause conflicts with jar files already used by the ThingWorx server, either when uploading an extension, or when an extension executes. Even if your extension works today, future updates to ThingWorx may require updates to your extensions. Similarly, packaging a verison of a commonly used library may mean that a customer will not be able to use your extension together with someone else’s extension.   Select the project to which you want to add the jar file to and select New Jar Resource.       Open the directory in which you have stored the training files for this tutorial. Browse to the Jars directory. Select the json-simple-1.1.1.jar file. Add a description and click Finish. NOTE: This will automatically add json-simple-1.1.1.jar to the lib folder and to your project’s build path. Add httpclient-4.5.6.jar, httpcore-4.4.10.jar and commons-logging-1.2.jar directly into the twx-lib folder in the Project folder. NOTE: These JARs are included in the group of JARs used by ThingWorx by default. In order to build your extension locally, without bundling the jars into your extension that are available on the ThingWorx server, add the above JARs to your project's build path by right-clicking on your project in the Package Explorer, right-click Your_Extension_Project (ie, MyThingWorxWeatherExtension) and select Build Path -> Configure Build Path. Verify the jars we added are in the build path. Otherwise, click Add JARs, then browse to the directory containing these JARs (lib) and add them. NOTE: twx-lib folder is a hidden folder and does not appear in the Eclipse package explorer. The twx-lib folder can be found in the WeatherExtension project inside the Eclipse workspace directory.   Step 8: Create Services   Now that you have created properties, configuration tables and added the required jars, you can begin to add services to your WeatherThingTemplate.   In this part of the lesson, we’ll add a service, UpdateWeatherInfo that will take a City parameter and update the properties of this template using the values obtained from the openWeatherMap API.   Right click inside the WeatherThingTemplate and select ThingWorx Source->Add Service. Create a new service with name UpdateWeatherInfo. Click Add in the Input Parameters frame to add City parameter with a base type STRING.   Set the name and base type of the Output Parameter based on the value that you want the service to return. For simplification, assume this service returns nothing. Set the Base Type to NOTHING. Click Finish to create the service. Copy and Paste the code for UpdateWeatherInfo as specified below. @ThingworxServiceDefinition(name = "UpdateWeatherInfo", description = "updates weather description and temperature properties", category = "", isAllowOverride = false, aspects = { "isAsync:false" }) @ThingworxServiceResult(name = "Result", description = "", baseType = "NOTHING") public void UpdateWeatherInfo( @ThingworxServiceParameter(name = "City", description = "city name", baseType = "STRING") String City) throws Exception { _logger.trace("Entering Service: UpdateWeatherInfo"); String cityProp = this.getPropertyValue("CurrentCity").getStringValue(); if (City == null){ City = cityProp; } else { this.setPropertyValue("CurrentCity", new StringPrimitive(City)); } String url = "http://api.openweathermap.org/data/2.5/weather?q=" +URLEncoder.encode(City,"UTF-8") + "&appid="+ _appid+"&units=imperial"; // create a http client HttpClient client = new DefaultHttpClient(); // create a get request with the URL HttpGet getRequest = new HttpGet(url); // add Accept header to accept json format response getRequest.addHeader("Accept", "application/json"); // send the get request and obtain a response HttpResponse response = client.execute(getRequest); // if response is successful the status code will be 200. if (response.getStatusLine().getStatusCode() == 200) { BufferedReader br = new BufferedReader(new InputStreamReader(response.getEntity().getContent())); StringBuilder sb = new StringBuilder(); String line = ""; while ((line = br.readLine()) != null) { sb.append(line); } JSONParser parser = new JSONParser(); JSONObject json = (JSONObject) parser.parse(sb.toString()); JSONArray weather = (JSONArray) json.get("weather"); Iterator<JSONObject> it = weather.iterator(); String description = (String) it.next().get("description"); this.setPropertyValue("WeatherDescription", new StringPrimitive(description)); double temp = (Double) ((JSONObject) json.get("main")).get("temp"); this.setPropertyValue("Temperature", new NumberPrimitive(temp)); /* fire event BadWeather */ _logger.trace("Exiting Service: UpdateWeatherInfo"); } }   Troubleshooting Issue Solution The iterator() is undefined in JSONArray Import only org.json.simple.*. Importing other JSON libraries can give this error. HttpClient/HttpGet could not be resolved to a type. Make sure you have imported the jars: httpclient-4.5.2.jar, httpcore-4.4.5.jar and commons-logging-1.2.jar, json-simple-1.1.1.jar as indicated in the previous chapter. Make sure you have imported the following packages in your template by Eclipse   Your code should be similar to the following:   package com.thingworx.weather; import java.io.BufferedReader; import java.io.InputStreamReader; import java.net.URLEncoder; import java.util.Iterator; import org.apache.http.HttpResponse; import org.apache.http.client.HttpClient; import org.apache.http.client.methods.HttpGet; import org.apache.http.impl.client.HttpClientBuilder; import org.json.simple.JSONArray; import org.json.simple.JSONObject; import org.json.simple.parser.JSONParser; import org.slf4j.Logger; import com.thingworx.logging.LogUtilities; import com.thingworx.metadata.annotations.ThingworxBaseTemplateDefinition; import com.thingworx.metadata.annotations.ThingworxConfigurationTableDefinition; import com.thingworx.metadata.annotations.ThingworxConfigurationTableDefinitions; import com.thingworx.metadata.annotations.ThingworxDataShapeDefinition; import com.thingworx.metadata.annotations.ThingworxFieldDefinition; import com.thingworx.metadata.annotations.ThingworxPropertyDefinition; import com.thingworx.metadata.annotations.ThingworxPropertyDefinitions; import com.thingworx.metadata.annotations.ThingworxServiceDefinition; import com.thingworx.metadata.annotations.ThingworxServiceParameter; import com.thingworx.metadata.annotations.ThingworxServiceResult; import com.thingworx.things.Thing; import com.thingworx.types.primitives.NumberPrimitive; import com.thingworx.types.primitives.StringPrimitive; @ThingworxBaseTemplateDefinition(name = "GenericThing") @ThingworxPropertyDefinitions(properties = { @ThingworxPropertyDefinition(name = "CurrentCity", description = "", category = "", baseType = "STRING", isLocalOnly = false, aspects = { "defaultValue:Boston", "isPersistent:true", "isLogged:true", "dataChangeType:VALUE" }), @ThingworxPropertyDefinition(name = "Temperature", description = "", category = "", baseType = "NUMBER", isLocalOnly = false, aspects = { "defaultValue:0", "isPersistent:true", "isLogged:true", "dataChangeType:VALUE" }), @ThingworxPropertyDefinition(name = "WeatherDescription", description = "", category = "", baseType = "STRING", isLocalOnly = false, aspects = { "dataChangeType:VALUE" }) }) @ThingworxConfigurationTableDefinitions(tables = { @ThingworxConfigurationTableDefinition(name = "OpenWeatherMapConfigurationTable", description = "", isMultiRow = false, ordinal = 0, dataShape = @ThingworxDataShapeDefinition(fields = { @ThingworxFieldDefinition(name = "appid", description = "", baseType = "STRING", ordinal = 0, aspects = { "isRequired:true" }) })) }) public class WeatherThingTemplate extends Thing { private static final long serialVersionUID = -5294151832877452442L; public WeatherThingTemplate() {} private static Logger _logger = LogUtilities.getInstance().getApplicationLogger(WeatherThingTemplate.class); private String _appid; @Override public void initializeThing() throws Exception {     super.initializeThing();     _appid = (String) this.getConfigurationSetting("OpenWeatherMapConfigurationTable", "appid"); } @ThingworxServiceDefinition(name = "UpdateWeatherInfo", description = "updates weather description and temperature properties", category = "", isAllowOverride = false, aspects = {     "isAsync:false" }) @ThingworxServiceResult(name = "Result", description = "", baseType = "NOTHING") public void UpdateWeatherInfo(     @ThingworxServiceParameter(name = "City", description = "city name", baseType = "STRING") String City) throws Exception {     _logger.trace("Entering Service: UpdateWeatherInfo");     String cityProp = this.getPropertyValue("CurrentCity").getStringValue();     if (City == null){         City = cityProp;     } else {         this.setPropertyValue("CurrentCity", new StringPrimitive(City));     }     String url = "http://api.openweathermap.org/data/2.5/weather?q=" +URLEncoder.encode(City,"UTF-8") + "&appid="+ _appid+"&units=imperial";     // create a http client     HttpClient client = HttpClientBuilder.create().build();     // create a get request with the URL     HttpGet request = new HttpGet(url);     // add Accept header to accept json format response     request.addHeader("Accept", "application/json");     // send the get request and obtain a response     HttpResponse response = client.execute(request);     // if response is successful the status code will be 200.     if (response.getStatusLine().getStatusCode() == 200) {         BufferedReader br = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));         StringBuilder sb = new StringBuilder();         String line = "";         while ((line = br.readLine()) != null) {             sb.append(line);         }         JSONParser parser = new JSONParser();         JSONObject json = (JSONObject) parser.parse(sb.toString());         JSONArray weather = (JSONArray) json.get("weather");         Iterator<JSONObject> it = weather.iterator();         String description = (String) it.next().get("description");         this.setPropertyValue("WeatherDescription", new StringPrimitive(description));         Double temp = (Double) ((JSONObject) json.get("main")).get("temp");         Number number = (Number) temp;         this.setPropertyValue("Temperature", new NumberPrimitive(number));         _logger.trace("Exiting Service: UpdateWeatherInfo");     } } }     Click here to view Part 4 of this guide.

Background: In very rare situations, it is possible that the Firewall-Friendly Agent process may stop running. If the Agent is not running, no machine monitoring or communication with the Cloud Server is possible. Recommendation: WatchDog is a little known yet very helpful feature available with Firewall-Friendly Agents. This program lets you monitor whether an Agent is running; if it’s not running, WatchDog can restart that Agent if needed. If the Agent process fails, WatchDog can bring it back up! WatchDog can also be configured to watch other processes. You can configure WatchDog to run as a service (for Windows) or daemon (for Linux). You will register the Watchdog to run as a service.  The Watchdog configuration file will specify the process(es) to be monitored and what to do when one exits. The options are to attempt to restart the process or to restart the system. Note: Watchdog detects only if a watched process exits. It will not detect or report on processes that may be “hanging”. Need more information? For information about configuring and using WatchDog, see the Agent User’s Guide for your Agent: either Axeda® Platform Axeda® Gateway User’s Guide (PDF) or Axeda® Platform Axeda® Connector User’s Guide (PDF).

Data Model Implementation Guide Part 1   Overview   This project will introduce you to methods for creating the data model that you have designed and are ready to implement. Following the steps in this guide, you will implement the Data Model you've already designed. After having insight into your desired Data Model, this guide will provide instructions and examples on how to build out your application using the ThingWorx platform. We will teach you how to utilize the ThingWorx platform to implement your fully functional IoT application. NOTE: This guide’s content aligns with ThingWorx 9.3. The estimated time to complete ALL 3 parts of this guide is 60 minutes. All content is relevant but there are additional tools and design patterns you should be aware. Please go to this link for more details.     Step 1: Completed Example   Download the completed files for this tutorial:  DataModelEntities.xml. The DataModelEntities.xml file provided to you contains a completed example of the completed data model implementation. Utilize this file to see a finished example and return to it as a reference if you become stuck during this guide and need some extra help or clarification. Keep in mind, this download uses the exact names for entities used in this tutorial. If you would like to import this example and also create entities on your own, change the names of the entities you create.   Step 2: Data Model Scenario   This guide will implement the scenario shown in the Data Model Design guide. Let's revisit our Smart Factory example scenario. Name Description Operations User to keep the line running and make sure that it’s producing quality products Maintenance User to keep machines up and running so that the operator can crank out products Management User in charge of dispatching production orders and making sure the quotas are being met Conveyor Belts Thing on factory line to pass items along to the next stage Pneumatic Gate Thing on factory line Robotic Arm Thing on factory line Quality Check Camera Final Thing on factory line to ensure quality In order to add this to our solution, we will want to build a "connector" between ThingWorx and the existing system. These connectors will be Things as well. Internal system connection Thing for Production Order System Internal system connection Thing for Maintenance Request System Operator   Required Functionality Description 1 File Maintenance Request 2 Get quality data from assets on their line 3 Get performance data for the whole line 4 Get a prioritized list of production orders for their line 5 Create Maintenance Requests   Required Information Description 1 Individual asset performance metrics 2 Full line performance metrics 3 Product quality readings   Maintenance   Required Functionality Description 1 Get granular data values from all assets 2 Get a list of maintenance requests 3 Update maintenance requests 4 Set triggers for automatic maintenance request generation 5 Automatically create maintenance requests when triggers have been activated   Required Information Description 1 Granular details for each asset: In order to better understand healthy asset behavior 2 Current alert status for each asset: to know if there is something going wrong with an asset 3 When the last maintenance was performed on an asset 4 When the next maintenance is scheduled for an asset 5 Maintenance request info: Creation date, due date, progress notes   Management   Required Functionality Description 1 Create production orders 2 Update production orders 3 Cancel Production orders 4 Access line productivity data 5 Elevate maintenance request priority   Required Information Description 1 Production line productivity levels (OEE) 2 List of open Maintenance requests   Overlapping Matrix   This matrix represents all of the overlapping Components that are shared by multiple types of Things in our system:   Unique Matrix   This matrix represents the unique Components to each type of Thing:     Step 3: LineAsset Thing Template   After prioritizing and grouping common functionality and information, we came up with the list below for the first Thing Template to create, LineAsset with five Properties, one Event, and one Subscription. The breakdown for the LineAsset Thing Template is as follows:   Follow the below instruction to create this Entity and get the implementation phase of your development cycle going.   Line Asset Properties   Let's build out our Properties. In the ThingWorx Composer, click the + New at the top of the screen. Select Thing Template in the dropdown.        3. In the name field, enter LineAsset and set the Project (ie, PTCDefaultProject). 4. For the Base Thing Template field, select GenericThing.     5. Click Save.  6. Switch to the Properties and Alerts tab.  7. Click the plus button to add a new Property.   The Properties for the LineAsset Thing Template are as follows: Name Base Type Aspects Data Change Type State String Persistent and Logged ALWAYS SerialNumber String Persistent, Read Only, and Logged NEVER LastMaintenance DATETIME Persistent and Logged VALUE NextMaintenance DATETIME Persistent and Logged VALUE PowerConsumption NUMBER, Min Value: 0 Persistent and Logged ALWAYS Follow the next steps for all the properties shown in our template property table. Click Add. Enter the name of the property (ie, State). Select the Base Type of the proprty from the dropdown. Check the checkboxes for the property Aspects. Select the Data Change Type from the dropdown.   Click Done when finished creating the property. Your properties should match the below configurations.     Line Asset Event   Switch to the Events tab. Click Add. Enter the name of the Event (ie, Error). Select AlertStatus as the Data Shape. This DataShape will allow us to provide simple information including an alert type, the property name, and a status message.   Click Done. Your Event should match the below configurations.          Line Asset Subscription   Switch to the Subscriptions tab. Click Add. Check the Enabled checkbox. Switch to the Inputs tab. Select the name of the Event (ie, Error). Click Done. Your Subscription should match the below configurations.             Challenge Yourself   We have left the Subscription code empty. Think of a way to handle Error Events coming from your line asset and implement it in this section.   Click here to view Part 2 of this guide. 

This blog post provide information on the technical changes in Thingworx 8.0, New Technical Changes in ThingWorx 8.0.0 Here are some common questions and answers in regards to the Licensing change: Does that mean all the extensions in the marketplace won't be free anymore? Depends on the extensions. The main extensions we are licensing for 8.0 are Navigate, Manufacturing and Utilties. We are not licensing the MailExtension on the marketplace, for example. Partners and customers can still import their custom apps/extensions If TWX connects to RP(remote platform) which has its own subscription based Flexera license (InService, for example), how does this interaction works- license validation.  Is server to sever connection counts as user login direct to PTC product? License files are per TWX instance. For RP, each would have their own license files. User counts (if entitled and enforced) are generic to each system.

This zip file contains my slides and buildable project from my Liveworx 207 developer chat presentation featuring the Thing-e Robot and raspberry pi zero integration with ThingWorx using the C SDK.

Building More Complex Tests in JMeter Overview This is the second in a series of articles which help inform how to do user load testing in ThingWorx. This article picks up where the previous left off, continuing with the project created there. The screenshots do appear a little differently here because a new “Look and Feel” was selected for the JMeter application (switched from “Metal” to “Windows Classic”) to provide more readable screenshots. In this guide, we are going to make the very simple project more complex, working towards a better representation of a real load test. The steps below walk you through how to create and configure thread groups and parameterize the processes and procedures defined by each thread group.   Adding More Thread Groups Within JMeter, thread groups are used to organize the HTTP requests in a test into various processes or procedures, such that different mashups (and all of the HTTP requests required on each) or processes can be executed simultaneously by different thread groups throughout the test. Varying the number of threads in a group is how to vary the number of users accessing that mashup during the test, a number which increases over time in accordance with the ramp up time. The thread group name will also show up in the Summary Report tab at the end of the test, making it easier to parse through and graph the results. Start by renaming the existing thread groups so that their process or procedure names are recognizable at the end of the test: Highlight the line which reads “HTTP(S) Test Script Recorder”. (Optional) Add an Include filter to only capture the URLs relevant to your application using the Requests Filtering tab. For example, with the escape character \ necessary for ‘.’, myhost.mycompany.mydomain becomes: myhost\.mycompany\.mydomain Now record a new thread group clicking the “Start” button: Once the control box shows up in the top left corner, click to open a browser and access the ThingWorx Navigate application. Then click on “View Parts List” or some other mashup: Once the mashup loads, search using a string and/or wildcard, or click on one of the recent results if any exist: Wait for the mashup to fully load with the details on that part or assembly, and then click “Stop” in the recording controller window: All of the HTTP requests performed in the process of loading and using this mashup will be added to the JMeter project here: Next, add a new thread group manually to the project: Highlight the newly created “Thread Group” (default name) and rename it to something that relates to the nature of that process: Drag and drop the new collection of requests so that it is considered a part of the new thread group: Then drag the whole group up so that it is next to the other thread groups in the test: In more complex projects, different thread groups may be added at different times, and each time, the service calls are all assigned an index (at the end of the request URL, for example: <request>-344). These indexes may not be unique depending on how and when the thread groups were created, especially in more complex tests. The easiest way to fix this issue is save the test from the JMeter GUI, then open the JMX file in a text editor and perform a find and replace within the relevant section of text.   This is usually done using a regular expression for the number. For example, if the request name indexes are numbered -500 through -525, a regular expression to increase them to -700 through -725 would be (in Notepad++): Find: -5([0-9])([0-9]) Replace: -7\1\2 Note that if you do not use a Request Filter, sometimes the recorder will log URLs that are not part of the target application, like these “generate” samplers. These URLs are typically happening in the background of the browser to track performance, security and errors. These can be deleted: At other times, you will be repeating steps that are already part of another thread group, for example: logging in. This genidkey is a part of the login, as you can see if you look back at the login thread group. Because logging in is only necessary once, and it is assumed to be complete by the time the test starts on the second thread group, this entire section can be deleted: To see for sure if a request can be removed because it is called in a previous thread group, do a non-case-sensitive search for the name of the request: All of the requests found in this particular instance were performed in the previous thread group, so therefore this entire category can also be deleted: Another odd thing you may see (if you do not use a Request Filter with the recording feature) are “blank” requests like these: The recorder isn’t sure what to call these “non-requests”, so anything like this that isn’t an actual URL within the target application should be deleted. Static downloads should be disabled or deleted from scale testing since they are usually cached by the user browser client. In this ThingWorx example, there are static “MediaEntites” which can be deleted or disabled: Within the JMeter client there is no good way to highlight and reset them all at once, unfortunately. The easiest way to remove all of these at once is to open the JMX file in a text editor and use regex expressions for search and replace “enabled=true” with “enabled=false”. Most text editors have examples on how to use regular expressions within their Help topics. The above example is for Notepad++. Parameterize Thread Groups Parameterization is usually the part of creating a JMeter test that takes the most effort and knowledge. Some requests will require the same information for every thread, information which can therefore be defined statically within the JMeter element rather than being parameterized. Some values used within the JMeter test script can be parameterized as inputs in the top level of the test controller, for example: Duration, RampUp time, ApplicationHost, ApplicationPort.   Other values may be unique to only one thread group and could be defined in a User Defined Variables element within that group controller. The value(s) used within a request can also be determined on the fly by the results of earlier requests within a thread group. These request results typically must be post-processed and parameterized for later thread elements to function correctly.   The highest level values that are unique to each thread should be inputs from a CSV file that are passed into the threads as parameters, for example Username and Password. Data used within the test is usually parameterized in order to better emulate real world application use by multiple users. In the following example, we will parameterize the number of users for each thread group by adding a user- defined variable.   Start by selecting the new thread group and parametrizing the number of threads (i.e. the number of users accessing this mashup at a time during the test). The way to enter a variable is with syntax like this: $(searchandviewpartstructure_threads) In this case, make this a user defined variable: or a variable for the whole project, by highlighting “Test Plan” and adding the information there. Begin looking at the samplers to see what types of things need to be parameterized in your test. Consider such things as: thread count (as shown above), ramp up time (also depicted above), duration, timings, roles, URL arguments, info table information, search result information, etc.   Another example here parameterizes the search parameters for a query by adding an overall search string column to a CSV file (which can then be randomly generated by some other script): First, parameterize the body data of the request by highlighting the request, and changing the value of the desired field to something like this: $(searchString) Next, define the parameter under the Test Plan and set a default value: Now define the searchString column again as part of the CSV Data Set: Now it can be varied simply by providing different pseudo-random values with wildcards and/or known values in the CSV file.   Post Processors and Extractors Most JMeter load tests become more complex when the results of one request are sent as parameters into later requests. This is done in JMeter by using Post Processors (Extractors), tools which facilitate extracting information out of the request results so it can be assigned to JMeter parameters. There are many different types of extractors which can process the results of previous requests: CSS Selector Extractor – commonly used extractor for values returned as html attributes JSON Extractor – processes JSON objects using regular expressions BeanShell Post Processor –facilitates using code scripts to process return text when needed Regular Expression Extractor – JMeter supports use of regular expressions on request results   The JSON Extractor can be used to find and store information like the partOID number for a Windchill part as a parameter in JMeter, which can then be used to build more realistic workflows within the JMeter test. The example below steps you through setting up a JSON Post Processor.   Start by right-clicking the request that contains the results of our search. Then click “Add” > “Post Processors”> “JSON Extractor”, as shown in the image below: The extractor will now show up under that request as a sub-menu item. Select it, and name the variable something easy to reference. For the JSON Path expressions, pull the object number or some other identifying characteristic out of the search results: $.rows[0].objNumber for example. Another option would be to take information like the partOID number send that into the search string field, by defining both as properties and having one refer to the other. To pull the partOID out, use a Regular Expression Extractor: Another thing to parameterize is the summary report result file name. Adding in the number of users and ramp up time can result in files that are easier to reference later being stored on your machine. We will cover generating and reviewing Summary Reports in full in the next article in this series.     Conclusion In this article we saw how to create new thread groups, removing extraneous requests from those groups, and reduce the overall ambiguity of which thread groups are representing which processes or mashup calls. We also covered how to parameterize the individual requests as well as the summary report. Note that things like Windchill URL and hostname, search parameters and part IDs, timings, durations, offsets, anything at all that influence the results of the test, should not be hard-coded. It is better to create variables for these things to ensure that all of the various simulated activities are configured in the exact same way every time. That way, the system can be tested again and again under various strains and loads until the capabilities of the application are verified.

Smoothing Large Data Sets Purpose In this post, learn how to smooth large data sources down into what can be rendered and processed more easily on Mashups. Note that the Time Series Chart  widget is limited to load 8,000 points (hard-coded). This is because rendering more points than this is almost never necessary or beneficial, given that the human eye can only discern so many points and the average monitor can only render so many pixels. Reducing large data sources through smoothing is a recommended best practice for ThingWorx, and for data analysis in general.   To show how this is done, there are sample entities provided which can be downloaded and imported into ThingWorx. These demonstrate the capacity of ThingWorx to reduce tens of thousands of data points based on a "smooth factor" live on Mashups, without much added load time required. The tutorial below steps through setting these entities up, including the code used to generate the dummy data.   Smoothing the Data on Mashups Create a Value Stream for storing the historical data. Create a Data Shape for use in the queries. The fields should be: TestProperty - NUMBER timestamp - DATETIME Create a Thing (TestChartCapacityThing) for simulating property updates and therefore Value Stream updates. There is one property: TestProperty - NUMBER - not persistent - logged The custom query service on this Thing (QueryNamedPropertyHistory) will have the logic for smoothing the data. Essentially, many points are averaged into one point, reducing the overall size, before the data is returned to the mashup. Unfortunately, there is no service built-in to do this (nothing OOTB service). The code is here (input parameters are to - DATETIME; from - DATETIME; SmoothFactor - INTEGER): // This is just for passing the property name into the query var infotable = Resources["InfoTableFunctions"].CreateInfoTable({infotableName: "NamedProperties"}); infotable.AddField({name: "name", baseType: "STRING"}); infotable.AddRow({name: "TestProperty"}); var queryResults = me.QueryNamedPropertyHistory({ maxItems: 9999999, endDate: to, propertyNames: infotable, startDate: from }); // This will be filled in below, based on the smoothing calculation var result = Resources["InfoTableFunctions"].CreateInfoTable({infotableName: "SmoothedQueryResults"}); result.AddField({name: "TestProperty", baseType: "NUMBER"}); result.AddField({name: "timestamp", baseType: "DATETIME"}); // If there is no smooth factor, then just return everything if(SmoothFactor === 0 || SmoothFactor === undefined || SmoothFactor === "") result = queryResults; else { // Increment by smooth factor for(var i = 0; i < queryResults.rows.length; i += SmoothFactor) { var sum = 0; var count = 0; // Increment by one to average all points in this interval for(var j = i; j < (i+SmoothFactor); j++) { if(j < queryResults.rows.length) if(j === i) { // First time set sum equal to first property value sum = queryResults.getRow(j).TestProperty; count++; } else { // All other times, add property values to first value sum += queryResults.getRow(j).TestProperty; count++; } } var average = sum / count; // Use count because the last interval may not equal smooth factor result.AddRow({TestProperty: average, timestamp: queryResults.getRow(i).timestamp}); } } Create a Timer for updating the property values on the Thing. The Timer should subscribe to itself, containing this code (ensure it is enabled as well): var now = new Date(); if(now.getMilliseconds() % 3 === 0) // Randomly reset the number to simulate outliers Things["TestChartCapacityThing"].TestProperty = Math.random()*100; else if(Things["TestChartCapacityThing"].TestProperty > 100) Things["TestChartCapacityThing"].TestProperty -= Math.random()*10; else Things["TestChartCapacityThing"].TestProperty += Math.random()*10; Don't forget to set the runAsUser in the Timer configuration. To generate many properties, set the updateRate to a small value, like 10 milliseconds. Disable the Timer after many thousands of properties are logged in the Value Stream. Create a Mashup for displaying the property data and capacity of the query to smooth the data. The Mashup should run the service created in step 4 on load. The service input comes from widgets on the mashup: Bindings: Place a Time Series Chart widget in the bottom of the Mashup layout. Bind the data from the query to the chart. View the Mashup. Note the difference in the data... All points in one minute: And a smooth factor of 10 in one minute: Note that the outliers still appear, and the peaks are much easier to see. With fewer points, trends become easier to spot and data is easier to understand. For monitoring the specific nature of the outliers, utilize alerts and other types of displays. Alternative forms of data reduction could involve using the mean of each interval (given by the smoothing factor) or the min or max, as needed for the specific use case. Display multiple types of these options for an even more detailed view. Remember, though, the more data needs to be processed, the slower the Mashup will load. As usual, ensure all mashups are load tested and that the number of end users per Mashup is considered during application design.

This script is provided as is and is not officially supported by PTC. Users are welcome to edit or change this script as desired. This script automates the installation steps outlined in the ThingWorx Installation Guide. If you are not familiar with these processes or are installing ThingWorx for the first time, it is recommended to refer to the guide for additional information. Requirements:   64-bit Linux: Ubuntu (all flavors) 14/15/16, Red Hat Enterprise Linux 6/7, or CentOS 6/7   Root privileges   Internet connection   A ThingWorx platform installation file for either H2 or PostgreSQL (available in the PTC downloads center https://support.ptc.com/appserver/cs/software_update/swupdate.jsp )   *NOTE: This script does not require a GUI and can be run in headless mode Minimal download option: The script will look in the working directory for pre-downloaded Java, Tomcat, and (for RHEL and CentOS) PostgreSQL installation files. If the installer detects one of the supported files, the user will be given the option to install with that file and skip the automated download. This feature is limited to specific versions and requires one (or more) of the following files:     jdk-8u92-linux-x64.tar.gz     apache-tomcat-8.0.33.tar.gz     pgdg-redhat94-9.4-3.noarch.rpm     NOTE: This reduces the amount of data downloaded by the script, but does not eliminate it entirely. An active internet connection is still required. Support for existing SSL certificates: If an existing SSL certificate exists and the user wishes to use it instead of automatically generating one with the script, the user can copy the '.keystore' file to the working directory before running the script. During the installation process the user will be prompted with the option to use this keystore file, and will need to provide the keystore password created with the certificate.     NOTE: this script does not support keystore passwords containing the following characters: '!/@\"#$%^&*()_+]    *Note for Red Hat Enterprise Linux users*     This script was designed to work on servers with an active Red Hat subscription. It has been tested and appears to work on unsubscribed servers if the following applications are installed: gcc make epel repository (see This Red Hat Announcement for more information) Download the file attached to this document Steps to use this script: 1. Create a new folder on your Linux server. (This will be referred to as the 'working folder') 2. Download the ThingWorx platform installer file from the PTC Software downloads page. (For example, MED-61111-CD-072_SP2_ThingWorx-Platform-Postgres-7-2-2.zip) 3. Copy the installer file to the working folder. Do not unzip. 4. Copy the installer.sh file to your working folder. 5. In the command prompt, cd to your working directory. 6. Make the installer.sh file excutable:   > chmod +x installer.sh 7. Run the installer.sh file either using sudo or logging in as root   > sudo bash installer.sh   OR   > su   > bash installer.sh   *NOTE: The 'bash' command is mandatory, running the installer with the 'sh' command will not work

Steps to to navigate to different mashups on double click of different google map markers in ThingWorx Create a DataShape with two fields Location MashupName Create a DataTable using the DataShape Add required Location data and corresponding MashupName to be opened in the DataTable Create a mashup Add GoogleMap and Navigate widget to the mashup Bind All Data from QueryDataTableEntries service of the DataTable in the mashup to the GoogleMap widget by dragging and dropping that item onto the GoogleMap and selecting the Data option In GoogleMap widget properties select LocationField as Location Bind MashupName from Selected Row(s) in QueryDataTableEntries service of the DataTable in the mashup to the Navigate widget by dragging and dropping that item onto the Mashup Link and selecting the MashupName option Bind the DoubleClick event of GoogleMap widget properties in the mashup to Navigate widget by dragging and dropping that item onto the Mashup Link and selecting the Navigate option Save and view mashup Copy of my article as blog post: https://support.ptc.com/appserver/cs/view/solution.jsp?n=CS249997&lang=en_US

Setting Up the Azure Load Balancer with a ThingWorx High Availability Deployment Purpose In this post, one of PTC’s most experienced ThingWorx deployment architects, Desheng Xu, explains the steps to configure Azure Load Balancer with ThingWorx when deployed in a High Availability architectural model.   This approach has been used successfully on customer implementations for several ThingWorx 7.x and 8.x versions. However, with some of the improvements planned for ThingWorx High Availability architecture in the next major release, this best practice will likely change (so keep an eye out for updates to come).   Azure Load Balancer The overview article What is Azure Load Balancer? from Microsoft will give you a high-level understanding of load balancers in general, as well as the capabilities and limitations of Azure Load Balancer itself. For our purposes, we will leverage Azure Load Balancer's capability to manage incoming internet traffic to ThingWorx Platform virtual machine (VM) instances. This configuration is known as a Public Load Balancer.   Important Note: Different load balancers operate at different “layers” of the OSI Model. Azure Load Balancer operates at Layer 4 (Transport Layer) – it is indifferent to the specific TCP Payload. As a result, you must either configure both the front-end and back-end to work on SSL, or configure both of them to work on non-SSL communications. “SSL Termination” or “TLS Offload” is not supported by Azure Load Balancer.   Azure offers multiple different load balancing solutions. If you need some guidance on choosing the right one for you, I highly recommending reviewing the Microsoft DevBlog post Azure Load Balancing Solutions: A guide to help you choose the correct option.   High-Level Diagram: ThingWorx High Availability with Azure Load Balancer To keep this article focused, we will not go into the setup of ThingWorx in a High Availability architecture. It will be assumed that ThingWorx is working correctly and the ZooKeeper cluster is managing failover for the Platform instances as expected. For more details on setting up this configuration, the best place to start would be the High Availability Administrator’s Guide.   Planning In this installation, let's assume we have following plan (you will likely need to change these values for your own implementation): Azure Load Balancer will have a public facing domain name: edc.ptc.io Azure Load Balancer will have a public IP: 41.35.22.33 ThingWorx Platform VM instance 1 has a local computer name, like: vm1 ThingWorx Platform VM instance 2 has a local computer name, like: vm2   ThingWorx Preparation By default, the ThingWorx Platform provides a healthcheck end point at /Thingworx/Admin/HA/LeaderCheck , which can only be accessed with a credential configured in platform-settings.json : "HASettings": { "LoadBalancerBase64EncodedCredentials":"QWRtaW5pc3RyYXRvcjphZG1pbg==" } However, Azure Load Balancer does not permit this Health Check with a credential with current versions of ThingWorx. As a workaround, you can create a pings.jsp (using the attached JSP example code) in the Tomcat folder $CATALINA_HOME/webapps/docs . This workaround will no longer be needed in ThingWorx 8.5 and newer releases.   There are two lines that likely need to be modified to meet your situation: The hostname in final String probeURL (line 10) must match your end point domain name. It's edc.ptc.io in our example, don’t forget to replace this with your real hostname! You also need to add a line in your local hosts file and point this domain name to 127.0.0.1 . For example: 127.0.0.1 edc.ptc.io The credential in final String base64EncodedCredential (line 14) must match the credential configured in platform-settings.json. Additionally: Don't forget to make the JSP file accessible and executable by the user who starts Tomcat service for ThingWorx. These changes must be applied to both ThingWorx Platform VM instances. Tomcat needs to be configured to support SSL on a specific port. In this example, SSL will be enabled on port 8443. Please make sure similar configuration is included in $CATALINA_HOME/conf/server.xml <Connector protocol="org.apache.coyote.http11.Http11NioProtocol" port="8443" maxThreads="200" scheme="https" secure="true" SSLEnabled="true" keystoreFile="/opt/yourcertificate.pfx" keystorePass="dontguess" clientAuth="false" sslProtocol="TLS" keystoreType="PKCS12"/> The values in keystoreFile and keyStorePass will need to be changed for your implementation. While pkcs12 format is used in above example, you can use a different certificate formats, as long as it is supported by Tomcat (example: jks format). All other parameters, like maxThreads , are just examples - you should adjust them to meet your requirements.   How to Verify Before configuring the load balancer, verify that health check workaround is working as expected on both ThingWorx Platform instances. You can use following command to verify: curl -I https://edc.ptc.io:8443/docs/pings.jsp The expected result from active node should look like: HTTP/1.1 200 There will be three or more lines in output, depending on your instance configuration but you should be able to see the keyword: HTTP/1.1 200.   Expected result from passive node should look like: HTTP/1.1 503   Load Balancer Configuration Step 1: Select SKU Search for “load balancer” in the Azure market and select Load Balancer from Microsoft Verify the correct vendor before you create a Load Balancer. Step 2: Create load balancer To create a proper load balancer, make sure to read Microsoft’s What is Azure Load Balancer? overview to understand the differences between “basic” and “standard” SKU offerings. If your IT policy only requires SSL communication to the outside but doesn't require a SSL communication in a health probe, then the “basic” SKU should be adequate (not considering zone redundancy). You have to decide following parameters: Region Type (public or Internal) SKU (basic or standard) IP address Public IP address name Availability zone PTC cannot provide specific recommendations for these parameters – you will need to choose them based on your specific business needs, or consult Microsoft for available offerings in your region.   Step 3: Start to configure Once a load balancer is successfully created by Azure, You should be able to see:   Step 4: Confirm frontend IP Click frontend IP configuration at left side and you should be able to see public IP address configuration. Please make sure to register this IP with your domain name ( edc.ptc.io in our example) in your Domain Name Server (DNS). If you unfamiliar with DNS configuration, you should consult with the administrator of your DNS server. If you are using Azure DNS, this Quickstart article on creating Azure DNS Zones and records may help.   Step 5: Configure Backend pools Click Backend pools and click “Add” to add a backend pool definition. Select a name for your Backend pool (using ThingworxBackend in our example). Next step is to choose Virtual network.   Once you select Virtual network, then you can choose which VM (or VMs) you want to put behind this load balancer. The VM should be the ThingWorx VM instance.   In a high availability architecture, you will typically need to choose two instances to put behind this load balancer.   Please Note: The “Virtual machine status” column in this table only shows VM status, but not ThingWorx status. ThingWorx running status will be determined by the health probe configured in the next step.     Step 6: Configure Health Probe Health Probe will be used to determine the ThingWorx Platform’s running status. When a ThingWorx Platform instance is running as the leader, then it will give HTTP status code 200 during a health probe. The Azure Load Balancer will rely on this status code to determine if the platform is running properly.   When a ThingWorx platform VM is not responding, offline, or not the leader in a High Availability setup, then this health probe will provide response with a different HTTP status code other than 200.   For the health probe, select HTTPS for the protocol. In our example port 8443 is used, though another port can be selected if necessary. Then, provide the “/docs/pings.jsp” we created earlier as the probe’s path. You may need to change this path value if you put this file in a different location.   Step 7: Configure Load balancing rules. Select “Load balancing rules” from left side and click “Add”   Select TCP as protocol, in our example we are using 443 as front-end port and 8443 as back-end port. You can choose other port numbers if necessary.   Reminder: Azure Load Balancer is a layer 4 (Transport Layer) router – it cannot differentiate between HTTP or HTTPS requests. It will simply forward requests from front-end to back-end, based on port-forwarding rules defined.   Session persistence is not critical for current versions of ThingWorx as only one active node is currently permitted in a High Availability architecture. In the future, selecting Client IP may be required to support active-active architectures.   Step 8: Verify health probe Once you complete this configuration, you can go to the $CATALINA_HOME/logs folder and monitor latest local_access log. You should see similar entries as pictured below - HTTP 200 responses should be observed from the ThingWorx leader node, and HTTP 503 responses should be observed from the ThingWorx passive node. In the example below, 168.63.129.16 is the internal IP Address of the load balancer in the current region.   Step 9: Network Security Group rules to access Azure Load Balancer On its own, Azure Load Balancer does not have a network access policy – it simply forwards all requests to the back-end pool. Therefore, the appropriate Network Security Group associated with the backend resources within the resource group should have a policy to allow access to the destination port of the backend ThingWorx Foundation server (shown as 8443 here, for example). The following image displays an inbound security rule that will accept traffic from any source, and direct it to port 443 of the IP Address for the Azure Load Balancer.   Enjoy!! With the above settings, you should be able to access ThingWorx via: https://edc.ptc.io/Thingworx (replacing edc.ptc.io with the hostname you have selected).   Q&A Can I configure the health probe running on a port other than the traffic port (8443) in this case? Yes – if desired you can use a different port for the health probe configuration.   Can I use different protocol other than HTTPS for health probe? Yes – you can use different protocol in the health probe configuration, but you will need to develop your own functional equivalent to the pings.jsp example in this article for the protocol you choose.   Can I configure ZooKeeper to support the health probe? No – the purpose of the health probe is to inform the Load Balancer which node is providing service (the leader), not to select a leader. In a High Availability architecture, ZooKeeper determines which VM is the leader and talking with the database. This approach will change in future releases where multiple ThingWorx instances are actively processing requests.   How well does Azure Load Balancer scale? This question is best answered by Microsoft – as a starting point, we recommend reading the DevBlog post: Azure Load Balancing Solutions: A guide to help you choose the correct option.   How do I access logs for Azure Load Balancer? This question is best answered by Microsoft – as a starting point, we recommend reviewing the Microsoft article Azure Monitor logs for public Basic Load Balancer.   Do I need to configure specifically for Websocket and/or AlwaysOn communication? No – Azure Load Balancer is a Layer 4 (Transport Protocol) router - it only handles TCP traffic forwarding.   Can I leverage this load balancer to access all VMs behind it via ssh? Yes – you could configure Inbound NAT rules for this. If you require specific help in configuring this, the question is best answered by Microsoft. As a starting point, we recommend reviewing the Microsoft tutorial Configure port forwarding in Azure Load Balancer using the portal.   Can I view current health probe status on a portal? No – Unfortunately there is no current approach to do this with Azure Load Balancer.

  Add a Google Map to your UI that visually presents geographical data.   GUIDE CONCEPT This project will introduce how to visually present geographical data in your application. Showing data on a map is a valuable feature for IoT application.   Following the steps in this guide, you will utilize the Google Maps Widget and explore it’s ability to show multiple Things.   We will teach you how to use Geological data to convey pertinent information in your UI.   YOU'LL LEARN HOW TO   Download and import the Google Maps Widget extension Create a Mashup and add a Google Maps Widget Configure the Google Maps Widget to display the locations of multiple Things   NOTE: This guide's content aligns with ThingWorx 9.3. The estimated time to complete ALL 2 parts of this guide is 30 minutes.      Step 1: Configure Google Maps Widget   When you download the Google Maps Widget, it does not include an API key. Google allows some limited use of their map API without a key, however it is recommended that you obtain and add your own Google Maps API key or Client ID (for Google Maps API for Work licenses) to the Google Maps Widget.   The ThingWorx hosted server has already been configured with the Google Maps Widget, including an API key for evaluation use.   Refer to the Google Maps API documentation to obtain your own API key, then follow the steps below.   Download the Google Maps widget from PTC Partner, IQNOX NOTE:  It is necessary to create an account on IQNOX, but the download is free       2. In the lower-left side of Composer, click Import/Export, then Import.           3. In the Import From File pop-up, under Import Option select Extension from the drop-down, then click Browse        4. Navigate to the .zip file you downloaded.          5. Click Import in the Import From File pop-up, then click Close after file is successfully imported.        6. In the ThingWorx Browse tab, in the System section, click on Subsystems, then PlatformSubsystem.            7. Click on the Configuration tab, then click the Edit button if you are not already in edit mode.        8. Scroll down to the Required string to connect with Google for Google widgets field where you enter the Google Maps JavaScript URL with your API key: https://maps.googleapis.com/maps/api/js?key=<Your API key>            9. Click Save     Step 2: Add Google Maps Widget to Mashup   Click the Browse folder icon on the top left of ThingWorx Composer.       2. Select Mashups in the left-hand navigation, then click + New to create a new Mashup.                          3. For Mashup Type select Responsive.               NOTE: A Responsive Mashup scales with a browser's screen size. In the steps below we will create 5 containers, one for each widget, to organize how the widgets are presented.        4. Click OK.        5. Enter a name for your Mashup.        6. If Project is not already set, click the + in the Project text box and select the PTCDefaultProject.        7. Click Save        8. Select the Design tab to display Mashup Builder.        9. Click the Widget tab on the top-left, then enter map inside the Filter Widgets field.        10. Drag-and-drop the Google Map widget onto the Mashup.         Step 3: Download Sample Entities   We have created a file with sample entities for this exercise.   Download demoTractors.xml to your computer.       2. Click Import/Export in the lower left of Composer, then select Import          3. Click From File in the drop down, then Browse.        4. Browse to the demoTractors.xml file and click Open.        5. Click Import, then Close after entities are successfully imported.     Step 4: Add Markers to Google Maps Widgets   To display markers on the map you must bind an Info Table to the Widget's Data Property and specify which column has location information. In this example we will use the QueryImplementingThingsWithData Service to bind a group of Things created with the same Template.   Click the + button in the Data tab on the right side of the Mashup Builder window             2. In the Add Data pop-up, click Thing Templates from the Entity Type drop down.                3. Select a Template from the list that has a Location property and was used to create Things. For this exercise, use the ConnectedTractor Template that was imported from the sample file downloaded in the prior step.            4. Enter query into the filter text box then click the arrow to the right of QueryImplementingThingsWithData.                                              5. Click the Execute on Load check box, then click Done. This will cause the QueryImplementingThingsWithData Service to execute as soon as Mashup is loaded.                                          6. Expand Returned Data if you do not see All Data then click and drag All Data onto the Google Maps Widget.        7. Click Data in the Select Binding Target pop-up.          8. In the Properties panel in the lower left, scroll to see the LocationField property and select Location, the name of the Property with location information.       Test Map Operation   Click Save to save your Mashup.       2. Click View Mashup to see the Google Maps Widget displaying the locations of each Thing.     NOTE: The Google Maps Widget has built-in functionality that allows users to pan and resize the map as well as switch to satellite photo maps.     Click here to view Part 2 of this Guide.

With the release of ThingWorx 8.1.3, we’ve made a few changes to our licensing model in order to simplify the process.  If you are using any version of ThingWorx 8.1 or later, all that is required to license the platform is a device ID, which is a unique identifier that is requested during the licensing process.  The ID is generated upon deployment of the platform. For those working with ThingWorx 8.1.3 and later, it is possible for the platform to automatically obtain a license from PTC should the server be able to communicate successfully with our license portal.  If the connection is blocked for any reason, or you are using a version of 8.1 prior to 8.1.3, a manual approach is also available via the PTC website. For customers who have already licensed any previous release version of ThingWorx 8.1, there is no need to re-request a license as your current license is compatible with subsequent releases of the platform (e.g. 8.1.3, 8.2.0, 9.0.0, etc.).  You will only need to request a new license file if you purchase new features that aren’t included in your current license. The article at the following link contains all of the necessary details for licensing any version of ThingWorx 8.1 or later. Licensing ThingWorx 8.1 and Later Should you have any questions regarding this process, please feel free to reach out to the ThingWorx Support team for further assistance.  The support team can be reached by logging a case as described in the following link: How to Create a New Support Case and Track an Existing Case on PTC eSupport Portal

Those who have been working with ThingWorx for many years will have noticed the work done around ingress stress testing and performance optimization.  Adding InfluxDB as a time-series data persistence provider really helped level up these capabilities while simultaneously decreasing the overall resources required by the infrastructure.  However with this ease comes a hidden challenge: query and data processing performance to work it into something useful.   Often It's Too Much Data In general most customers that I work with want to collect far too much data -- without knowing what it will be used for, or what processing will be required in order to make it usable and useful.  This is a trap in general with how many people envision IoT projects, being told by infrastructure providers that cloud storage and compute resources are abundant and cheap and that they should get as much data as possible.  This buildup of data means that more effort needs to be spent working it into something useful (data engineering/feature extraction) and addressing common data issues (quality, gaps, precision, etc.).  This might be fine for mature companies with large data analytics teams; however this is a makeup that I've only seen in the largest of our customers.  Some advice - figure out what you need and how you'll use it, and then collect that.  Work on extracting value today rather than hoping that extra data collected  now will provide some insights years from now.   Example - Problem Statement You got your Thing Model designed, and edge devices connected.  Now you've got data flowing in and being stored every 5 seconds in InfluxDB.  Great progress!  Now on to building the applications which cover the various use cases. The raw data is most likely going to need to be processed and potentially even significantly transformed into other information in order to make it useful.  Turning a "powered on and running" BOOLEAN to an "hour meter" INTEGER is a simple example.  Then you may need to provide a report showing equipment run time hours by day over a month.  The maintenance team may also have asked to look for usage patterns which lead to breakdowns, requiring extracting other data points from the initial one like number of daily starts, average daily run time, average time between restarts. The problem here is that unless you have prepared these new data points and stored them as well (say in a Stream), you are going to have to build these data sets on the fly, and that can be time and resource intensive and not give you the response time expected.  As you can imagine, repeatedly querying and processing large volumes of unchanging raw data is going to have resource and time implications - so this is why data collection and data use need to be thought about separately.   Data Engineering In the above examples, the key is actually creating new data points which are calculated progressively throughout normal operation.  This not only makes the information that you want available when you need it - in the right format - but it also significantly reduces resource requirements by constantly reprocessing raw data.  It also helps managing data purging, because as you create and store usable insights, you can eventually just archive away your old raw data streams.   Direct Database Queries vs. Thingworx Data Services Despite the above being a rule of thumb, sometimes a simple well structured database query can get you exactly what you need and do so quite quickly.  This is especially true for InfluxDB when working with extremely large time-series datasets.  The challenge here is that ThingWorx persistence providers abstract away the complexity of writing ones own database queries, so we can't easily get at the databases raw power and are forced to query back more data than needed and work it into a usable format in memory (which is not fast).   Leveraging the InfluxDB API using the ContentLoader Technique As InfluxDBs API is 100% REST, we can access it using in-built ThingWorx Content Loader services.  Check out this demonstration and explanation video where I talk about how to interact directly with InfluxDB in order to crush massive time-series data and get back much more usable and manageable data sets.  It is important to note here that you should use a read-only database user here, as you should never modify the ThingWorx databases to avoid untested scenarios which may lead to data corruption.   Optimizing ThingWorx query performance with the InfluxDB REST API - YouTube InfluxToolBox ThingWorx demo project (by T. Wobben)      

Video Author:                     Christophe Morfin Original Post Date:            October 2, 2017 Applicable Releases:        ThingWorx Analytics 8.1   Description:​ In this video we will walk thru the installation steps of ThingWorx Analytics Server 8.1.  This covers the Native Linux installation though the steps will be similar for a docker installation on Windows or Linux.    

Presentation associated with the November 10, 2107 Tips & Tricks Webinar.

In this video you would see how to start to use your already created Virtual Image of ThingWorx Analytics using Oracle Virtual Box. This Video is Part-1 of the Series Getting Started with ThingWorx Analytics.   Updated Link for access to this video:  Getting Started with ThingWorx Analytics: Part 1 of 2

This video is the 2 nd part of a series of 3 videos walking you through how to setup ThingWatcher for Anomaly Detection. In this video you will learn how to use “Discover UI” from the “New Composer” to bind simulated data coming through KEPServer for Anomaly Detection.   Updated Link for access to this video:  Anomaly Detection 8.0: Configuring Anomaly Alerts:  Part 2 of 3