IoT & Connectivity Tips

ThingWorx Manufacturing Tips & Tricks Webinar is a weekly opportunity to hear PTC Subject Matter Experts present on various topics related to the manufacturing space and applications.   Agenda for this week's recorded session - - Overview and demo of the MFG Apps - Aron Semle, Solution Manager - Licensing Options - Serge Romano, VP Manufacturing -Installation Options and configuration of the apps - Varathan Ranganathan, SME - Q&A  

Why we need improve ThingWorx Query Performance? ThingWorx is good at injesting data from systems and devices and persisting data in Value Stream. When users build mashups/services which pull anything more than small amounts of data back, or have many users making queries at the same time, they encounter slow performance and/or server failure. Let's take a typical example which may happen among many of our end users: User A is currently gathering data from their edge devices and placing them in a Value Stream. About 80 properties are sharing the same Value Stream and they are estimating roughly 10 million rows a month of data into the platform. Their application requires querying large sections of this data and displaying aggregate info to certain mashups. A ton of work has been done to optimize these queries to make them as efficient as possible. However, querying with results larger than a few thousand rows causes RAM usage to spike so high, that the JVM runs out of heap space and brings down the server. What we have done to improve Query Performance in ThingWorx 8.2? 1. Improvements to the implementation of QueryPropertyHistory and QueryNamedPropertyHistory services:      1) These two query services are optimized by:          (1) Before ThingWorx 8.2, we query property history from database for each properties(imagine 80 properties in our example in the beginning), and then combine the            dataset(80 datasets in our example!) and display them in one table.          (2) In ThingWorx 8.2, we have replaced the "query for each property" with "single query for all properties"      2) Moved most of the query work to database level which is formally done post processing after grabing all the dataset from database including:          (1) Moved Combiner logic to database          (2) Moved Filter functionality in database Note: This improvement is only implemented on MS SQL and PostgreSQL, other persistence providers are not considered in this version, and this may be the plan for future release. 2. Improved string handling for the Query<Basetype>PropertyHistory services Memory reduction for these services are implemented by storing only one copy of each distinct string How much memory reduction could the Query Service Improvement bring QueryPropertyHistory and QueryNamedPropertyHistory Services From our internal test scenario we have seen approximately a 20% reduction in Thingworx platform memory utilization for these services with no Filter applied. Filtering may reduce memory an additional amount. Besides, a approximately 10% execution time saved as a result of this new improvement! Query<Basetype>PropertyHistory services Memory reduction is highly dependent on the stored data. From our internal test scenario, this improvement is providing up to 10% memory reduction on the platform. Note: This improvement is only implemented on MS SQL and PostgreSQL, so the memory reduction is not applied to other persistence providers. Best practice to call Query Services to improve Performance Although there are some Query service improvements in 8.2, still by following some rules or choosing a suitable service would bring extra performance improvement according to different use scenarios. From this point of view, this secion applies to all ThingWorx versions not specially for 8.2. 1. Limit the number of calls to the Query services from a given mashup/app       By repeatedly calling QueryPropertyHistory to display data may cause severe performance problems. Try to clean up unnecessary service calls in mashup and there should be a significant       improvement to the system.       Note: That was without any of the above improvements. 2. Use Query<BASETYPE>PropertyHistory service as much as possible     Unless customer needs to create a normalized dataset for multiple properties, try using QueryIntegerPropertyHistory, QueryStringPropertyHistory, etc. as they return smaller datasets and do not     use the combiner to normalize the data across a large number of properties. 3. Use QueryNamedPropertyHistory as much as possible     Similarly, QueryNamedPropertyHistory can provide a smaller dataset if you don’t need the values from all the properties on the Thing. So for example if a Thing has 10 logged properties but     you only need 3 returned; using QueryNamedPropertyHistory and identifying the three properties specifically needed will return a significantly reduced data set. 4. Use a Query to narrow down the dataset Where/when it makes sense, use a Query to narrow down the dataset; similarly to option 3, the returned dataset will be smaller. Note: this will have limited use scenario though. What needs to be done for end user when upgrading or utilize the service in ThingWorx 8.2 All the changes are transparent to the user! We improved the underlying implementation of the existing services so users won’t need to do anything after upgrade to see the improvements. There are no changes to the databases or schema.  

The Protocol Adapter Toolkit (PAT) is an SDK that allows developers to write a custom Connector that enables edge devices to connect to and communicate with the ThingWorx Platform.   In this blog, I will be dabbling with the MQTT Sample Project that uses the MQTT Channel introduced in PAT 1.1.2.   Preamble All the PAT sample projects are documented in detail in their respective README.md. This post is an illustrated Walk-thru for the MQTT Sample project, please review its README.md for in depth information. More reading in Protocol Adapter Toolkit (PAT) overview PAT 1.1.2 is supported with ThingWorx Platform 8.0 and 8.1 - not fully supported with 8.2 yet.   MQTT Connector features The MQTT Sample project provides a Codec implementation that service MQTT requests and a command line MQTT client to test the Connector. The sample MQTT Codec handles Edge initiated requests read a property from the ThingWorx Platform write a property to the ThingWorx Platform execute a service on the ThingWorx Platform send an event to the ThingWorx Platform (also uses a ServiceEntityNameMapper to map an edgeId to an entityName) All these actions require a security token that will be validated by a Platform service via a InvokeServiceAuthenticator.   This Codec also handles Platform initiated requests (egress message) write a property to the Edge device execute a service without response on the Edge device  Components and Terminology       Mqtt Messages originated from the Edge Device (inbound) are published to the sample MQTT topic Mqtt Messages originated from the Connector (outbound) are published to the mqtt/outbound MQTT topic   Codec A pluggable component that interprets Edge Messages and converts them to ThingWorx Platform Messages to enable interoperability between an Edge Device and the ThingWorx Platform. Connector A running instance of a Protocol Adapter created using the Protocol Adapter Toolkit. Edge Device The device that exists external to the Connector which may produce and/or consume Edge Messages. (mqtt) Edge Message A data structure used for communication defined by the Edge Protocol.  An Edge Message may include routing information for the Channel and a payload for Codec. Edge Messages originate from the Edge Device (inbound) as well as the Codec (outbound). (mqtt) Channel The specific mechanism used to transmit Edge Messages to and from Edge Devices. The Protocol Adapter Toolkit currently includes support for HTTP, WebSocket, MQTT, and custom Channels. A Channel takes the data off of the network and produces an Edge Message for consumption by the Codec and takes Edge Messages produced by the Codec and places the message payload data back onto the network. Platform Connection The connection layer between a Connector and ThingWorx core Platform Message The abstract representation of a message destined for and coming from the ThingWorx Platform (e.g. WriteProperty, InvokeService). Platform Messages are produced by the Codec for incoming messages and provided to the Codec for outgoing messages/responses.   Installation and Build  Protocol Adapter Toolkit installation The media is available from PTC Software Downloads : ThingWorx Connection Server > Release 8.2 > ThingWorx Protocol Adapter Toolkit Just unzip the media on your filesystem, there is no installer The MQTT Sample Project is available in <protocol-adapter-toolkit>\samples\mqtt Eclipse Project setup Prerequisite : Eclipse IDE (I'm using Neon.3 release) Eclipse Gradle extension - for example the Gradle IDE Pack available in the Eclipse Marketplace Import the MQTT Project : File > Import > Gradle (STS) > Gradle (STS) Project Browser to <protocol-adapter-toolkit>\samples\mqtt, then [Build Model] and select the mqtt project     Review the sample MQTT codec and test client Connector : mqtt > src/main/java > com.thingworx.connector.sdk.samples.codec.MqttSampleCodec decode : converts an MqttEdgeMessage to a PlatformRequest encode (3 flavors) : converts a PlatformMessage or an InfoTable or a Throwable to a MqttEdgeMessage Note that most of the conversion logic is common to all sample projects (websocket, rest, mqtt) and is done in an helper class : SampleProtocol The SampleProtocol sources are available in the <protocol-adapter-toolkit>\samples\connector-api-sample-protocol project - it can be imported in eclipse the same way as the mqtt. SampleTokenAuthenticator and SampleEntityNameMapper are also defined in the <protocol-adapter-toolkit>\samples\connector-api-sample-protocol project. Client : mqtt > src/client/java > com.thingworx.connector.sdk.samples.MqttClient Command Line MQTT client based on Eclipse Paho that allows to test edge initiated and platform initiated requests. Build the sample MQTT Connector and test client Select the mqtt project then RMB > Gradle (STS) > Task Quick Launcher > type Clean build +  [enter] This creates a distributable archive (zip+tar) in <protocol-adapter-toolkit>\samples\mqtt\build\distributions that packages the sample mqtt connector, some startup scripts, an xml with sample entities to import on the platform and a sample connector.conf. Note that I will test the connector and the client directly from Eclipse, and will not use this package. Runtime configuration and setup MQTT broker I'm just using a Mosquitto broker Docker image from Docker Hub​   docker run -d -p 1883:1883 --name mqtt ncarlier/mqtt  ThingWorx Platform appKey and ConnectionServicesExtension From the ThingWorx Composer : Create an Application Key for your Connector (remember to increase the expiration date - to make it simple I bind it to Administrator) Import the ConnectionServicesExtension-x.y.z.zip and pat-extension-x.y.z.zip extensions available in <protocol-adapter-toolkit>\requiredExtensions  Connector configuration Edit <protocol-adapter-toolkit>\samples\mqtt\src\main\dist\connector.conf Update the highlighted entries below to match your configuration :   include "application" cx-server {   connector {     active-channel = "mqtt"     bind-on-first-communication = true     channel.mqtt {       broker-urls = [ "tcp://localhost:1883" ]       // at least one subscription must be defined       subscriptions {        "sample": [ "com.thingworx.connector.sdk.samples.codec.MqttSampleCodec", 1 ]       }       outbound-codec-class = "com.thingworx.connector.sdk.samples.codec.MqttSampleCodec"     }   }   transport.websockets {     app-key = "00000000-0000-0000-0000-000000000000"     platforms = "wss://thingWorxServer:8443/Thingworx/WS"   }   // Health check service default port (9009) was in used on my machine. Added the following block to change it.   health-check {      port = 9010   } }  Start the Connector Run the Connector directly from Eclipse using the Gradle Task RMB > Run As ... > Gradle (STS) Build (Alternate technique)  Debug as Java Application from Eclipse Select the mqtt project, then Run > Debug Configurations .... Name : mqtt-connector Main class:  com.thingworx.connectionserver.ConnectionServer On the argument tab add a VM argument : -Dconfig.file=<protocol-adapter-toolkit>\samples\mqtt\src\main\dist\connector.conf Select [Debug]  Verify connection to the Platform From the ThingWorx Composer, Monitoring > Connection Servers Verify that a Connection Server with name protocol-adapter-cxserver-<uuid> is listed  Testing  Import the ThingWorx Platform sample Things From the ThingWorx Composer Import/Export > From File : <protocol-adapter-toolkit>\samples\mqtt\src\main\dist\SampleEntities.xml Verify that WeatherThing, EntityNameConverter and EdgeTokenAuthenticator have been imported. WeatherThing : RemoteThing that is used to test our Connector EdgeTokenAuthenticator : holds a sample service (ValidateToken) used to validate the security token provided by the Edge device EntityNameConverter : holds a sample service (GetEntityName) used to map an edgeId to an entityName  Start the test MQTT client I will run the test client directly from Eclipse Select the mqtt project, then Run > Run Configurations .... Name : mqtt-client Main class:  com.thingworx.connector.sdk.samples.MqttClient On the argument tab add a Program argument : tcp://<mqtt_broker_host>:1883 Select [Run] Type the client commands in the Eclipse Console  Test Edge initiated requests     Read a property from the ThingWorx Platform In the MQTT client console enter : readProperty WeatherThing temp   Sending message: {"propertyName":"temp","requestId":1,"authToken":"token1234","action":"readProperty","deviceId":"WeatherThing"} to topic: sample Received message: {"temp":56.3,"requestId":1} from topic: mqtt/outbound Notes : An authToken is sent with the request, it is validated by a platform service using the SampleTokenAuthenticator (this authenticator is common to all the PAT samples and is defined in <protocol-adapter-toolkit>\samples\connector-api-sample-protocol) EntityNameMapper is not used by readProperty (no special reason for that) The PlatformRequest message built by the codec is ReadPropertyMessage   Write a property to the ThingWorx Platform In the MQTT client console enter : writeProperty WeatherThing temp 20   Sending message: {"temp":"20","propertyName":"temp","requestId":2,"authToken":"token1234","action":"writeProperty","deviceId":"WeatherThing"} to topic: sample Notes : An authToken is sent with the request, it is validated by a platform service using the SampleTokenAuthenticator EntityNameMapper is not used by writeProperty The PlatformRequest message built by the codec is WritePropertyMessage No Edge message is sent back to the device   Send an event to the ThingWorx Platform   In the MQTT client console enter : fireEvent Weather WeatherEvent SomeDescription   Sending message: {"requestId":5,"authToken":"token1234","action":"fireEvent","eventName":"WeatherEvent","message":"Some description","deviceId":"Weather"} to topic: sample Notes : An authToken is sent with the request, it is validated by a platform service using the SampleTokenAuthenticator fireEvent uses a EntityNameMapper (SampleEntityNameMapper) to map the deviceId (Weather) to a Thing name (WeatherThing), the mapping is done by a platform service The PlatformRequest message built by the codec is FireEventMessage No Edge message is sent back to the device   Execute a service on the ThingWorx Platform ... can be tested with the GetAverageTemperature on WeatherThing ... Test Platform initiated requests     Write a property to the Edge device The MQTT Connector must be configured to bind the Thing with the Platform when the first message is received for the Thing. This was done by setting the bind-on-first-communication=true in connector.conf When a Thing is bound, the remote egress messages will be forwarded to the Connector The Edge initiated requests above should have done the binding, but if the Connector was restarted since, just bind again with : readProperty WeatherThing isConnected From the ThingWorx composer update the temp property value on WeatherThing to 30 An egress message is logged in the MQTT client console :   Received message: {"egressMessages":[{"propertyName":"temp","propertyValue":30,"type":"PROPERTY"}]} from topic: mqtt/outbound   Execute a service on the ThingWorx Platform ... can be tested with the SetNtpService on WeatherThing ...

Protocol Adapter Toolkit (PAT) is an SDK that allows developers to write a custom Connector that enables edge devices (without native AlwaysOn support) to connect to and communicate with the ThingWorx Platform. A typical use case is edge communication using a protocol that can't be changed (e.g. MQTT). Prior to PAT, developers had to use the ThingWorx (Edge or Platform) SDKs, or the ThingWorx REST interface, to enable the edge devices to communicate with ThingWorx. Overview PAT provides three main components: the Channel, the Codec, and the ThingWorx Platform Connection. The Channel implements a network protocol to communicate directly with the Edge Device. Its responsibilities include reading data from an Edge Device, writing data to an Edge Device, and routing data to the correct Codec. You can implement your own custom channel or use one of the out of the box channels provided by PTC : WebSocket, HTTP (1.0.x) and MQTT (1.1.x). The Codec translates messages from your edge devices into messages that ThingWorx platform can process (property read/write,service call, events), and provides a means to take the results of those actions and turn them back into messages for the device.  You must implement the Codec. The Platform Connection layer sends and receives messages with the ThingWorx platform. Note : The PAT Connector capabilities depend on edge protocol and channel implementation. Installation The PAT installation media contains : README.md - start here SDK (Java API) and runtime libraries PAT skeleton project (Gradle) Sample codec implementations for the WebSocket, HTTP, and MQTT channels (Gradle) Sample Custom Channel implementation (basic TCP protocol adapter) (Gradle) Required extensions to be installed on the platform : ConnectionServicesExtension and pat-extension Reference Documents ThingWorx Protocol Adapter Toolkit Developers Guide 1.0.0 README.md in various levels of installation folders ThingWorx Connection Services and Compatibility Matrix 1.0.0 Related Knowledge Protocol Adapter Toolkit - MQTT Sample Project hands-on (1.1.x)

ThingWorx 8.2 System Requirements ThingWorx 8.2 Helpcenter The following feature enhancements and bug fixes exist in ThingWorx 8.2.0: Due to security updates, a minimum version of Apache Tomcat 8.0.47 or 8.5.23 should be used with ThingWorx. Enhancements Platform • Included information about opting out of metrics reporting. For more information, see the ThingWorx Metrics Reporting Services Configuration section in the Platform Subsystem help topic. • The Script Log Error has been added to improve error logging for scripts. • Added support to allow mashups to be rendered using jQuery 3.x runtime. • Query service optimization. This includes improved performance for the QueryPropertyHistory and QueryPropertyNamedHistory services. Previously, a database call was made for every logged property. With this improvement, one database call is made for all logged properties, resulting in the following improvements: ▪ A ~20% decrease in memory usage for the QueryPropertyHistory and QueryNamedPropertyHistory service queries if no filters are applied (PostgreSQL and MSSQL). ▪ Decreased time to execute query (~10%) for the QueryPropertyHistory and QueryNamedPropertyHistory services. Depends on latency to the database (PostgreSQL and MSSQL). ▪ Additional decrease in memory, based on the filter supplied during the query for QueryPropertyHistory and QueryNamedPropertyHistory services. (PostgreSQL and MSSQL). If a filter is applied that reduces the record count by 50%, then there is an additional 50% decrease in memory usage on top of the other 50% described in the first point. This optimization also results in an approximate 10% decrease in memory for single property queries. The Audit Subsystem has been added. It supports the following capabilities: • Automatically add audit entries to online storage. • Search for audit entries (use the QueryAuditHistory service) stored online. • Archive online audit entries to offline storage (automatically performed daily by default). • Export audit data, using the language selected for the export. • Purge online audit data on the basis of a specified number of days for audit data to remain online and also on the specified number of rows to keep online. • Clean up archived audit data automatically, based on a configured schedule. • The security of the PASSWORD base type has been enhanced and is now encrypted. See Passwords for more information. • Added the Collection Widget, which allows you to replicate/repeat mashups and content by using infotables to dynamically supply visual content and data. Refer to the KCS article for additional information here • Additional capability has been added to New Composer. For more information, refer to the ThingWorx Community blog. • The licensing process has been improved. An activation ID is no longer required to obtain a license and a new license file is not required for minor or major release upgrades. ◦ For connected scenarios, activation IDs are no longer required in the platform-settings.json file. ◦ For disconnected scenarios, go to the enhanced PTC Support site pages, select the product, enter a Device ID, and retrieve a license. • You can enable the Application Key Authenticator when SSO is enabled by editing the sso-settings.json configuration. For more information, see Configure the sso-settings.json File. • The CSS Editor was added to Mashup Builder, which allows developers to create modern experiences with responsiveness, animations, and advanced styling and behaviors. Refer to the KCS article for additional information here. • Added support for "Store and Forward" functionality to the interface between KEPServerEX and the ThingWorx platform. KepServerEX can be configured to store updated property data to disk when disconnected from the ThingWorx platform and will send that data gracefully when the connection is re-established. • In mashups, row and column gadget sizes 1 to 8 are now available. TW-25477 Bug Fixes Platform Related JIRA • Fixed an issue with Thing Shapes when editing subscriptions twice before canceling or closing in which the second edit was not saved. TW-28718 • Fixed an issue that was causing SQL Server apparent deadlock exceptions. TW-28208 • Added useful log information for troubleshooting LDAP and Active Directory errors. TW-23873 • Fixed an issue with exception handling in DSLProcessor in which line numbers were not included in the log. TW-18042 TW-17255 • Fixed an issue in which opening/closing brackets are not highlighted if there were 100 or more lines of code in a JavaScript service. TW-12740 Mashup Builder • Service error notification messages were fixed to display on multiple lines based on line breaks in the message. TW-24738 • Fixed an issue in which a master mashup header image was not fully displayed. PSPT-3365 Extensions Related JIRA • The Google Maps JavaScript API was updated to prevent the use of the library without an api key. If you are using the Google Map extension in your application, verify that the extension's metadata.xml file is updated with the correct URL (https://maps.google.com/maps/api/js?sensor=false&key=YOUR_API_KEY). Re-zip the extension and reimport into ThingWorx after making this change.

With the release of ThingWorx 8.1.3, we’ve made a few changes to our licensing model in order to simplify the process.  If you are using any version of ThingWorx 8.1 or later, all that is required to license the platform is a device ID, which is a unique identifier that is requested during the licensing process.  The ID is generated upon deployment of the platform. For those working with ThingWorx 8.1.3 and later, it is possible for the platform to automatically obtain a license from PTC should the server be able to communicate successfully with our license portal.  If the connection is blocked for any reason, or you are using a version of 8.1 prior to 8.1.3, a manual approach is also available via the PTC website. For customers who have already licensed any previous release version of ThingWorx 8.1, there is no need to re-request a license as your current license is compatible with subsequent releases of the platform (e.g. 8.1.3, 8.2.0, 9.0.0, etc.).  You will only need to request a new license file if you purchase new features that aren’t included in your current license. The article at the following link contains all of the necessary details for licensing any version of ThingWorx 8.1 or later. Licensing ThingWorx 8.1 and Later Should you have any questions regarding this process, please feel free to reach out to the ThingWorx Support team for further assistance.  The support team can be reached by logging a case as described in the following link: How to Create a New Support Case and Track an Existing Case on PTC eSupport Portal

What is Meltdown and Spectre? Windows KB Page Blog on Altaro containing  some performance tests with pre/post-patch machines https://www.altaro.com/hyper-v/meltdown-spectre-hyperv-performance/ For any further information about PTC’s reaction to the Meltdown&Spectre, please reference this link:  https://www.ptc.com/en/support/article?n=CS277711

Video Author:                     Asia Garrouj Original Post Date:            March 31, 2017 Applicable Releases:        ThingWorx Analytics 7.4 to 8.1   Description: This video is the second part of a two part video series walking thru the configuration of Analysis Event which is applied for Real-Time Scoring.  This second video will walk you thru the configuration of Analysis Event for Real Time Scoring and validating that a predictions job has been executed based on new input data.    

Video Author:                     Stefan Taka Original Post Date:            July 14, 2017 Applicable Releases:        ThingWorx 7.4   Description: This video will provide you with a brief introduction to the New Composer Interface, which has been made available in the 7.4 and later releases of the ThingWorx Platform.  For complete details on what functionality is available within this next generation composer interface, and to also see what lies ahead on our road map, please refer to the following post in the Community.   NG Composer feature availability  

Video Author:                     Durgesh Patel Original Post Date:            June 12, 2017 Applicable Releases:        ThingWorx   Description: In this video we review the different features available in Grid Advanced Extension version 2.0.  

Original Post Date:            September 30, 2016   Description: This tutorial video will walk you through the installation process for the PostgreSQL-based version of the ThingWorx Platform (7.2) in a RHEL environment.  All required software components will be covered in this video.    

Video Author:                     Polina Osipvoa Original Post Date:            June 10, 2016 Applicable Releases:        ThingWorx   Description: This is a video tutorial on creating a Media Entity, and importing and displaying an image.      

Video Author:                     Polina Osipova Original Post Date:            June 10, 2016   Description: This is a video tutorial on configuring properties for a Thing, and using "Manage Bindings" to bind properties to a Thing.      

Video Author:                     Polina Osipova Original Post Date:            June 10, 2016   Description: This is a video tutorial on adding State Formatting in a Mashup using State Definitions.      

Original Post Date:     June 6, 2016 Description: This is a video tutorial on creating a Value Stream, adding the Value Stream to a Remote Thing, adding, binding and querying the (remote) properties.      

Original Post Date:     June 6, 2016 Description: This tutorial video will walk you through the installation process for the PostgreSQL-based version of the ThingWorx Platform in a Windows environment.  All required software components will be covered in this video.    

Original Post Date:            June 6, 2016   Description: This is a video tutorial on creating a Stream, adding a Data Shape with properties, and writing values to the Stream.    

Video Author:                      Stefan Taka Original Post Date:            June 6, 2016   Description: In part 2 of the ThingWorx REST API tutorial, we will demonstrate various common uses of the ThingWorx REST API.      

Video Author:                     Stefan Taka Original Post Date:            June 6, 2016   Description: In part 1 of the ThingWorx REST API tutorial, we will introduce you to the ThingWorx API structure, and also demonstrate how the API can be invoked.    Blog post with text examples found here:  REST API Overview and Examples    

Original Post Date:      June 6, 2016   Description: This is a video tutorial on creating an InfoTable through a service, creating and adding a Data Shape adding rows to the Infotable through a service, adding rows to an InfoTable property and querying the InfoTable.    

Video Author:                     Stefan Tatka Original Post Date:            June 6, 2016   Description: This ThingWorx Tutorial will demonstrate how to configure and initiate remote file transfers using the .NET SDK.