IoT & Connectivity Tips

Overview This document is targeted towards covering basic PostgreSQL monitoring and health check related system objects like tables, views, etc. This allows simple monitoring of PostgreSQL database via some custom services, which I'll attach at the end of this document, from the ThingWorx Composer itself. I'll also try to cover short detail on some of the services that are included with the Thing: PostgreSQLHealthCheck which implements Database ThingTemplate   Pre-requisite The document assumes that the user already has ThingWorx running with PostgreSQL as a Persistence Provider.   How to install Usage for this is fairly straight forward, import the Entities.twx and it will create required Thing which implements Database ThingTemplate and some DataShapes. Each Service under the Thing: PostgreSQLHealthCheck has its own DataShape. Feel free edit these services / DataShapes if you are looking to use output of these services  as part of your mashup(s).   Make sure to edit the PostgeSQL's JDBC Connection String, Username & password under the configuration section in order to connect to your PostgreSQL instance under the Thing PostgreSQLHealthCheck which will be created when Entities.twx is imported (attached with this blog)   Note : Users can use these services to query non-ThingWorx related database created with PostgreSQL as part of the external JDBC connection.   Reviewing Services from Thing: PostgreSQLHealthCheck   1. DescribeTableStructure - Takes two inputs **Table Name** and the **Schema Name** in which the ThingWorx database tables exists both inputs have default values that can be modified to match your PostgreSQL schema setup and required table name - It provides information on a Table's structure, see below     2. GetAllPSQLConfig - Provides runtime details on all the configurations done in the postgresql.conf which are in-effect - For detail on pg_settings see Postgresql 9.4 Doc     3. GetAllPSQLConfigLimited Similar to GetAllPSQLConfig, however with limited information   4. GetAllPSQLRoles - Lists all the database roles/users - Also lists their access rights permissions together with OID - Helpful in identifying if the role is active/inactive or carries any limitation on the DB connections     5. GetPG_Stat_Activity - Part of the Statistics Collector subsystem for the PostgreSQL DB - Shows current state of the schema e.g. connections, queries, etc. - For more detail on the output refer to the PostgreSQL 9.4 doc   6. GetPSQLDBLocksInformation - Shows the kind of locks in effect on which database and on which relation (table) - Particularly useful in identifying the relations and what lock mode is enabled on them     7. GetPSQLDBStat - Shows database wide statistics - Like Commits, reads, block reads, tupples (rows) fetched, inserted, deadlocks, etc - For more detail refer to PostgreSQL 9.4 doc   8. GetPSQLLogDesitnation - Checks where the PostgreSQL server logs are directed to - I.e. stderr, csvlog or syslog - Default is stderr   9. GetPSQLLogFileName - Fetches the log PostgreSQL log file name and the filename format - E.g. postgresql-%Y-%m-%d_%H%M%S.log    10. GetPSQLLoggingLocation - Fetches the location where the logs are stored for PostgreSQL - e.g. pg_log, which is also the default location - Desired location for the logs can be done in the postgresql.conf file   11. GetPSQLRelationIndexes - Gets information on the Indexes - Information like index size, number of rows, table names on which the index is created   12. GetPSQLReplicationStat - Shows information related to the Replication on PostgreSQL DB - Applicable to the PostgreSQL DBs where replication is enabled   13. GetPSQLTablespaceInfo - Takes tablespace name as input (String DataType), service defaults to 'thingworx' - modify if needed - Fetches information like owner oid, tablespace ACL     14. GetPSQLUserIndexIO - Fetches index that are created only on the User created DB objects - Shows relations (table) vs index relations ids (index on table), together with their names - Also shows additional info like number of disk blocks read from this index & number of buffer hits in this index     15. GetPSQLUserSequencesIOStats - Fetches informtion on Sequence objects used on user defined relations (tables) - Number of disk blocks read from this sequence & buffer hits in this sequence     16. GetPSQLUserTableIOStat - Fetches disk I/O information on the user created tables     17. GetPSQLUserTables - Fetches all the user created tables, together with their name, OID Disk I/O Last auto vacuum , vacuum Also lists the amount of rows each relation (table) has   Finally The attached entity has some additional service not yet covered in this blog, as they are minor services. Therefore for brevity of this blog I've left them out for now, feel free to explore or enhance this. I will continue to look for any additional services and will enhance this document and the entities belong to this.    If you are looking to enhance this feel free to fork from twxPostgreSqlHealthCheck over Github.

Thing Subscription This post is intended for novice ThingWorx users who wants to understand what the definition of Thing Subscription is and the overall purpose of using Thing Subscriptions.   Definition of a Thing Subscription? A Thing subscription is a script(JavaScript) that is called each time an event occurs. Events are property states which are of end users interest (e.g. temperature) and therefore indicators to kick off some functionality in a Thing subscription when any action needed. Events can e.g. be triggered by an Alert that detects a change or an anomaly in property values. The Thing subscription is explicitly linked to an event and when the event is fired the data is being passed to the subscriber.    Why Use a Thing Subscription? Imagine your machine is running 24 hours 7 days a week with supervised human interaction. If a pump temperature exceeds accepted value it needs to be regulated by the manufacturing department. But no one in the department knows when the temperature will exceed accepted value or drop suddenly therefore, the machines is always sporadically physically supervised by humans which leads to heavy costs for the manufacture. With a Thing Subscription a notification alert email can be sent directly to the department manager who acts based on the email notification.   Thing Subscription must have A Thing subscription must have defined a rule which gets executed when an event occurs. The definition of the rule may accommodate any appropriate business logic.   Thing Subscription example process In this scenario Thing subscription is using a predictive analytics model to detect Data Change or any anomaly values going through a Thing Property. So, based on historical data including failure information, a predictive analytics model begins to analyze run-time values from individual Things/properties to the analytics server. The predictive analytics model detects a pattern which detects past failures, when the analytics model predicts a failure/event based on the analyzed patterns an action is being fired via a Thing subscription. That action could be for ThingWorx to create a service ticket or send a notification email to the service department.   Example of a simple Thing Subscription set-up without using Analytics model to analyze data but instead a build-in ThingWorx alert Below example of Thing Subscription will send a notification email when temperature exceeds defined values from ThingWorx alert configuration. Prerequisites; it is necessary to have a mail server extension imported into the ThingWorx Composer this enables the service department to receive the email notification when an event have occurred. The extension can be downloaded from the marketplace. 1. Create a Thing with the MailServer[i] as the Base Thing template.     2. Create a new Thing and add Properties together with an alert that is triggered when the value exeeds user defined temerature.   3. Enable the Thing Subcriptions by Select Subscription and click +Add Make sure to mark the checkbox Enabled Selecting your Event name and your Property name In the right side of the screen you can enter your script/function that will notify ThingWorx email service to create the email notification Select Done and Save   4. Enable Email notification by selecting Services Provide an name Select Me/Entities Mark Other entity Find your Thing where the MailServer is the Thing Template   5. Then find the SendMessage snippet/script and fill out the snippet with your personal information.   [i] View this blog for more information on how to install the MailServer

Large files could cause slow response times. In some cases large queries might cause extensively large response files, e.g. calling a ThingWorx service that returns an extensively large result set as JSON file.   Those massive files have to be transferred over the network and require additional bandwidth - for each and every call. The more bandwidth is used, the more time is taken on the network, the more the impact on performance could be. Imagine transferring tens or hundreds of MB for service calls for each and every call - over and over again.   To reduce the bandwidth compression can be activated. Instead of transferring MBs per service call, the server only has to transfer a couple of KB per call (best case scenario). This needs to be configured on Tomcat level. There is some information availabe in the offical Tomcat documation at https://tomcat.apache.org/tomcat-8.5-doc/config/http.html Search for the "compression" attribute.   Gzip compression   Usually Tomcat is compressing content in gzip. To verify if a certain response is in fact compressed or not, the Development Tools or Fiddler can be used. The Response Headers usually mention the compression type if the content is compressed:     Left: no compression Right: compression on Tomcat level   Not so straight forward - network vs. compression time trade-off   There's however a pitfall with compression on Tomcat side. Each response will add additional strain on time and resources (like CPU) to compress on the server and decompress the content on the client. Especially for small files this might be an unnecessary overhead as the time and resources to compress might take longer than just transferring a couple of uncompressed KB.   In the end it's a trade-off between network speed and the speed of compressing, decompressing response files on server and client. With the compressionMinSize attribute a compromise size can be set to find the best balance between compression and bandwith.   This trade-off can be clearly seen (for small content) here:     While the Size of the content shrinks, the Time increases. For larger content files however the Time will slightly increase as well due to the compression overhead, whereas the Size can be potentially dropped by a massive factor - especially for text based files.   Above test has been performed on a local virtual machine which basically neglegts most of the network related traffic problems resulting in performance issues - therefore the overhead in Time are a couple of milliseconds for the compression / decompression.   The default for the compressionMinSize is 2048 byte.   High potential performance improvement   Looking at the Combined.js the content size can be reduced significantly from 4.3 MB to only 886 KB. For my simple Mashup showing a chart with Temperature and Humidity this also decreases total load time from 32 to 2 seconds - also decreasing the content size from 6.1 MB to 1.2 MB!     This decreases load time and size by a factor of 16x and 5x - the total time until finished rendering the page has been decreased by a factor of almost 22x! (for this particular use case)   Configuration   To configure compression, open Tomcat's server.xml   In the <Connector> definitions add the following:   compression="on" compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json"     This will use the default compressionMinSize of 2048 bytes. In addition to the default Mime Types I've also added application/json to compress ThingWorx service call results.   This needs to be configured for all Connectors that users should access - e.g. for HTTP and HTTPS connectors. For testing purposes I have a HTTPS connector with compression while HTTP is running without it.   Conclusion   If possible, enable compression to speed up content download for the client.   However there are some scenarios where compression is actually not a good idea - e.g. when using a WAN Accelerator or other network components that usually bring their own content compression. This not only adds unnecessary overhead but is compressing twice which might lead to errors on client side when decompressing the content.   Especially dealing with large responses can help decreasing impact on performance. As compressing and decompressing adds some overhead, the min size limit can be experimented with to find the optimal compromise between a network and compression time trade-off.

The Squeal functionality has been discontinued with ThingWorx 8.1, see ThingWorx 8.1.0 Release Notes   There might be scenarios where it should be disabled in earlier versions as well. This can be achieved e.g. with Tomcat Security Constraints. To add such a constraint, open <Tomcat>\webapps\Thingworx\WEB-INF\web.xml At the end of the file add a new constraint just before closing the </web-app> tag:   <security-constraint> <web-resource-collection> <web-resource-name>Forbidden</web-resource-name> <url-pattern>/Squeal/*</url-pattern> </web-resource-collection> <auth-constraint/> <user-data-constraint> <transport-guarantee>CONFIDENTIAL</transport-guarantee> </user-data-constraint> </security-constraint> Save the file and restart Tomcat. Accessing the /Thingworx/Squeal resource now will result in an error message:   HTTP Status 403 - Access to the requested resource has been denied   One scenario to be aware of is when the web.xml changes, e.g. due to updating ThingWorx or other manual changes. In such a case, ensure that the filter is still present in the file and taken into account.

This Best Practices document should offer some guidelines and tips & tricks on how to work with Timers and Schedulers in ThingWorx. After exploring the configuration and creation of Timers and Schedulers via the UI or JavaScript Services, this document will also highlight some of the most common performance issues and troubleshooting techniques.   Timers and Schedulers can be used to run jobs or fire events on a regular basis. Both are implemented as Thing Templates in ThingWorx. New Timer and Scheduler Things can be created based on these Templates to introduce time based actions. Timers can be used to fire events in a certain interval, defined in the Timer's Update Rate (default is 60000 milliseconds = 1 minute). Schedulers can be used to run jobs based on a cron pattern (such as once a day or once an hour). Schedulers will also allow for a more detailed time based setup, e.g. based on seconds, hours, days of week or days months etc. Events fired by both Timers and Schedulers can be subscribed to with Subscriptions which can be utilized to execute custom service scripts, e.g. to generate "fake" or random demo data to update Remote Things in a test environment. In general subscriptions and scripts can be used to e.g. run regular maintenance tasks or periodically required functions (e.g. for data aggregation) For more information about setting up Timers and Schedulers it's recommended to also have a look at the following content:   How to set up and configure Timers How to set up and configure Schedulers How to create and configure Timers and Schedulers via JavaScript Services Events and Subscriptions for Timers and Schedulers   Example   The following example will illustrate on how to create a Timer Thing updating a Remote Thing using random values. To avoid any conflicts with permissions and visibility, use the Administrator user to create Things.   Remote Thing   Create a new Thing based on the Remote Thing Template, called myRemoteThing. Add two properties, numberA and numberB - both Integers and marked as persistent. Save myRemoteThing. Timer Thing   Create a new Thing based on the Timer Template, called myTimerThing. In the Configuration, change the Update Rate to 5000, to fire the Event every 5 seconds. User Context to Administrator. This will run the related services with the Administrator's user visibility and permissions. Save myTimerThing. Subscriptions   To update the myRemoteThing properties when the Timer Event fires, there are two options: Configure a Subscription on myRemoteThing and listen to Timer Events on the myTimerThing. Configure a Subscription on myTimerThing and listen to Timer Events on itself as a source. In this example, let's go with the first option and Edit myRemoteThing. Create a new Subscription pointing to myTimerThing as a Source. Select the Timer Event Note that if no source is selected, the Timer Event is not availabe, as myRemoteThing is based on the Remote Thing Template and not the Timer Template Enable the Subscription. In the Script area use the following code to assign two random numbers to the Thing's custom properties: me.numberA = Math.floor(Math.random() * 100); me.numberB = Math.floor(Math.random() * 100); Save myRemoteThing. Validation   The Subscription will be enabled and active on saving it. Switch to the myRemoteThing Properties Refreshing the Values will show updates with random numbers between 0 and 99 every 5 seconds (Timer Update Rate).   Performance considerations   Timers and Schedulers are handled via the Event Processing Subsystem. Metrics that impact current performance can be seen in Monitoring > Subsystems > Event Processing Implementing Timers and Schedulers on a Thing Template level might flood the system with services executions originating from Subscriptions to Timer / Scheduler triggered Events. Subscribing to another Thing's Events will be handled via the Event Processing Subsystem. Subscribing to an Event on the same Thing will not be handled via the Event Processing Subsystem, but rather execute on the already open in memory Thing. If Timers and Schedulers are not necessarily needed, the Services can be triggered e.g. via Data Change Events, UI Interactions etc. Recursion can be a hidden performance contributer where a Subscription to a certain Event executes a service, triggering another Event with recursive dependencies. Ensure there are no circular dependencies and service calls across Entities. If possible, reads for each and every action from disk should be avoided. Performance can be increased by storing relevant information in memory and using Streams or Datatables or for persistence. If possible, call other Services from within the Subscription instead of handling all code within the Subscription itself. For full details, see also Timers and Schedulers - Best Practice   How to identify and troubleshoot technical issues   Check the Event Processing Subsystem for any spikes in queued Events (tasks submitted) while the total number of tasks completed is not or only slowly increasing. For a historical overview, search the ApplicationLog for "Thingworx System Metrics" to get system metrics since the server has been (re-) started. In the ApplicationLog the message "Subsystem EventProcessingSubsystem is started" indicates that the Subsystem is indeed started and available. Use custom loggers in Services to get more context around errors and execution in the ScriptLog Custom Loggers can be used to identify if Events have fired and Subscriptions are actually triggered Example: logger.debug("myThing: executing subscribed service") For issues with Service execution, see also CS268218 Infinite loops in Services could render the server unresponsive and might flood the system with various Events To change the timing for a Timer, restarting the Thing is not enough. The Timer must be disabled and enabled at the desired start time. Schedulers will allow for a much more flexible timing and setting / changing execution times in advance. For further analysis it's recommended to generate Thread Dumps to get more information about the current state of Threads in the JVM. The ThingWorx Support Tools Extension can help in generating those. See also CS245547 for more information and usage.

Events   Timers and Schedulers both come with a specific Event inherited from the Thing Template: Timer ScheduledEvent Both have a Data Shape allowing to capture the timestamp of when the Event was actually fired. Events in ThingWorx are triggered when a specific condition is met. In this context the condition is met and the Event is fired when a Timer has expired or a Scheduler's time is reached. Once an Event is triggered, Subscriptions will take care of executing custom Services to react to the Event. Subscriptions   Subscriptions listen to Events and can be used to react to certain Events with running custom Service scripts. To follow-up on Timers and Schedulers, a new Subscription must be created, listening to any related Event fired. Add a new Subscription to the Thing with       As the Subscription is usually listening to the Thing that it is configured on, the Source has to be left empty. When listening to other Entities' Subscriptions the corresponding Entity can be picked in the Source Entity picker. Ensure to check the Enabled checkbox to actually enable the Subscription and allow it for executing code in the Script area. The following Script will log into the ScriptLog once the Timer Event is fired     The following Script will log into the ScriptLog once the ScheduledEvent Event is fired  

Timers and Schedulers can also be created and configured programmatically via custom services. The following service, which can be created on any Thing, will create a new Timer using the following Inputs:         // create new Thing var params = { name: ThingName /* STRING */, description: undefined /* STRING */, thingTemplateName: "Timer" /* THINGTEMPLATENAME */, tags: undefined /* TAGS */ }; Resources["EntityServices"].CreateThing(params); // read initial configuration // result: INFOTABLE var configtable = Things[ThingName].GetConfigurationTable({tableName: "Settings"}); // update configuration with service parameters configtable.updateRate = updateRate configtable.runAsUser = user // set new configuration table var params = { configurationTable: configtable /* INFOTABLE */, persistent: true /* BOOLEAN */, tableName: "Settings" /* STRING */ }; Things[ThingName].SetConfigurationTable(params);   This code is an example which could also be used to create a new Scheduler. The configuration table for a Timer has the following attributes: updateRate enabled runAsUser The configuration table for a Scheduler has the following attributes: schedule enabled runAsUser  

Create a new Thing using the Scheduler Thing Template. The Scheduler Thing will fire a ScheduledEvent Event when the configured schedule is fired. The event is automatically present and does not need to be added manually. Configuration   The Scheduler Configuration is quite straightforward and allows for an exact setup of schedule based on units of time, e.g. seconds, minutes, hours, days of week etc. It can be accessed via the Thing's Entity Configuration   Configuration allows for Changing the runAsUser context - in which the Events will be handled. The user will need visibility and permission on e.g. executing Services or depending Things, which are required to run the Service triggered by the Event. Changing the Schedule - in which time the Events will be fired (by default every minute). The schedule is displayed in CRON String notation and can be changed and viewed in detail by clicking on "More". The CRON String will be generated automatically based on the inputs. Schedules can be configured in Manual mode - allowing for full configuration of each and every time based attribute. Schedules can be configured for a specific time Type - allowing for configuration only based on seconds, minutes, hours, days, weeks, months or years. Below screenshots show schedules running every minute and every Saturday / Sunday at 12:00 ("Every Weekend Day").     Services   Scheduler Things inherit two Services by default from the Thing Template DisableScheduler EnableScheduler These will activate / de-activate the Scheduler and allow / disallow firing Events once a scheduled time is reached If a Scheduler is currenty enabled or disabled can be seen in its properites  

Create a new Thing using the Timer Thing Template. The Timer Thing will fire a Timer Event when the Timer's Update Rate has expired. The event is automatically present and does not need to be added manually. Configuration   The Timer Configuration is quite straightforward. It can be accessed via the Thing's Entity Configuration. Configuration allows for Enabling the Timer on Thing-Startup - whenever the Thing is started, e.g. when restarting ThingWorx or via the RestartThing Generic Service, also the Timer is enabled and will fire Events. Changing the Update Rate - in which intervall the Events will be fired (by default every minute [60000 milliseconds]). Changing the User Context - in which the Events will be handled. The user will need visibility and permission on e.g. executing Services or depending Things, which are required to run the Service triggered by the Event.           Services   Timer Things inherit two Services by default from the Thing Template DisableTimer EnableTimer These will activate / de-activate the Timer and allow / disallow firing Events once the Update Rate has expired If a Timer is currently enabled or disabled can be seen in its properties  

Video Author:                     Asia Garrouj Original Post Date:            March 31, 2017 Applicable Releases:        ThingWorx Analytics 7.4 to 8.1   Description: This video is the second part of a two part video series walking thru the configuration of Analysis Event which is applied for Real-Time Scoring.  This second video will walk you thru the configuration of Analysis Event for Real Time Scoring and validating that a predictions job has been executed based on new input data.    

Video Author:                     Durgesh Patel Original Post Date:            June 12, 2017 Applicable Releases:        ThingWorx   Description: In this video we review the different features available in Grid Advanced Extension version 2.0.  

Original Post Date:            September 30, 2016   Description: This tutorial video will walk you through the installation process for the PostgreSQL-based version of the ThingWorx Platform (7.2) in a RHEL environment.  All required software components will be covered in this video.    

Video Author:                     Polina Osipvoa Original Post Date:            June 10, 2016 Applicable Releases:        ThingWorx   Description: This is a video tutorial on creating a Media Entity, and importing and displaying an image.      

Video Author:                     Polina Osipova Original Post Date:            June 10, 2016   Description: This is a video tutorial on configuring properties for a Thing, and using "Manage Bindings" to bind properties to a Thing.      

Video Author:                     Polina Osipova Original Post Date:            June 10, 2016   Description: This is a video tutorial on adding State Formatting in a Mashup using State Definitions.      

Original Post Date:     June 6, 2016 Description: This is a video tutorial on creating a Value Stream, adding the Value Stream to a Remote Thing, adding, binding and querying the (remote) properties.      

Original Post Date:     June 6, 2016 Description: This tutorial video will walk you through the installation process for the PostgreSQL-based version of the ThingWorx Platform in a Windows environment.  All required software components will be covered in this video.    

Original Post Date:            June 6, 2016   Description: This is a video tutorial on creating a Stream, adding a Data Shape with properties, and writing values to the Stream.    

Original Post Date:      June 6, 2016   Description: This is a video tutorial on creating an InfoTable through a service, creating and adding a Data Shape adding rows to the Infotable through a service, adding rows to an InfoTable property and querying the InfoTable.