IoT Tips

ThingWorx DevOps By Victoria Firewind,  IoT EDC This presentation accompanies a recent Expert Session, with video content including demos of the following topics:   found here!   DevOps is a process for taking planned changes through development, through testing, and into production,   where they can be accessed by end users.   One test instance typically has automated tests (integration testing) which ensure application logic is preserved in spite of whatever changes the developers are making, and often there is another test instance to ensure the application is usable (UAT testing) and able   to handle a production load (load testing).   So, a DevOps Pipeline starts with a task manager tasking out planned changes, where each task will become a branch in the repository. Each time a new branch is created, a new pipe is needed, which in this case, is produced by Docker Hub.   Developers then make changes within that pipe, which then flow along the pipe into testing. In this diagram, testing is shown as the valve which when open (i.e. when tests all pass) then   allows the changes to flow along the pipe into production.   A good DevOps process has good flow along the various pipelines, with as much automated or scripted as possible to reduce the chances for errors in deployments.   In order to create a seamless pipeline, whether or not it winds up automated, several third party tools are useful:       b                  Container software is a very good way to improve the maintainability and updatability of a ThingWorx instance, while minimizing the amount of resources needed to host each component.   n  1. Create Docker Image Consult the Help Center if need be. Update your YML file with everything you need before starting the image: see the example in the PTC community.   License the instance using the license management website. Follow the instructions from Docker for installing those tools: Docker itself (docker) and Docker Compose (docker-compose).   n   2. Save Docker Image in Docker Repo Docker Hub has some free options, and if a license is purchased,   can host more than a single Docker image and tag. It is also possible to set up your own Docker registry.           n 3. Access the image in Docker Desktop Download Docker Desktop and sign-in to the Docker account which hosts the repository.   Create some folders for storing the h2.env file and the ThingworxStorage and ThingWorxPlatform mounted folders.   Remember to license these containers as well. Developers login to the license management site themselves and put those into the ThingWorxPlatform mounted folder (“license_capability_response.bin”).         Git is a very versatile tool that can be used through many different mediums, like Azure DevOps or Github Desktop.  To get started as a totally new Git user, try downloading Github Desktop on your local machine and create a local repository with the provided sample code.    This can then be cloned on a Linux machine, presumably whichever instance hosts the integration ThingWorx instance, using the provided scripts (once they are configured).  Remember to install Git on the Linux machine, if necessary (sudo apt-get install git).     A sample ThingWorx application (which is not officially supported, and provided just as an example on how to do DevOps related tasks in ThingWorx) is attached to this post in a zip file, containing two directories, one for scripts and one for ThingWorx entities.   Copy the Git scripts and config file into the top level, above   the repository folder, and update the GitConfig.sh file with the URL for your Git repository and your login credentials. Then these scripts can be used to sync your Linux server with your Git repository, which any developer could easily update from their local machine. This also ensures changes are secure, and enables the potential use of other DevOps procedures like tasks, epics, and corresponding branches of code.     Steps to DevOps using the provided code as an example: Clone the repository into the SystemRepository or any other created repository, use the provided scripts in a Linux environment. Import the DeploymentUtilities entity, which again is scripted for Linux or for use with a development IDE with bash support. Then import the ThingWorx application from source control or use the script (which itself makes use of that DeploymentUtilities entity). Now create some local changes, add things, etc. and try out the UpdateApplication script or export to source control and then push to the Git repo. Data and localization table exports are also possible. Run the tests using the provided IntegrationTester thing or create your own by overriding the IntegrationTestTS thing shape, or use the TestTwxApplication script from a Linux terminal. Design a process for your application which  allows for easy application exports and updates to and from a repository, so that developers can easily send in their changes, which can then be easily loaded and tested in another environment.   In Conclusion: DevOps is a complex topic and every PTC customer will have their own process based around their unique requirements and applications. In the future, more mature pipeline solutions will be covered, ones that involve also publishing to Solution Central for easier deployment between various testing instances and production.        

The following videos are provided to help users get started with ThingWorx: ThingWorx Installation Installing ThingWorx (Neo4j) in Windows ThingWorx PostgreSQL Setup for Windows ThingWorx PostgreSQL for RHEL ThingWorx Data Storage Introduction to Streams Introduction to Value Streams Introduction to DataTables Introduction to InfoTables ThingWorx Concepts & Functionality Introduction to Media Entities Using State Formatting in a Mashup Configuring Properties ThingWorx REST API REST API (Part 1) REST API (Part 2) ThingWorx Edge SDK Configuring File Transfer with the .NET SDK ThingWorx Analytics *new* Getting Started with ThingWorx Analytics Part 1 Getting Started with ThingWorx Analytics Part 2 Installing ThingWorx Analytics Builder Part 1 of 3 Installing ThingWorx Analytics Builder Part 2 of 3 Installing ThingWorx Analytics Builder Part 3 of 3 Creating Signals in the Analytics Builder How to Access the ThingWorx Analytics Interactive API Guide ThingWorx Widgets How to Create and Configure the Auto Refresh Widget How to Create and Define a Blog Widget How to Create and Configure a Button Widget How to Use the Divider and Shape Widgets How to Create and Configure a Chart Widget How to Use a Contained Mashup How to Use the Data Filter Widget How to Use an Expression Widget How to Create and Configure a Gauge Widget How to Create and Configure a Checkbox Widget How to Use a Contained Mashup Widget How to Use a Data Export Widget How to Use the DateTime Picker Widget How to Use the Editable Grid Widget Using Fieldset and Panel Widgets How to Use the File Upload Widget How to Use the Folding Panel Widget How to Use the Google Location Picker How to Use the Google Map Widget How to Use a Grid Widget How to Use an HTML TextArea Widget How to Use the List Widget How to Use a Label Widget How to Use the Layout Widget How to Use the LED Display Widget How to Use the List Widget How to Use the Masked Textbox Widget Navigation in ThingWorx: Using Menus, the Navigation Widget, Link Widget, and Contained Mashups How to Use the Numeric Entry Widget How to Use the Pie Chart Widget How to Use the Property Display Widget How to Use the Radio Button Widget How to Use the Repeater Widget How to Use the Slider Widget How to Use the SQUEAL Search Widget How to Use the Responsive Tab Widget How to Use the Tag Cloud Widget How to Use the Tag Picker Widget How to Use the TextArea and TextBox Widgets How to Use the Time Selector Widget How to Use the Tree Widget How to Use the Value Display Widget How to Use the Web Frame Widget How to Create and Define a Wiki How to Use the XY Chart Quick note: Thread will be updated with more videos as they are added.

Timers and schedulers can be useful tool in a Thingworx application.  Their only purpose, of course, is to create events that can be used by the platform to perform and number of tasks.  These can range from, requesting data from an edge device, to doing calculations for alerts, to running archive functions for data.  Sounds like a simple enough process.  Then why do most platform performance issues seem to come from these two simple templates? It all has to do with how the event is subscribed to and how the platform needs to process events and subscriptions.  The tasks of handling MOST events and their related subscription logic is in the EventProcessingSubsystem.  You can see the metrics of this via the Monitoring -> Subsystems menu in Composer.  This will show you how many events have been processed and how many events are waiting in queue to be processed, along with some other settings.  You can often identify issues with Timers and Schedulers here, you will see the number of queued events climb and the number of processed events stagnate. But why!?  Shouldn't this multi-threaded processing take care of all of that.  Most times it can easily do this but when you suddenly flood it with transaction all trying to access the same resources and the same time it can grind to a halt. This typically occurs when you create a timer/scheduler and subscribe to it's event at a template level.  To illustrate this lets look at an example of what might occur.  In this scenario let's imagine we have 1,000 edge devices that we must pull data from.  We only need to get this information every 5 minutes.  When we retrieve it we must lookup some data mapping from a DataTable and store the data in a Stream.  At the 5 minute interval the timer fires it's event.  Suddenly all at once the EventProcessingSubsystem get 1000 events.  This by itself is not a problem, but it will concurrently try to process as many as it can to be efficient.  So we now have multiple transactions all trying to query a single DataTable all at once.  In order to read this table the database (no matter which back end persistence provider) will lock parts or all of the table (depending on the query).  As you can probably guess things begin to slow down because each transaction has the lock while many others are trying to acquire one.  This happens over and over until all 1,000 transactions are complete.  In the mean time we are also doing other commands in the subscription and writing Stream entries to the same database inside the same transactions.  Additionally remember all of these transactions and data they access must be held in memory while they are running.  You also will see a memory spike and depending on resource can run into a problem here as well. Regular events can easily be part of any use case, so how would that work!  The trick to know here comes in two parts.  First, any event a Thing raises can be subscribed to on that same Thing.  When you do this the subscription transaction does not go into the EventProcessingSubsystem.  It will execute on the threads already open in memory for that Thing.  So subscribing to a timer event on the Timer Thing that raised the event will not flood the subsystem. In the previous example, how would you go about polling all of these Things.  Simple, you take the exact logic you would have executed on the template subscription and move it to the timer subscription.  To keep the context of the Thing, use the GetImplimentingThings service for the template to retrieve the list of all 1,000 Things created based on it.  Then loop through these things and execute the logic.  This also means that all of the DataTable queries and logic will be executed sequentially so the database locking issue goes away as well.  Memory issues decrease also because the allocated memory for the quries is either reused or can be clean during garbage collection since the use of the variable that held the result is reallocated on each loop. Overall it is best not to use Timers and Schedulers whenever possible.  Use data triggered events, UI interactions or Rest API calls to initiate transactions whenever possible.  It lowers the overall risk of flooding the system with recourse demands, from processor, to memory, to threads, to database.  Sometimes, though, they are needed.  Follow the basic guides in logic here and things should run smoothly!

This is going to cover one way of configuring an SSL passthrough using HAProxy.  This guide is intended to be a reference document, and administrators looking to configure an SSL passthrough should make sure the end solution meets both their company's business and security needs.   Why use SSL Passthrough instead of SSL Termination? The main reason for ThingWorx would be if a company requires encrypted communication internally, as well as externally.  With SSL Termination, the request between the load balancer and the client is encrypted.  But the load balancer takes on the role to decrypt and passes that back to the server.  With SSL Passthrough, the request goes through the load balancer as is, and the decryption happens on the ThingWorx Application server.   What you will need to continue with this guide:   HAProxy installed A working ThingWorx application server (Guide to getting one setup can be found here) Tomcat configured for ssl NOTE : Always contact your Security team and make sure you have a certificate that meets your business policy For this tutorial, I created a self-signed certificate following along with the below guide.  If you have already obtained a valid certificate, then you can just skip over the step of creating it, and follow along with the Tomcat portion https://www.ptc.com/en/support/article?n=CS193947 Once configured, restart Tomcat and verify it is working by navigating to https://<yourServer>:<port>/Thingworx   With ThingWorx running as SSL and HAProxy installed, we just need to make sure the HAProxy configuration is setup to allow SSL traffic through.  We use 'mode tcp' to accomplish this.   On your HAProxy machine, open /etc/haproxy/haproxy.cfg for editing.  While most of this can be customized to fit your business needs, some variation of the highlighted portions below need to be included in your final configuration:   global         log /dev/log    local0         log /dev/log    local1 notice         chroot /var/lib/haproxy         stats socket /run/haproxy/admin.sock mode 660 level admin         stats timeout 30s         user haproxy         group haproxy         daemon           # Default SSL material locations         ca-base /etc/ssl/certs         crt-base /etc/ssl/private           # Default ciphers to use on SSL-enabled listening sockets.         # For more information, see ciphers(1SSL). This list is from:         #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/         # An alternative list with additional directives can be obtained from         #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy         ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS         ssl-default-bind-options no-sslv3   defaults         log global          option tcplog          mode tcp          option http-server-close          timeout connect 1s          timeout client  20s          timeout server  20s          timeout client-fin 20s          timeout tunnel 1h          errorfile 400 /etc/haproxy/errors/400.http          errorfile 403 /etc/haproxy/errors/403.http          errorfile 408 /etc/haproxy/errors/408.http          errorfile 500 /etc/haproxy/errors/500.http          errorfile 502 /etc/haproxy/errors/502.http          errorfile 503 /etc/haproxy/errors/503.http          errorfile 504 /etc/haproxy/errors/504.http         frontend https          bind *:443          mode tcp          default_backend bk_app           backend bk_app          mode tcp          server TWXAPP01  <twxapp01IP>:<port>         In this example, the user would connect to https://<loadbalancer>/Thingworx and the load balancer would forward the requests to https://<twxapp01IP>:<port>/Thingworx   That’s it!   A couple of side notes:   The load balancer port the clients connect to does not need to be the same as the ThingWorx port the load balancer will forward to If working in a Highly Available configuration, each ThingWorx Application server needs to have its own certificate configured If HAProxy seems unstable, try updating to the latest release If it is on the latest release according to the Unix repository, check https://www.haproxy.org/ and see if there is a later stable release.  There have been some issues where Ubuntu's latest update in the repository is actually a few years old

Error: Failed to load SQL Modules into database Cluster. This error is usually seen during initializing the database cluster phase (in the setup as shown below). To resolve this issue, follow below steps: Create a PostgreSQL data folder before you start the installation (c:\postgres-data) and give full control for the user. Select the newly created data directory during the setup. After the successful installation, you can follow the remaining procedure for configuring it with ThingWorx from the respective installation document.

Welcome to the ThingWorx Analytics Training Course! Through these 11 modules, you will learn all about the functionality of this software, as well as techniques to help you build a successful and meaningful predictive analytics application.

Overview REST stands for representational state transfer and is a software architectural style common in the World Wide Web. Anything with a RESTful interface can be communicated with using standard REST syntax. ThingWorx has such an interface built-in to make viewing and updating Thing properties as well as executing services easy to do independently of the Web UI.   How to Use REST API The ThingWorx REST API is entirely accessible via URL using the following syntax:    (Precision LMS. Getting Started With ThingWorx 5.4 (Part 1 of Introduction to ThingWorx 5.4). PTC University. https://precisionlms.ptc.com/viewer/course/en/21332822/page/21332905.)   The above example shows how to access a service called “GetBlogEntriesWithComments” found on the “ThingWorxTrainingMaintenanceBlog” Thing. Notice that even though this service gets XML formatted data, the method is type “POST” and “GET” will not work in this scenario (Further reading: https://support.ptc.com/appserver/cs/view/solution.jsp?n=CS214689&lang=en_US).   In order to be able to run REST API calls from the browser, one must allow request method switching. This can be enabled by checking the box “Allow Request Method Switch” in PlatformSubsystem (Further reading: https://support.ptc.com/appserver/cs/view/solution.jsp?n=CS224211&lang=en_US).   Access REST API from Postman Postman is a commonly used REST client which can ping servers via REST API in a manner which mimics third party software. It is free and easy-to-use, with a full tutorial located here: https://www.getpostman.com/docs/   In order to make a request, populate the URL field with a properly formatted REST API call (see previous section). Parameters will not automatically be URL-encoded, but right-clicking on a highlighted portion of the URL and selecting EncodeURIComponent encodes the section.   Next click the headers tab. Here is where the content-type, accept, and authorization are set for the REST call. Accept refers to which response format the REST call is expecting while content-type refers to the format of the request being sent to the server. Authhorization is required for accessing ThingWorx, even via REST API (see previous section for examples authenticating using an app key, but in Postman you can also use Basic Auth using a username and password)   In Postman, there is also ample opportunity to modify the request body under the Body tab. There are several options here for setting parameters. Form-data and x-www-form-urlencoded both allow for setting key value pairs easily and cleanly, and in the latter case, encoding occurs automatically (e.g. “Hello World” becomes %22Hello%20World%22). Raw request types can contain anything and Postman will not touch anything entered except to replace environment variables. Whatever is placed in the text area under raw will get sent with the request (normally XML or JSON, as specified by content-type). Finally, binary allows for sending things which cannot normally be entered into Postman, e.g. image, text, or audio files.     REST API Examples For introductory level examples, see the previous Blog document found here: https://community.thingworx.com/docs/DOC-3315   Retrieving property values from “MyThing” using GET, the default method type (notice how no “method=GET” is required here, though it would still work with that as well): http://localhost/Thingworx/Things/MyThing/Properties/   Updating “MyProperty “with the value “hello” on “MyThing” using PUT: http://localhost/Thingworx/Things/MyThing/Properties/MyProperty?method=PUT&value=hello In Postman, you can send multiple property updates at once via query body (in this case updating all of the properties, the string “Prop1” and the number “Prop2” on MyThing) § Query: http://localhost/Thingworx/Things/MyThing/Properties/* § Query Type: PUT § Query Headers: Content-Type: application/json Authorization: Basic Auth (input username and password on Authorization tab and this will auto-populate) § Body JSON: {"Prop1":"hello world","Prop2":10} Note: you can also specify multiple properties as shown, but only update one at a time in Postman by utilizing the browser syntax given above   Calling “MyService” (a service on “TestThing)” with a String input parameter (“InputString”): http://localhost/Thingworx/Things/TestThing/Services/MyService?method=post&InputString=input   It is easier to pass things like XML and JSON into services using Postman. This query calls “MyJSONService” on “MyThing” with a JSON input parameter § Query: http://localhost/Thingworx/Things/MyThing/Services/MyJSONService § Query Type: § Queries Headers: Accept should match service output (text/html for String) Content-Type: application/json or Authorization: Basic Auth (input username and password on Authorization tab and this will auto-populate) Body JSON: {"InputJSON":"{\"JSONInput\":{\"PropertyName\":\"TestingProp\",\"PropertyValue\":\"Test\"}}"} Body XML:{"xmlInput": "<xml><name>User1</name></xml"}   Viewing “BasicMashup” using AppKey authentication (so no login is required because this Application Key is set-up to login as a user who has permissions to view the Mashup): http://localhost/Thingworx/Mashups/BasicMashup?appKey=b101903d-af6f-43ae-9ad8-0e8c604141af&x-thingworx-session=true Read more here: https://support.ptc.com/appserver/cs/view/solution.jsp?n=CS227935   Downloading Log Information from “ApplicationLog” (or other log types): http://localhost/Thingworx/Logs/ApplicationLog/Services/QueryLogEntries?method=POST   In Postman, more information can be passed into some queries via query body § Query: http://localhost/Thingworx/Logs/ApplicationLog/Services/QueryLogEntries Query Type: POST Query Headers: Accept: application/octet-stream or Content-Type: application/json Authorization: Basic Auth (input username and password on Authorization tab and this will auto-populate) Body: {\"searchExpression\":\"\",\"origin\":\"\",\"instance\":\"\",\"thread\":\"\", \"startDate\":1462457344702,\"endDate\":1462543744702,\"maxItems\":100}   Downloading “MyFile.txt” from “MyRepo” FileRepository (here, “/” refers to the home folder of this FileRepository and the full path would be something like “C:\ThingworxStorage\repository\MyRepo\MyFolder\MyFile.txt”): http://localhost/Thingworx/FileRepositoryDownloader?download-repository=MyRepo&download-path=/MyFolder/MyFile.txt   Uploading files to FileRepository type Things is a bit tricky as anything uploaded must be Base64 encoded prior to making the service call. In Postman, this is the configuration to used to send a file called “HelloWorld.txt”, containing the string “Hello World!”, to a folder called “FolderInRepo” on a FileRepository named “MyRepo”:   Query: http://localhost/Thingworx/Things/MyRepo/Services/SaveBinary Query Type: POST Query Headers: Accept: application/json Content-Type: application/json Authorization: Basic Auth (input username and password on Authorization tab and this will auto-populate) Body: {"path" : "/FolderInRepo/HelloWorld.txt", "content" : "SGVsbG8gV29ybGQh"} Notice here that the content has been encoded to Base64 using a free online service. In most cases, this step can be handled by programming language code more easily and for more challenging file content   Resources and other built-in Things can be accessed in similar fashion to user-created Things. This query searches for Things with the “GenericThing” ThingTemplate implemented: http://localhost/Thingworx/Resources/SearchFunctions/Services/SearchThingsByTemplate?method=POST&thingTemplate=GenericThing   Deleting “MyThing” (try using services for this instead when possible since they are likely safer): http://localhost/Thingworx/Things/MyThing1?method=DELETE&content-type=application/JSON   Exporting all data within ThingWorx using the DataExporter functionality: http://localhost/Thingworx/DataExporter?Accept=application/octet-stream   Exporting all entities which have the Model Tag “Application.TestTerm” within ThingWorx using the Exporter functionality: http://localhost/Thingworx/Exporter?Accept=text/xml&searchTags=Applications:TestTerm

This video gives an introduction to the Descriptive Services: what they are how to install them how to configure them how to use them  

Recently a customer from the ThingWorx Academic Program sent in a sample program they were having problems with. They were trying to post data from a Raspberry PI using Python to their ThingWorx server. It turns out that their program did work just fine and was also a great example of posting data from a PI using REST. Here is how to set up this example. 1. Import the attached "Things_TempAndHumidityThing.xml" entity file. 2. from the PI run 'sudo pip install requests' 3. from the PI run 'sudo pip install logging' 4. from the PI run 'sudo pip install http_client' 5. Create a python file call test.py that contains this example code: #!/usr/bin/python import requests import json import logging import sys # These two lines enable debugging at httplib level (requests->urllib3->http.client) # You will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA. # The only thing missing will be the response.body which is not logged. try:     import http.client as http_client except ImportError:     # Python 2     import httplib as http_client http_client.HTTPConnection.debuglevel = 1 # You must initialize logging, otherwise you'll not see debug output. logging.basicConfig() logging.getLogger().setLevel(logging.DEBUG) requests_log = logging.getLogger("requests.packages.urllib3") requests_log.setLevel(logging.DEBUG) requests_log.propagate = True #NYP Webserver URL in Thingworx NYP_Webhost = sys.argv[1] App_Key = sys.argv[2] ThingName = 'TempAndHumidityThing' headers = { 'Content-Type': 'application/json', 'appKey': App_Key } payload = { 'Prop_Temperature': 45, 'Prop_Humidity': 33 } response = requests.put(NYP_Webhost + '/Thingworx/Things/' + ThingName + '/Properties/*', headers=headers, json=payload, verify=False) 6. From the command line run, './test.py http://twhome:8080 e9274d87-58aa-4d60-b27f-e67962f3e5c4' except substitute your server and your app key. 7. A successful response should look like: INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): twhome send: 'PUT /Thingworx/Things/TempAndHumidityThing/Properties/* HTTP/1.1\r\nHost: twhome:8080\r\nappKey: e9274d87-58aa-4d60-b27f-e67962f3e5c4\r\nContent-Length: 45\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.8.1\r\nConnection: keep-alive\r\nContent-Type: application/json\r\n\r\n{"Prop_Temperature": 45, "Prop_Humidity": 33}' reply: 'HTTP/1.1 200 OK\r\n' header: Server: Apache-Coyote/1.1 header: Set-Cookie: JSESSIONID=E7436D2E6AE81C84EC197D406E7E365A; Path=/Thingworx/; HttpOnly header: Expires: 0 header: Cache-Control: no-store, no-cache header: Cache-Control: post-check=0, pre-check=0 header: Pragma: no-cache header: Content-Type: text/html;charset=UTF-8 header: Transfer-Encoding: chunked header: Date: Mon, 09 Nov 2015 12:39:24 GMT DEBUG:requests.packages.urllib3.connectionpool:"PUT /Thingworx/Things/TempAndHumidityThing/Properties/* HTTP/1.1" 200 None My thanks to the customer who sent in the simple example.