IoT Tips

  DevOps. It’s not just a buzzword. It’s a true development methodology that can make all the difference in your application quality and release time. Today, I’ll walk you through how you can continuously integrate and deploy your ThingWorx applications to achieve CI/CD objectives as part of a DevOps-focused culture. At the end, I’ll provide you a sneak peek of what you can expect in a future release (hint: we’re working on some awesome new CI/CD functionality). Overview of ThingWorx DevOps and Common Tools I’ll start by providing an overview of the DevOps cycle, and then I’ll provide more details around each step of the cycle. Before we can start, you’ll need to define your high-level architecture and functional requirements as part of the “Plan” phase.   Now, let’s build your ThingWorx app. Ready? Here we go!   Code As with any software platform, developers can start working in any number of areas of the IoT application—from edge, to visualization, rules authoring to data modelling. For the purpose of this article, we’ll start with the UI, but much of the same steps can be applied in any order. Also, we’ll just call out high level steps of development, but for more info on building out each aspect of your application, please visit developer.thingworx.com.   In ThingWorx Composer, build out your user interface with Mashups. Starting with UI can help you think about the types of data you want to collect from devices and systems and how you want to solve your unique requirements for the business. Starting at this point can also help you show live POCs and functional mockups to stakeholders. Once you’ve built some starter screens and a skeleton of app navigation, you can start adding in data through configuration in Composer by creating your Things, Templates, properties and services. [Optional] We offer 65 out-of-the-box widgets for the UI in the ThingWorx platform. There are times when you have specific visualization requirements for your application and the out-of-the-box widgets don’t quite satisfy them. We have a path for that, through our custom widget extensions. If you choose to develop your own widget extensions, you can do so through other IDEs like Eclipse or WebStorm. Custom development and extensions are not just for UI. We also allow you to define Thing entities and their custom services in Java. If you are developing extensions in this way, we’d recommend you do so using Eclipse to code and Gradle to build and drive tests. For instructions on how to create your own extension, see “Creating Customized ThingWorx Widgets” on page 42 of the ThingWorx Application Development Guide posted on Ask Kaya. With a good start on the data model, business logic and UI, some quick testing and validation is in order. You’ll probably also want to save all of this work also to share with colleagues or move to other integration environments. Capture all of your entity and code artifacts (Mashups, style definitions, Thing shapes, Thing templates, JavaScript, etc.) by using the “Export to Source Control” feature from ThingWorx Composer to write entities to the file system. You can use Git or other source systems to monitor the file system and push to the remote repository of your choice (e.g. GitHub, Bitbucket, etc.). Again, if you are developing extensions outside of Composer, you’ll want to source control those items, too, from Eclipse or the file system directly. Build [Optional] You can build an application package as an extension with all entities and code from Eclipse using the ThingWorx Eclipse plugin. When you build the project, it will create an extension zip file. Again, more info in the Application Development Guide. Make your life easy by using tools like Gradle or Maven. ThingWorx is very similar to other Java development systems, so Gradle and Maven track your dependencies and create a package with all of the referenced extensions you may be using and put them into one single zip file package. Once you have a package built, you can import it into test or integration environments. For added automation, create repeatable tasks like a job in Jenkins so that every time your code is changed in the source repository (e.g. Git), it triggers a job to increment the version, build the project and create the package deliverables. Consider also configuring the Jenkins jobs to push artifacts to a central repository like Artifactory. Test Once your code has been built, we can’t forget about testing! Automation is king for DevOps! For ThingWorx apps, you should still design a test strategy for your application, and then define and create your tests. These can run in your local developer environment, as well as be triggered via build tasks/changes in the source repository. Tools like JUnit for your entities and Java-backed services or Selenium for testing the Mashup UIs can be used. You can create separate jobs in Jenkins along with the build to run the integration and unit tests against an instance of ThingWorx that has the latest artifacts deployed into it. You can also do static code analysis using tools like PMD to find bugs, check style issues or identify inefficient code paths. To round out your app also with performance and load testing, JMeter is one tool that you can leverage. Release Releasing is the culmination of the team’s great work! If the test results pass and the builds are green, you are good to go, and it’s time to establish your release build. Make sure that you consider a versioning scheme for your application and its artifacts. Semantic versioning is a pattern that can be implemented for your ThingWorx application. Correct versioning of ThingWorx packages affects your upgrade plans and is a signal to your users on the intent and content of the release. Again, see the Application Development Guide. Once a release milestone is met, you can create a source branch in Git for that milestone, which will have all the changes encompassed in that release. Configure a Jenkins job to create builds from that milestone branch for maintenance purposes. Deploy + Operate + Monitor   If you’ve tested and released your application, it’s time for production and real users! Using the build and testing infrastructure you’ve set up earlier in the development process, you can also deploy your release builds to your target staging and production ThingWorx environments with Jenkins jobs, Artifactory and automated steps. Finally, as with anything, it is important to measure success and monitor performance via KPIs, trends and logs. You can also extract application insights and recommendations from the PTC System Monitor (PSM) tool, which uses Dynatrace; here is a guide on how to install and deploy PSM.  There are many different paths through the platform and options for developers to match your local team processes and tools—this was simply a quick overview. Congrats! You’re now equipped to build ThingWorx apps while leveraging software best practices and incorporating a DevOps culture!   What can I expect in a future release of ThingWorx? Coming in a near-term release of ThingWorx, we’ll make it easier for you to continuously integrate and deploy your ThingWorx applications. How? Through new functionality that bolsters our packaging concepts, new cloud services to assist in deployment to environments and an error-proof way to integrate applications with an automated dependency awareness.   Stay tuned for more info about this exciting new deployment and application management functionality targeted for Fall 2019!   Reach out with any questions and stay connected.   -Kaya

  Question: What should I know about using ThingWorx with InfluxDB to store my time series data? Hi, ThingWorx users!   It’s here! Thanks for waiting patiently since my previous post announcing ThingWorx’ new support of InfluxDB as a time series persistence provider.   As of our 8.4 release, you can now use InfluxDB to store your ThingWorx time series data with incredible power and ease.   Want to learn more? Check out the following FAQs:   1. What is InfluxDB? Who is InfluxData? InfluxDB is a time series database designed to handle high write and query loads. It is meant to be used as a backing store for any use case involving large amounts of timestamped data, like monitoring, application metrics, IoT sensor data, and real-time analytics that you’d find in ThingWorx.   InfluxDB is created by InfluxData, an awesome company that we are proud to call a PTC partner.   2. When would I want to use InfluxDB for IIoT? While the ThingWorx IIoT platform supports multiple databases to persist IIoT data and is agnostic when it comes to the storage layer, InfluxDB is the ideal choice for time series. When the number of connected devices increases, along with the amount of streaming data, the need to have a high-scale telemetry database choice is obvious.   For very high scale data ingestion, InfluxDB should be used as a persistent provider with the ThingWorx platform for multiple reasons. Its flexibility and ease of use provides native support for standard time series functions, including: sampling, interpolation, time bucketing, aggregation, selector, transformation, predictor, etc. It does all of this while supporting a high compression of data (~45x) with the ability to handle thousands of writes per second and read thousands of rows in milliseconds.   Check out this article by our Enterprise Deployment Center (EDC) explaining why InfluxDB is great for small ThingWorx applications.   3. What are the three different flavors of InfluxDB? InfluxDB Open Source (TICK Stack), InfluxDB Enterprise & InfluxDB Cloud. Here’s more info on each: InfluxDB Open Source (TICK Stack): This is the open-source version of the product available to download via the InfluxData website. Also included here are the other projects that comprise the TICK Stack, including: [T] Telegraf; open source collection agent [I] InfluxDB; open source time series database [C] Chronograf; open source visualization application [K] Kapacitor; open source streaming processing engine; side car to InfluxDB InfluxDB Enterprise: This is the commercial software version of InfluxDB for high availability clustering and the recommended time series database to be used for production with ThingWorx 8.4 and later. InfluxDB Enterprise works with the rest of the TICK stack interchangeably (Telegraf, Chronograf, Kapacitor). InfluxDB Cloud: This is the commercial service version of InfluxDB, hosted on AWS, managed by InfluxData, and delivered as a service to customers. InfluxDB Cloud works with the rest of the TICK stack interchangeably (Telegraf, Chronograf, Kapacitor). To learn more about the different modules of InfluxDB (Telegraf, Chronograf, Kapcitor), check out InfluxData Introduction for documentation or InfluxData Products for product info.   4. What is the difference between InfluxDB opensource and enterprise? InfluxDB Open Source is available in a single (1 only) data node configuration only, albeit with “n” number of vCPU or “cores” provisioned on that single node.  InfluxDB Enterprise is available in multiple (2 or more) data node configuration, also with “n” number of vCPU or “cores” provisioned to each node. The Enterprise edition is generally preferred for production deployments that require high availability, replication, and redundancy. Provisioned along with the data nodes are three (3) meta nodes and a load balancer to distribute data workload across the multiple nodes. Typical configurations are in even increments of data nodes (i.e. 2, 4, 6, 8, etc.).   5. Where can I find the pricing overview for buying enterprise licenses for InfluxDB? The PTC product and go-to-market team have defined commercial pricing for InfluxDB Enterprise. For help with pricing, reach out to Chris Wensley (cwensley@ptc.com) and Anders Hinrichsen (anders@influxdata.com).   6. How do I configure InfluxDB with ThingWorx? We’ve outlined the steps for you in the ThingWorx Help Center and created a quick video to instruct you on how to install InfluxDB with ThingWorx. (view in My Videos) To see the current version of InfluxDB that we support, read our ThingWorx 9.0 System Requirements guide.   7. How do I configure InfluxDB and ThingWorx in a high availability scenario? With the ability to leverage multiple data stores, we work to provide the flexibility to best meet the needs of your IT preferences and investments. InfluxDB helps us do that. To configure ThingWorx for High Availability, please refer to this section of the ThingWorx Platform 9 Help Center. To configure InfluxDB for High Availability at the database level, please refer to InfluxData’s documentation on how to Install and deploy InfluxDB Enterprise clusters.   8. Where can I learn more about how to monitor and manage InfluxDB? Monitoring info for InfluxDB can be found here: Monitoring Tools for TICK Stack.   9. How can I tune and optimize InfluxDB with ThingWorx? The best approach for running InfluxDB with PTC ThingWorx 8.4 (or later) is to treat the workload and configuration just as you would in a stand-alone deployment. We suggest to stick to the recommendations in the InfluxDB and TICK stack documentation.   10. How do I perform backup and recovery of ThingWorx with InfluxDB? Please see the ThingWorx Platform Backup and Recovery Planning Technical Brief to plan for back and recovery. You can also find more more details on taking backups and restoring data from InfluxDB in the Backing up and restoring in InfluxDB Enterprise overview.   11. Where can I learn more about sampling, interpolation, time bucketing, aggregation, pivot​ and other key features of InfluxDB? Features of InfluxDB can be found here: InfluxData Time Series Platform. Implementation of InfluxDB features can be found here: Getting Started with InfluxDB.   12. What are all the different persistence providers supported with ThingWorx? When should I use InfluxDB? ThingWorx supports the following model and data provider storage options: H2, PostgreSQL, MS SQL Server and AzureSQL ThingWorx supports the following data provider only storage options: InfluxDB Please refer to the model and data best practices section of the ThingWorx 9 Help Center for further information on options how to store your model and data with ThingWorx.   We have also updated the ThingWorx Platform 9.0 Sizing Guide to provide relevant information to estimate the amount of processing and memory that ThingWorx may need to meet your requirements. It also provides guidance on when to use InfluxDB for your scale needs.   13. When should I use InfluxDB over DataStax Enterprise (DSE)? Here is a good blog post that benchmarks time series data performance of InfluxDB vs. Cassandra, which is the core of DataStax Enterprise (DSE). In specific use cases, InfluxData Enterprise may be more cost effective when compared to similar telemetry use cases with DSE.   14. How can I migrate my data from PostgreSQL to InfluxDB? Migration from PostgreSQL or MSSQL is supported by the ThingWorx in-built data tools, which can export entities and data from PostgreSQL or MSSQL and then import them into InfluxDB.   Details on how to upgrade to ThingWorx 9.0 can be found in the Upgrading ThingWorx  section of the ThingWorx 9 Help Center.   15. Should I use InfluxDB as a time series store rather than OSI PI, IP21, or others? For ThingWorx 8.4 and later, InfluxDB is the recommended time series store. This can be implemented at the edge with ThingWorx (i.e. “front end”) using the open source edition and can also be implemented at the hub (i.e. “back end”) using either of the commercial editions designed for HA production workloads.   As always, ThingWorx can connect to most industrial software, including OSI PI, IP21, etc. with our integration toolset.   That’s a wrap—almost! We’ve added two extra questions for you.   16. What’s on the roadmap for ThingWorx with InfluxDB? Key development work to fully leverage built-in InfluxDB querying capabilities and support InfluxDB 2.0 in future ThingWorx releases Leveraging query operations capabilities from InfluxDB to further improve query performance Supporting additional native InfluxDB features (e.g. continuous queries)   17. What should I do if I need technical support with InfluxDB? If you select InfluxDB as your persistence provider, then all support requests related to configuring InfluxDB can be logged through PTC Technical Support at https://support.ptc.com or by calling 1-800-477-6435. You may also want to use the PTC Community to learn and collaborate with the growing PTC developer community. For all other requests related to database management, troubleshooting, monitoring, and administration, we encourage you to reach out to InfluxData directly based on your enterprise purchase contract made with InfluxData. PTC customers using InfluxDB can also email ptc-support@influxdata.com for support requests related to InfluxData.   If you’re as excited as I am about the ability to store your time series data with InfluxDB, let me know in the comments below!   Until next time, if you have any questions, just ask Kaya!

Hi Community,   I've recently had a number of questions from colleagues around architectures involving MQTT and what our preferred approach was.  After some internal verification, I wanted to share an aggregate of my findings with the ThingWorx Architect and Developer Community.   PTC currently supports four methods for integrating with MQTT for IoT projects. ThingWorx Azure IoT Hub Connector ThingWorx MQTT Extension ThingWorx Kepware Server Choice is nice, but it adds complexity and sometimes confusion.  The intent of this article is to clarify and provide direction on the subject to help others choose the path best suited for their situation.   ThingWorx MQTT Extension The ThingWorx MQTT extension has been available on the marketplace as an unsupported “PTC Labs” extension for a number of years.  Recently its status has been upgraded to “PTC Supported” and it has received some attention from R&D getting some bug fixes and security enhancements.  Most people who have used MQTT with ThingWorx are familiar with this extension.  As with anything, it has advantages and disadvantages.  You can easily import the extension without having administrative access to the machine, it’s easy to move around and store with projects, and can be up and running quite quickly.  However it is also quite limited when it comes to the flexibility required when building a production application, is tied directly to the core platform, and does not get feature/functionality updates.   The MQTT extension is a good choice for PoCs, demos, benchmarks, and prototypes as it provides MQTT integration relatively quickly and easily.  As an extension which runs with the core platform, it is not a good choice as a part of a client/enterprise application where MQTT communication reliability is critical.   ThingWorx Azure IoT Hub Connector Although Azure IoT Hub is not a fully functional MQTT broker, Azure IoT does support MQTT endpoints on both IoT Hub and IoT Edge.  This can be an interesting option to have MQTT devices publish to Azure IoT and be integrated to ThingWorx using the Azure IoT Hub Connector without actually requiring an MQTT broker to run and be maintained.  The Azure IoT Hub Connector works similarly to the PAT and is built on the Connection Server, but adds the notion of device management and security provided by Azure IoT.   When using Azure IoT Edge configured as a transparent gateway with buffering (store and forward) enabled, this approach has the added benefit of being able to buffer MQTT device messages at a remote site with the ability to handle Internet interruptions without losing data.   This approach has the added benefit of having far greater integrated security capabilities by leveraging certificates and tying into Azure KeyVault, as well as easily scaling up resources receiving the MQTT messages (IoT Hub and Azure IoT Hub Connector).  Considering that this approach is build on the Connection Server core, it also follows our deployment guidance for processing communications outside of the core platform (unlike the extension approach).   ThingWorx Kepware Server As some will note, KepWare has some pretty awesome MQTT capabilities: both as north and southbound interfaces.  The MQTT Client driver allows creating an MQTT channel to devices communicating via MQTT with auto-tag creation (from the MQTT payload).  Coupled with the native ThingWorx AlwaysOn connection, you can easily connect KepWare to an on-premise MQTT broker and connect these devices to ThingWorx over AlwaysOn.   The IoT Gateway plug-in has an MQTT agent which allows publishing data from all of your KepWare connected devices to an MQTT broker or endpoint.  The MQTT agent can also receive tag updates on a different topic and write back to the controllers.  We’ve used this MQTT agent to connect industrial control system data to ThingWorx through cloud platforms like Azure IoT, AWS, and communications providers.   ThingWorx Product Segment Direction A key factor in deciding how to design your solution should be aligned with our product development direction.  The ThingWorx Product Management and R&D teams have for years been putting their focus on scalable and enterprise-ready approaches that our partners and customers can build upon.  I mention this to make it clear that not all supported approaches carry the same weight.  Although we do support the MQTT extension, it is not in active development due to the fact that out-of-platform microservices-based communication interfaces are our direction forward.   The Azure IoT Hub Connector, being built on the Connection Server is currently the way forward for MQTT communications to the ThingWorx Foundation.   Regards,   Greg Eva

Exciting news! ThingWorx now has improved support for Docker containers to help you manage CI/CD, improve development efficiency in your organization and save costs. Check out these FAQs below and, as always, reach out to me if you have any additional questions.   Stay connected, Kaya   FAQs: ThingWorx Docker Containers   What are Docker Containers? From Docker.com: “a Docker container image is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings”. Learn more here.   What's the difference between Docker containers and VMs? Containers are an abstraction at the app layer that packages code and dependencies together, whereas Virtual Machines (VMs) are an abstraction of physical hardware turning one server into many servers. Here are some great discussions on it on Stack Overflow. Containers vs. VMs   How can I build ThingWorx Docker images? Check out the Building ThingWorx 8.3 Docker Images Guide or watch this video to instruct you on how to build and test Docker containers. (view in My Videos)   How does PTC support building ThingWorx Docker images? PTC provides the ability for customers and partners to build ThingWorx Docker images. A customer can download the Dockerfiles and scripts packaged as a zip folder from the PTC Software Downloads Portal under “ThingWorx Platform,” then “Release 8.3”  then“ThingWorx Dockerfiles.” (Please note that you must be logged in for the link to function properly.) PTC Software Downloads PortalThe zip folder contains the Dockerfiles, template jar, and scripts to fetch Tomcat, and ThingWorx WAR files using CLI. Java must be downloaded manually from the vendor's website. We also provide an instructional guide called “Build ThingWorx Docker Images” available on the Reference Documents page on the Support Portal.   How are ThingWorx Docker images different from the usual delivery media of WAR files? The WAR file delivery is typically accompanied by an installation guide that contains the manual steps for creating the VM or bare-metal environment. That guide includes instructions for the administrator to manually install the prerequisites, including Tomcat, Java, and ThingWorx platform settings files. To deploy and run the WAR file, the administrator follows the guide to create the runtime environment on an OS. In contrast, the Dockerfile build in this delivery automates the creation of a Docker image once supplied with the prerequisites.   Do you have any reference deployment and guidance? Yes, you can refer to our blog post to learn how to deploy and run ThingWorx Docker containers on your existing Kubernetes environment.   Is there any recommendation on which Container Orchestrator as a Service (CaaS) a customer should run ThingWorx Foundation Docker container images on? You can use Docker-Compose for testing, but it is generally not suggested for production deployment use cases. In a production environment, customers should use container orchestrators such as Kubernetes, OpenShift, Azure Kubernetes Service (AKS), or Amazon Elastic Container Service for Kubernetes (Amazon EKS), to deploy and manage ThingWorx Docker images.   What are the skill sets required? Familiarity with OS CLI and Docker tools is required to build building the ThingWorx Docker images. Familiarity with Docker-compose to run the resulting Docker containers is needed to test the resulting builds. We don’t recommend Docker-Compose for production use, but when using it for local testing and demo purposes, users can rapidly install ThingWorx and get it up and running in minutes. We expect PTC partners and customers who want to run ThingWorx containerized instances in their production environment to possess the required skill sets within their DevOps team.   How is ThingWorx licensing handled with the Docker images? By default, the container created from these Docker images starts up in a limited mode with no license supplied. You can configure your username and password for the PTC licensing portal to automatically load a license via environment variables passed into the container on startup. Additionally, you can mount a volume to the /ThingworxPlatform directory, which contains your license file, or to retrieve a license request. To keep your Host ID consistent, ensure that the /ThingworxStorage and /ThingworxPlatform directories are persisted and not removed with individual container restarts. More detailed instructions can be found in the build guide or in a Kubernetes blog post .   Is Docker free? What version of Docker does PTC support for ThingWorx? Docker is open-source and licensed under the Apache 2 license. Information on Docker licensing can be found here. The following Docker versions are required: Docker Community Edition (docker-ce) Version 18.05.0-ce is recommended. To install the Docker Community Edition on your system, follow the instructions for your operating system on the Docker website here. Docker Compose (docker-compose) Version 1.17.1 is recommended. To install the Docker Compose on your system, follow the instructions for your operating system on the Docker website here. What persistence providers are currently supported? PTC provides the ability to build ThingWorx Foundation containers for the following supported persistence providers: H2 Microsoft SQL Server PostgreSQL Additional persistence providers will be added to the Docker build delivery as the ThingWorx Foundation Platform releases support for those new databases in future releases.   What are some of the security best practices? For production use, customers are strongly advised to secure their Docker environments by following all the recommendations provided by Docker. Review and implement the best practices detailed at https://docs.docker.com/engine/security/security/.   Can we build Docker images for ThingWorx High Availability (HA) architecture? Yes. ThingWorx Dockerfiles are provided for both basic ThingWorx deployment architecture and HA ThingWorx deployment architecture.   How easy is the rehosting and upgrading of ThingWorx releases on Docker with existing data? In Kubernetes environment, data is kept in a separate volume and can be attached to different containers. When one container dies, the data can be attached to a different container and the container should start without issue. For more information, please refer to the upgrade section of the Building ThingWorx 8.3 Docker Images Guide.   Is it okay to use the Docker exec and access the bash shell to make config changes or should I always rebuild the image and re-deploy?­ Although using Docker exec to gain access to the container internals is useful for testing and troubleshooting issues, any changes made will not be saved after a container is stopped. To configure a container's environment, variables are passed in during the start process. This can be done with Docker start commands, using compose files with environment variables defined, or with helm charts. More detailed instructions can be found in the build guide or in this blog post .   What if there are issues? Should I call PTC Technical Support? We are providing the scripts and reference documents solely to empower our community to build ThingWorx Docker images. We believe that customers using Docker in their production processes would have expertise to manage running Docker containers themselves. If there are any issues or questions regarding the build scripts provided in the PTC official downloads portal, then customers can contact PTC Technical Support at 1-800-477-6435 or visit us online at: http://support.ptc.com. PTC does not provide support for orchestration troubleshooting.   What can you share about future roadmap plans? As we are enabling our customers and partners to build ThingWorx Foundation Platform Docker images, we plan to do the same for upcoming products such as ThingWorx Integration & Orchestration, ThingWorx Analytics, upcoming persistence providers such as InfluxDB, and many more. We also plan to provide additional reference architecture examples and use cases to help developers understand how to use Docker containers in their DevOps and production environments.   Where can I learn more about Docker containers and container orchestrators? See these resources below for additional information: https://training.docker.com/ https://kubernetes.io/docs/tutorials/online-training/overview/

In ThingWorx Analytics, you have the possibility to use an external model for scoring. In this written tutorial, I would like to provide an overview of how you can use a model developed in Python, using the scikit-learn library in ThingWorx Analytics. The provided attachment contains an archive with the following files: iris_data.csv: A dataset for pattern recognition that has a categorical goal. You can click here to read more about this dataset TestRFToPmml.ipynb: A Jupyter notebook file with the source code for the Python model as well as the steps to export it to PMML RF_Iris.pmml: The PMML file with the model that you can directly upload in Analytics without going through the steps of training the model in Python The tutorial assumes you already have some knowledge of ThingWorx and ThingWorx Analytics. Also, if you plan to run the Python code and train the model yourself, you need to have Jupyter notebook installed (I used the one from the Anaconda distribution). For demonstration purposes, I have created a very simple random forest model in Python. To convert the model to PMML, I have used the sklearn2pmml library. Because ThingWorx Analytics supports PMML format 4.3, you need to install sklearn2pmml version 0.56.2 (the highest version that supports PMML 4.3). To read more about this library, please click here Furthermore, to use your model with the older version of the sklearn2pmml, I have installed scikit-learn version 0.23.2.  You will find the commands to install the two libraries in the first two cells of the notebook.   Code Walkthrough The first step is to import the required libraries (please note that pandas library is also required to transform the .csv to a Dataframe object):   import pandas from sklearn.ensemble import RandomForestClassifier from sklearn2pmml import sklearn2pmml from sklearn.model_selection import GridSearchCV from sklearn2pmml.pipeline import PMMLPipeline   After importing the required libraries, we convert the iris_data.csv to a pandas dataframe and then create the features (X) as well as the goal (Y) vectors:   iris_df = pandas.read_csv("iris_data.csv") iris_X = iris_df[iris_df.columns.difference(["class"])] iris_y = iris_df["class"]   To best tune the random forest, we will use the GridSearchCSV and cross-validation. We want to test what parameters have the best validation metrics and for this, we will use a utility function that will print the results:   def print_results(results): print('BEST PARAMS: {}\n'.format(results.best_params_)) means = results.cv_results_['mean_test_score'] stds = results.cv_results_['std_test_score'] for mean, std, params in zip(means, stds, results.cv_results_['params']): print('{} (+/-{}) for {}'.format(round(mean, 3), round(std * 2, 3), params))   We create the random forest model and train it with different numbers of estimators and maximum depth. We will then call the previous function to compare the results for the different parameters:   rf = RandomForestClassifier() parameters = { 'n_estimators': [5, 50, 250], 'max_depth': [2, 4, 8, 16, 32, None] } cv = GridSearchCV(rf, parameters, cv=5) cv.fit(iris_X, iris_y) print_results(cv)   To convert the model to a PMML file, we need to create a PMMLPipeline object, in which we pass the RandomForestClassifier with the tuning parameters we identified in the previous step (please note that in your case, the parameters can be different than in my example). You can check the sklearn2pmml  documentation  to see other examples for creating this PMMLPipeline object :   pipeline = PMMLPipeline([ ("classifier", RandomForestClassifier(max_depth=4,n_estimators=5)) ]) pipeline.fit(iris_X, iris_y)   Then we perform the export:   sklearn2pmml(pipeline, "RF_Iris.pmml", with_repr = True)   The model has now been exported as a PMML file in the same folder as the Jupyter Notebook file and we can upload it to ThingWorx Analytics.   Uploading and Exploring the PMML in Analytics To upload and use the model for scoring, there are two steps that you need to do: First, the PMML file needs to be uploaded to a ThingWorx File Repository Then, go to your Analytics Results thing (the name should be YourAnalyticsGateway_ResultsThing) and execute the service UploadModelFromRepository. Here you will need to specify the repository name and path for your PMML file, as well as a name for your model (and optionally a description)   If everything goes well, the result of the service will be an id. You can save this id to a separate file because you will use it later on. You can verify the status of this model and if it’s ready to use by executing the service GetDetails:   Assuming you want to use the PMML for scoring, but you were not the one to develop the model, maybe you don’t know what the expected inputs and the output of the model are. There are two services that can help you with this: QueryInputFields – to verify the fields expected as input parameters for a scoring job   QueryOutputFields – to verify the expected output of the model The resultType input parameter can be either MODELS or CLUSTERS, depending on the type of model,    Using the PMML for Scoring With all this information at hand, we are now ready to use this PMML for real-time scoring. In a Thing of your choice, define a service to test out the scoring for the PMML we have just uploaded. Create a new service with an infotable as the output (don’t add a datashape). The input data for scoring will be hardcoded in the service, but you can also add it as service input parameters and pass them via a Mashup or from another source. The script will be as follows:   // Values: INFOTABLE dataShape: "" let datasetRef = DataShapes["AnalyticsDatasetRef"].CreateValues(); // Values: INFOTABLE dataShape: "" let data = DataShapes["IrisData"].CreateValues(); data.AddRow({ sepal_length: 2.7, sepal_width: 3.1, petal_length: 2.1, petal_width: 0.4 }); datasetRef.AddRow({ data: data}); // predictiveScores: INFOTABLE dataShape: "" let result = Things["AnalyticsServer_PredictionThing"].RealtimeScore({ modelUri: "results:/models/" + "97471e07-137a-41bb-9f29-f43f107bf9ca", //replace with your own id datasetRef: datasetRef /* INFOTABLE */, });   Once you execute the service, the output should look like this (as we would have expected, according to the output fields in the PMML model):   As you have seen, it is easy to use a model built in Python in ThingWorx Analytics. Please note that you may use it only for scoring, and the model will not appear in Analytics Builder since you have created it on a different platform. If you have any questions about this brief written tutorial, let me know.

Back in 2018 an interesting capability was added to ThingWorx Foundation allowing you to enable statistical calculation of service and subscription execution.   We typically advise customers to approach this with caution for production systems as the additional overhead can be more than you want to add to the work the platform needs to handle.  This said, these statistics is used consciously can be extremely helpful during development, testing, and troubleshooting to help ascertain which entities are executing what services and where potential system bottlenecks or areas deserving performance optimization may lie.   Although I've used the Utilization Subsystem services for statistics for some time now, I've always found that the Composer table view is not sufficient for a deeper multi-dimensional analysis.  Today I took a first step in remedying this by getting these metrics into Excel and I wanted to share it with the community as it can be quite helpful in giving developers and architects another view into their ThingWorx applications and to take and compare benchmarks to ensure that the operational and scaling is happening as was expected when the application was put into production.   Utilization Subsystem Statistics You can enable and configure statistics calculation from the Subsystem Configuration tab.  The help documentation does a good job of explaining this so I won't mention it here.  Base guidance is not to use Persisted statistics, nor percentile calculation as both have significant performance impacts.  Aggregate statistics are less resource intensive as there are less counters so this would be more appropriate for a production environment.  Specific entity statistics require greater resources and this will scale up as well with the number of provisioned entities that you have (ie: 1,000 machines versus 10,000 machines) whereas aggregate statistics will remain more constant as you scale up your deployment and its load.   Utilization Subsystem Services In the subsystem Services tab, you can select "UtilizationSubsystem" from the filter drop down and you will see all of the relevant services to retrieve and reset the statistics.     Here I'm using the GetEntityStatistics service to get entity statistics for Services and Subscriptions.     Giving us something like this.      Using Postman to Save the Results to File I have used Postman to do the same REST API call and to format the results as HTML and to save these results to file so that they can be imported into Excel.   You need to call '/Thingworx/Subsystems/UtilizationSubsystem/Services/GetEntityStatistics' as a POST request with the Content-Type and Accept headers set to 'application/xml'.  Of course you also need to add an appropriately permissioned and secured AppKey to the headers in order to authenticate and get service execution authorization.     You'll note the Export Results > Save to a file menu over on the right to get your results saved.   Importing the HTML Results into Excel As simple as I would like to hope that getting a standard web formatted file into Excel should be, it didn't turn out to be as easy as I would have hoped and so I have to switch over to Windows to take advantage of Power Query.   From the Data ribbon, select Get Data > From File > From XML.  Then find and select the HTML file saved in the previous step.     Once it has loaded the file and done some preparation, you'll need to select the GetEntityStatistics table in the results on the left.  This should display all of the statistics in a preview table on the right.     Once the query completed, you should have a table showing your statistical data ready for... well... slicing and dicing.     The good news is that I did the hard part for you, so you can just download the attached spreadsheet and update the dataset with your fresh data to have everything parsed out into separate columns for you.     Now you can use the column filters to search for entity or service patterns or to select specific entities or attributes that you want to analyze.  You'll need to later clear the column filters to get your whole dataset back.     Updating the Spreadsheet with Fresh Data In order to make this data and its analysis more relevant, I went back and reset all of the statistics and took a new sample which was exactly one hour long.  This way I would get correct recent min/max execution time values as well as having a better understanding of just how many executions / triggers are happening in a one hour period for my benchmark.   Once I got the new HTML file save, I went into Excel's Data ribbon, selected a cell in the data table area, and clicked "Queries & Connections" which brought up the pane on the right which shows my original query.     Hovering over this query, I'm prompted with some stuff and I chose "Edit".     Then I clicked on the tiny little gear to the right of "Source" over on the pane on the right side.     Finally I was able to select the new file and Power Query opened it up for me.     I just needed to click "Close & Load" to save and refresh the query providing data to the table.     The only thing at this point is that I didn't have my nice little sparklines as my regional decimal character is not a period - so I selected the time columns and did a "Replace All" from '.' to ',' to turn them into numbers instead of text.     Et Voila!   There you have it - ready to sort, filter, search and review to help you better understand which parts of your application may be overly resource hungry, or even to spot faulty equipment that may be communicating and triggering workflows far more often than it should.   Specific vs General Depending on the type of analysis that you're doing you might find that the aggregate statistics are a better option.  As they'll be far, far less that the entity specific statistics they'll do a better job of giving you a holistic view of the types of things that are happening with your ThingWorx applications execution.   The entity specific data set that I'm showing here would be a better choice for troubleshooting and diagnostics to try to understand why certain customers/assets/machines are behaving strangely as we can specifically drill into these stats.  Keep in mind however that you should then compare these findings with the general baseline to see how this particular asset is behaving compared to the whole fleet.   As a size guideline - I did an entity specific version of this file for a customer with 1,000 machines and the Excel spreadsheet was 7Mb compared to the 30kb of the one attached here and just opening it and saving it was tough for Excel (likely due to all of my nested formulas).  Just keep this in mind as you use this feature as there is memory overhead meaning also garbage collection and associated CPU usage for such.

  Hi, everyone!   We’re actively working towards the ThingWorx 9.0 release and we’re ready to provide a sneak peek into the biggest feature of 9.0: Active-Active Clustering for High Availability configuration.   You may be wondering: doesn’t ThingWorx already offer High Availability?   Yes, ThingWorx already supports a High Availability configuration. Previous versions of ThingWorx, such as ThingWorx 8.X version releases, support Active-Passive configuration, where one “active” ThingWorx server performs all processing and maintains the live connections to other systems such as databases and connected assets. Meanwhile, in parallel, there is a second “passive” ThingWorx server that is a mirror image and regularly updated with data but does not maintain active connections to any of the other systems. If the “active” ThingWorx server fails, the “passive” ThingWorx server is made the primary server, but this can take a few minutes to establish connections to the other systems.   So, how is ThingWorx Active-Active different?   Active-Active configuration differs from Active-Passive in that all the ThingWorx servers in the cluster are “active.” Not only is data mirrored across all ThingWorx servers, but all of the servers, instead of only one, maintain live connections with the other systems. This way, if any of the ThingWorx servers fail, the other ThingWorx servers take over instantaneously with no recovery time.   Since all ThingWorx servers are active, they are processing in parallel and, as a result, the cluster can process more data than that of a single server or a cluster with an Active-Passive configuration. Simply put, multiple servers working together outperform a single server. This allows customers to scale their deployment by simply adding more ThingWorx servers to the cluster (horizontally scaling), which does not have the same limitations of scale that is achieved by increasing the performance of the server itself.   What does that mean for me? Higher Availability - You can avoid single points of failure and configure the ThingWorx Foundation platform in an Active-Active cluster mode to achieve the highest availability for your IIoT systems and applications. Increased Scalability - Now, you can horizontally scale from one to many ThingWorx servers to easily manage large amounts of your IIoT data at scale more smoothly than ever before.   Stay tuned! We’ll be posting more information on Active-Active Clustering—how it's achieved in ThingWorx, architectural component overviews, and what it means for your ThingWorx deployment!   In the meantime, we're running the Active-Active Clustering Beta Program. Interested in participating? Reach out to Ryan Servais (rservais@ptc.com) or Ayush Tiwari (atiwari@ptc.com) to learn more about participation!   Stay connected! Kaya

  The ThingWorx team would like to be one of the first to wish you a happy holiday season.   To kick it off, I’ll share the top three things we’re thankful for: Our customers—you! Thank you for all that you do. The entire team behind ThingWorx. Thank you for being such a fun team to work with! Our robust platform—recognized by Gartner as a leader in industrial IoT platforms. Launching today, 9.1 features a cornucopia of new functionality, including: Runtime support for AWS Corretto Open JDK 11, now available to help you save Oracle Java licensing costs A new Project View in Composer Pareto charts, auto refresh, pagination and chip-based data filter web components Enhancements to auditing, Thing Group UI & export to source control Asset Advisor scale improvements Operator Advisor process plan caching Azure AD integration with Solution Central to manage users ARM templates now available to allow you to easily deploy the necessary ThingWorx infrastructure on Azure Read the release notes and download 9.1 from the Software Downloads Page today!   Stay connected & happy holidays, Kaya      

Now that ThingWorx 9.3 is live, let’s take a closer look at some of our new features we released for Composer and Mashup Builder.   Referenced By Find where entities, code references, and project dependencies are used in your existing projects using the new “Referenced by” report feature. This feature is not automatically enabled because it is intended to be used during development, since it can call upon all of the entities in your project and can impact your load times in production. That being said, this is your friendly reminder to turn off this feature during production.     How to enable: Go to the relationship subsystem and tick the check box to enable during development.   How it works: The “Referenced By” feature finds any entity that is referenced in your ThingWorx environment based on the supplied search characteristics. You can run this “Referenced By” report in the Composer or via a ThingWorx service “GetWhereUsed”.  When you use the “Referenced By” feature on an entity, you can find all of the references in the system for that entity in Mashup bindings, service script references, thing property bindings, and more.     Grid Widget The new grid widget component, which was available for preview in ThingWorx 9.2, is now complete, so it’s time to get your grid on! We have improved styling and performance capabilities with this new widget, including greater support for inline editing, autocreation of columns based on infotables, and adding new footer areas. You can also configure the grid using the property editor in Mashup Builder, where previously we had plain text entry.   Style Migration The new Style Migration is a game changer. It allows you to retain the same look and feel of your Mashups as you upgrade from previous versions of ThingWorx to the newest web components and features available in 9.3. Improved from our previous migrator, this allows you to move to the latest platform version and capabilities without having to re-implement or redesign your applications and widgets.     How it works: When you upgrade to the latest version of ThingWorx, you will see a pop-up window appear if you have any legacy widgets or layouts in your Mashup. The window will have the option for you to apply one of three style themes to your Mashup: PTC Convergence Theme (the new ThingWorx Default theme), Legacy Styles Theme (the old ThingWorx theme, from version 8.0 and earlier), or Custom Theme (choose from custom themes you defined using the Theme Editor and Style Theme that will appear when custom theme is selected in the pop-up). Depending on how you already styled or would like to style your Mashups, select an option and click migrate. This migrator maintains previous coloring, spacing, and other design properties better than previous migrators. You, of course, have the option to not upgrade your Mashup, but we recommend that you migrate, especially where we have new widgets available to replace legacy versions. If there are any issues with your migration, you can always click “Undo” in the toolbar.   Things to Consider: This migration will work best with ThingWorx default styling, out of the box styling, and Mashups with widgets that we now have replacements for (these are marked legacy in the builder). Always make sure you review your Mashups to make sure bindings and properties remained. Note that custom CSS will not be migrated, and custom widgets developed outside the standard platform installation will remain the same on the new Mashup.   Other Bonus Features That’s not all we rolled out in ThingWorx 9.3. You will also see Composer enhancements for test execution on ThingTemplates and dynamic use of Master Mashups to allow for swapping out Masters at run time based on predefined conditions based on your users. Plus, we now have truncation support for the breadcrumb component and tabs component, which utilizes an ellipsis pattern for long  text for a more user-friendly application.  With enhancements to our charts, you can now show/hide legends and format axis in new ways. We also support localization for our new web component widgets, .   How has your experience been building solutions with the latest updates to Composer and Mashup Builder? How can we continue to build upon these enhancements? Let us know what you think.   Stay connected, Rachel

Background Getting a performance benchmark of your running application is an important thing to do when deploying and scaling up an application in production.  This not only helps focus in on performance issues quickly, but also allows for safely planning for scaling up and resource sizing based on real concrete data.   I recently created a tool and made a post about capturing and analysing ThingWorx utilisation statistics to do such an analysis, as well as identifying potential performance bottlenecks. Although they are rich and precise, utilisation statistics fall short in a number of areas however - specifically being able to count and time specific service executions, as well as identifying and sorting based on the host executing the service.   Tomcat Access Log Analysis As ThingWorx is a Tomcat web application, Tomcat logs details of the requests being made to the application server and ThingWorx REST API.  The default settings include the host (IP address), date/timestamp, and request URI; which can be decoded to reveal relevant details like the calling entities and service executions.   Adding 3 key additional variables (%s %B %D) to the server.xml access log value also gives us the HTTP response code, service execution time, and bytes returned from Tomcat.  This is super useful as we can now determine exact time of service executions, and run statistics on their execution totals and execution time.     Once you have an access log file looking like the one above, you can attempt to load it into the access_log sheet in the analysis Excel workbook that I created.  You do this by click on the access_log table, then selecting "Data > Get Data > Data Source Settings".  You'll then be prompted with the following or similar pop-up allowing you to navigate to your access_log file to select and then load.     It should be noted that you'll have to Refresh the table after selecting the new access_log.txt file so that it is read in and populates the table.  You can do this by right-clicking on the table and saying Refresh, or using the Data > Refresh button.   This workbook relies on a number of formulas to slice and dice the timestamp, and during my attempts at importing I had significant issues with this due to some of the ways that Excel does things automatically without any manual options.  You really need to make sure that the timestamps are imported and converted correctly, or something in the workbook will likely not work as intended.  One thing that I had to do was to add 1 second to round up 00:00:00 for the first entries as this was being imported as a date without the time part, and then the next lines imported as a date/time.   Depending on how many lines your file is, you'll likely also have to "Fill Down" the formulas on the right side of the sheet which may be empty in the table after importing your new data set.  I had the best results by selecting the cells in question on the last row, then going down to the bottom corner, pushing and holding Shift, clicking on the last cell bottom right, and then selecting Home > Fill > Down to pull the formulas down from the top.   Once the data is loaded, you'll be able to start poking around.  The filters and sorting by the named columns is really helpful as you can start out by doing things like removing a particular host, sorting by longest execution times, selecting execution times greater than 4 seconds, or only showing activity aimed at a particular entity or service.     You really need to make sure that the imported data worked fine and looks perfect, as the next steps will totally break if not.  With the data loaded, you can now go to the Summary Data table and right-click on one of the tables and select Refresh.  This is reload the data in into the pivot table and re-run their calculations.   Once the refresh is complete, you should see the table summary like shown here; there are Day, Hour, and Minute expand/collapse buttons.  You should also see the Day, Hour, Month fields showing in the Field Definitions on the right.  This is the part that is painful -- if the dates are in the wrong format and Excel is unable to auto-detect everything in the same way, then you will not get these automatically created fields.     With the data reloaded, and Pivot Tables re-built, you should be able to go over to the Dashboard sheet to start looking at and analysing the graphs.  This one is showing the Top 10 services organised into hourly buckets with cumulated service execution times.     I'm not going to go into all of the workbooks features, but you can also individually select a set of key services that you want to have a look at together across both the execution count and execution time dimensions.     Next you can see the coordinated view of both total service execution time over number or service executions.  This is helpful for looking for patterns where a service may be executing longer but being triggered the same amount of times, compared to both being executed and taking more time.  I've created a YouTube video (see bottom) which goes through using all of the features as well as providing other pointers to using it.     Getting into a finer level of detail, this "bonus" sheet provides a Pivot Table and Pivot Chart which allows for exploring minimum, maximum and average execution time for a specific service.  Comparing this with the utilisation subsystem metrics taken during the same period now provide much deeper insight as we can pinpoint there the peaks were, how long they lasted, and where the slow executions were in relation to other services being executed at that time (example: identifying many queries/data processing occurring simultaneously).     Without further ado, you can download and play with my ThingWorx Tomcat Access Log Analysis Excel Workbook, and check out the recorded demonstration and explanation for more details on loading and analysis use. [YouTube] ThingWorx Tomcat Access Logs - Service Performance Analysis

  Hi, everyone!     Today, we’re launching an exciting new series called “PTC Community Spotlights.” Each post in the series explores a community member’s experience with ThingWorx—how they’re using it, what their favorite part about ThingWorx is, and any tips or tricks they may have to share with the PTC Community.   For the first installment, I spoke with @nmilleson of EAC. Check out our conversation below. Our first PTC Community Spotlight Speaker -- Nick Milleson of EAC Product Development Systems. @Kaya: Hi, @nmilleson, welcome! Thank you for taking the time to meet with me and volunteering to be our first ThingWorx Community spotlight!   @nmilleson: Of course, @Kaya. Happy to be here.   @Kaya: To start, can you tell me a little about yourself?   @nmilleson: Absolutely. My name is Nick Milleson.  I work as an IoT Solution Architect at EAC Product Development Systems (a PTC Partner). I’m located in Apple Valley, Minnesota, which is a suburb of the Twin Cities.   @Kaya: Nice! We always love hearing from our partners about the awesome work they do. As a PTC Partner, what industries do you typically work in?   @nmilleson: I consult for many, many different industries, including defense, transportation, medical devices, construction & aerospace.   @Kaya: Wow, so what PTC products are you most familiar with?   @nmilleson:  My schooling is in mechanical engineering, so I’ve also used Creo, Windchill, and MathCAD.  I have been working with the ThingWorx application and helping clients get the most out of ThingWorx for approximately 7 years.   @Kaya:  Seven years—that’s a while! Do you have any “ThingWorx” stories from over the years you can share with your community peers?   @nmilleson:  Sure thing. I think the coolest thing that I’ve done with ThingWorx was create a custom SVG infographic that featured animations, click events, zoom-ins, and heatmaps based on temperature deltas.  It was a custom widget and it worked really well in ThingWorx.  When I first started learning to use ThingWorx, I took apart an old RC car and hooked up an Arduino to the motors and steering.  I was then able to control it using a ThingWorx mashup.  Pretty fun! I’ll be sure to share a visual so people can check it out.   Nick's awesome custom SVG infographic featuring a ton of neat functionality like zoom-ins & heatmaps. @Kaya:  That’s awesome! Sounds like a fun time indeed. I saw that one of your first publications about ThingWorx for EAC was from 2015 and titled “Updating ThingWorx Using an Arduino Uno and a Serial Connection.”  The ThingWorx platform has certainly evolved since then.  What would you say is your favorite thing about ThingWorx today?   @nmilleson:  It sure has evolved. I would say my favorite thing is that it’s flexible enough to allow you the freedom to design all sorts of applications, while also providing you with all these great tools that make it easy to use as well.   @Kaya:  Thanks for that. I can see that you have been a member of the PTC Community for five years.  Thank you for providing such great contributions.  What do you enjoy most about the PTC Community?   @nmilleson:  I enjoy this Community because everyone seems very willing to help each other out, regardless of the complexity of the issue.  I stick mostly with the IoT Developers section, but I’ll meander into the Manufacturing Apps and ThingWorx Ideas once in a while as well.   @Kaya:  Love to hear it. Now, so the PTC Community can learn a little more about you, how do you spend your time when you aren’t playing with ThingWorx or engaging on the PTC Community?   @nmilleson: Great question. I have been a professional piano player for almost 20 years, so I’m often at a piano bar making music when I’m not doing software development with EAC.   @Kaya: Awesome. Well those are all the questions I have for today. Thank you for sharing your experience with ThingWorx! Truly appreciate it.   @nmilleson: Of course. Happy to be a part of it!   Kaya, here. We love hearing from community members like @nmilleson about how ThingWorx creates value for them amongst a variety of use cases. If you’re active on the community and interested in being featured on the PTC Community Spotlight series, send me a direct message and we’ll get the ball rollin’.   For now, we’ll let Nick “play us” out. Until next time, stay connected!   -Kaya

  Hi everyone,   In anticipation of ThingWorx 9.0’s biggest feature, active-active clustering, we’d like to provide an architectural overview of a sample active-active configuration and its underlying components. If you haven’t already seen it, we invite you to read our previous Community tech tip, where we introduce the concept of active-active clustering for ThingWorx Foundation, which enables you to: significantly reduce unplanned downtime for your mission-critical services and apps support horizontal scalability of the ThingWorx Server where you can scale your services up and down based on your requirements easily run, package, deploy, and operate advanced apps and services with the help of intuitive browser-based navigation, interactive monitoring and debugging tools, and more deploy anywhere - public cloud, private datacenter, on-premise, hybrid, or even locally on your laptop with deep optimizations for Azure Now, before we go too deep, we’d like to let you know that you can continue to seamlessly upgrade from previous versions of ThingWorx releases to upcoming ThingWorx 9.0  releases. Previously, you could deploy ThingWorx Foundation in a “single server” mode, and, for a high availability in “active-passive cluster” mode (see here for details). In the ThingWorx 9.0 release and onwards, you’ll be able to continue to deploy ThingWorx Foundation in a “single server” mode and for high availability scenarios via our new “active-active cluster” mode. Please note that active-passive clustering configuration will no longer be supported in ThingWorx 9.0 or onwards.   Let’s start with a quick recap on how the ThingWorx Foundation 9.0 release would look like in single server deployment.   ThingWorx 9.0 Deployed in a "Single-Server" Architecture Below is a high-level diagram depicting the main architectural layers and components of ThingWorx Foundation deployed in a single server mode.   Deployment Components Below is a brief summary of all major architectural components and their purpose in the deployment architecture:   The Client Layer This layer is comprised of everything that connects with, sends data to, and receives content from the ThingWorx platform. It be broken down into two groups: Devices/Things: Things, devices, agents, and other assets. Users/Clients: People and the respective products (primarily web browsers) they use to access ThingWorx.   The Application Layer This layer is where ThingWorx Foundation and other applications deployed with ThingWorx Foundation reside, such as ThingWorx Analytics, ThingWorx Connection Server, ThingWorx Azure IoT Hub Connector and others. This layer provides connectivity to the client layer, performs authentication and authorization checks, ingests/processes/analyzes content, and reacts to conditions by sending alerts. For a specific ThingWorx Foundation deployment that needs basic device data ingestion, processing and storage, you can setup only ThingWorx Foundation server. In some cases, with large number of device connections, you may want to setup ThingWorx Connection Server with ThingWorx Foundation in a single server for further scalable connectivity. ThingWorx Foundation: ThingWorx Foundation is a java-based application that serves as a rapid, model-based application development platform. Shared File Storage: Shared Disk space to contain ThingWorx Storage repositories, store and archive log files accessed by all ThingWorx Foundation servers. A NAS file storage, AWS Elastic File System, or Azure Files and others could be used for this purpose.   The Data Layer ThingWorx Foundation includes several persistence provider implementations that enable you to choose a database option that best fits your use case. A persistence provider enables the connection to a data store and the ability to perform a CRUD operation on that data. See here for more information. Currently, there are two basic variations of persistence providers: Model Provider – Responsible for ThingWorx model metadata and system data. Data Provider – Responsible for runtime data ingested against the model elements, including streams, value streams, data tables, etc. ThingWorx supports H2 (in-memory Database), PostgreSQL, MS SQL Server and AzureSQL as both model and data providers, and InfluxDB as only a data provider. Please see here for model and data best practices.   ThingWorx 9.0 Deployed in an Active-Active Clustering Reference Architecture Below is a reference architecture diagram for ThingWorx 9.0 with multiple ThingWorx Foundation servers configured in an active-active cluster deployment. Please note that this is only one reference example of how ThingWorx 9.0 can be deployed in an active-active clustered environment. There could be other architectural configurations dependent upon the needs of the specific deployment. Deployment Components Once you have developed an understanding of the basic architectural components in a single server mode, below are the additional components required to run ThingWorx in active-active cluster mode.   The Client Layer This will be similar to what has been mentioned in the above single server configuration.   The Application Layer In this layer, if you’re familiar with ThingWorx active-passive cluster configuration, then you may be aware of most of the components used below—with the exception of a new component: Apache Ignite that provides Distributed Caching for the horizontally scalable ThingWorx Foundation servers. Load Balancer: A third-party device that receives network traffic and distributes requests among available servers. In active-active cluster configuration, the load balancer is used to direct WebSocket-based traffic to the ThingWorx Connection Servers while user requests (http/https) traffic is directly distributed to the ThingWorx Foundation servers. Users can continue to use a load balancer with ThingWorx 9.0 that they might already be using for their existing active-passive or single server deployments with ThingWorx 8.X or previous releases.  Some example load balancers include, but are not limited to: HAProxy, Azure Application Gateway, and AWS Application Load Balancer. ThingWorx Connection Services: These services handle all messages routing to and from devices, providing scalable connectivity to the ThingWorx Foundation Server. With the ThingWorx 9.0 release, ThingWorx Connection Services have been upgraded with many additional features to support active-active clustering of the ThingWorx Foundation servers, where now they route all WebSocket traffic in a round robin fashion to the connected ThingWorx Foundation servers. Depending upon the various use cases, one could use multiple ThingWorx Connection Services available, such as ThingWorx Connection Server, ThingWorx Azure IoT Hub Connector, and ThingWorx Protocol Adapter Toolkit. Please see here for further details. Please note that for ThingWorx 9.0 releases, ThingWorx Connection Server would be required in an active-active configuration to support all the WebSocket-based traffic routing, including egress of files and device messages from multiple ThingWorx Foundation servers back to the devices, and it would also serve as WebSocket communication from ThingWorx Mashup-based applications to ThingWorx Composer. ThingWorx Foundation: With ThingWorx 9.0 and onwards, you can set up ThingWorx Foundation servers in an N-active-active cluster model to provide higher availability to your applications and horizontally scale the Foundation server nodes up and down based on your scalability needs. Apache Zookeeper: Apache ZooKeeper is a centralized service for maintaining configuration information and naming as well as providing distributed synchronization and group services. It is a coordination service for distributed applications that enables synchronization across a cluster. Specific to ThingWorx, ZooKeeper is used for distributed locking, selecting a singleton server during the server initialization, service discovery for Apache Ignite, allowing it to find instances of ThingWorx Foundation servers. Apache Ignite: This offers a distributed cache for the active-active cluster setup. It is used by ThingWorx Foundation Servers to share state. It may be embedded with each ThingWorx instance or can be run as a standalone cluster for larger scale. In this configuration, Ignite is set up in a standalone cluster but can be run embedded within the ThingWorx Foundation. Running Ignite in a standalone cluster is more ideal for larger scale, as it supports higher vertical scale of memory in the deployment setup.   The Data Layer ThingWorx is largely database agnostic. You can continue to use officially supported persistence providers that you may already be using in your existing deployments based off of ThingWorx 8.X or previous releases. Please look out for an upcoming ThingWorx update as well as enhanced installation documents to help with your upgrade and migration questions with the general availability of ThingWorx 9.0. Please note that this diagram does not make the distinction between model and data providers; depending on your data ingestion needs, separate model and data providers can be used. As a reminder, all databases should be deployed in a high-availability configuration to help eliminate any single point of failure.   In closing, we can't wait to launch active-active clustering in 9.0 to help you: dramatically further reduce application downtime scale your deployments and more efficiently manage your apps, regardless of where they’re deployed   If you have any questions about active-active clustering or its architecture, please do not hesitate to reach out!   Stay connected! Kaya

  Question: What are some best practices around building IIoT solutions with ThingWorx?   Meet Ward. Ward works on the product management team for our Manufacturing Apps (i.e. Asset Advisor, Operator Advisor, Production Advisor, etc.). He’s a super cool and smart guy, and he always has an answer to my ThingWorx questions. He has so many answers, in fact, that he worked closely with other ThingWorx experts like Sangeeta to create the ThingWorx Application Development Guide.   I sat down with him to hear his top few tips from the guide. And, just in case we don’t have enough fun around here on “Ask Kaya,” we decided to list his top tips not by “1”-“2”-“3”, but by “W”-“A”-“R”-“D”.   Without further to do, here are Ward’s top tips from the ThingWorx Application Development Guide.   Whitelist your IPs for application keys. (See page 67.) Auto Refresh widget vs. GetProperties service? How should I update live data to my mashup? (See page 25.) Reuse components to increase efficiency and improve your application design. (See page 69.) Don’t use a Thing Template when you really should use a Thing Shape. (See page 10.)   To see more, check out the full ThingWorx Application Development Guide here!   Look out for our next release of the App Development Guide in July! It’ll feature our Manufacturing Apps to share even more ThingWorx best practices!   Reach out with any questions and stay connected! - Kaya

  If you’ve ever wished you could see into the future, you’ve come to the right place! Put your reflective suits and sunglasses on to prepare for a glimpse into the future of our upcoming ThingWorx 8.4 release! Here are sneak peeks of the top three features you may not have known are coming in ThingWorx 8.4.   1. Thing Presence While it sounds like something from an episode of Ghost Hunters, Thing Presence provides insight into the communication state of polling or duty cycle Things (those that check in and out on a periodic basis). We’re introducing a new IsReporting state, which would be set to true when polling assets check in on time and are considered “present in the network.” This helps to bridge the gap where the traditional ThingWorx IsConnected state reports offline and does not coincide with the actual network presence of the device.   Thing Presence: New "IsReporting" State2. Data Helpers You may not know what Data Helpers are, but if you’re a longstanding ThingWorx developer you likely know about Expression and Validator widgets. These widgets were handy because they allowed you to write conditional logic or input validation to drive behaviors in the UI, but were super frustrating to use. They took up lots of room on the visual layout canvas and only had a very little textbox to edit them. In the 8.4 release, we are happy to announce that these two widgets will no longer be placed on the layout canvas. Instead, they will have a dedicated editor to work from with plenty of room for code development, parameter configuration and event definition and binding. We’re wrapping all of this functionality into a nice little feature called…Data Helpers. Data Helpers: Expression and Validator Widgets No Longer in Layout Canvas3. ThingWorx Flow In case Thing Presence and Data Helpers aren’t exciting enough, we’re also introducing ThingWorx Flow, a neat new feature set that dramatically speeds development of connected applications through integrations with business systems like Salesforce and SAP. Imagine that, when a certain alert triggers, you want to automatically create a Salesforce service ticket and even send an emergency text to an operator to prevent damage to a device. A large set of out-of-the-box system connectors (PTC Windchill, Office 365, Google Docs, Slack, Jira and more) are included, which you can drag and drop onto a canvas to visually define a workflow. In the example below, a ThingWorx-connected device element, a Salesforce “create case” action and a Twilio text message connector were dropped into the canvas to create a visual workflow. Orchestration: Example Workflow that Creates Salesforce Cases and Alerts OperatorsThing Presence, Data Helpers & Flow—get ready for these and more in ThingWorx 8.4!   Stay tuned for future posts that go into greater depth about each of these features and comment your thoughts below!   Stay connected, Kaya

Disclaimer: Please note that, while the ThingWorx Git Backup Extension is a very useful tool, it is not a PTC product, nor is it supported by PTC.   Hi ThingWorx users,   Trying to manage your ThingWorx application artifacts in a CI process? Wondering who changed that line of code in your Thing Service? Trying to see what your Mashup looked like last release? Time to Git excited! Introducing the Git Backup Extension, an open-source tool available here to offer a stronger integration with the Git source repository. This Git feature can push or pull code and artifacts (like entities, data exports or extension dependencies) to your Git repository.   Here are some highlights of how this works within ThingWorx:   First, configure your Git repo to work with ThingWorx by creating a Git Backup Thing. Then, simply open your new Thing, navigate to the Configuration editor and enter information like your Git URL, your Git username and password, your repo and branch names, etc. See example below. Configuring your Git repoWith this configuration in place, you can now use the Home Mashup of this new Git thing to browse the repository and pull down contents to your local ThingWorx instance. For new projects, you can also push new entities to the repo as you work on your application.   As you and your team are working, you’ll want to see the differences of the files you are editing and working on collaboratively. The Git extension feature makes this easy. Just like you can see diffs clearly delineated for a file with your Git client, you can see the same with this Git integration in ThingWorx. Similar to the git status command, the Git ThingWorx extension will show you the list of files you have changed that are available to push, as well as their diffs. See an example below. Checking the Git status While working, if you want to switch branches or pull down a new project, you can check out a specific version and see all commits available on that branch (see below). Checking out a specific commit Want to learn more or try it for yourself? Find the open-source Git Backup Extension here and check out the Git Backup Extension User Guide for guidance.   Stay connected, Kaya   P.S. What do you think? Comment your thoughts below!

Hi, ThingWorx users! We’re excited to share that we have partnered with InfluxData to make time series analysis in ThingWorx even easier. InfluxDB is a database by InfluxData that is “built specifically for metrics and events that empower developers to build next-generation IIoT, analytics and monitoring applications.”   Why InfluxData?  Today, application developers expect robust querying capabilities, fast response time, easy ways of aggregating and pivoting on data and leveraging results for reporting and visualization. IT and devops administrators also expect cost-effective storage and easy ways of aging data through archiving and the ability to keep large amounts of historical data to satisfy analysis requirements.   That’s why we’ve partnered with InfluxData to make it easier for developers to store, analyze and act on IIoT data in real-time. With InfluxData, developers can build connected IIoT applications more quickly while still incorporating the following capabilities: monitoring real-time alerting predictive maintenance streaming data anomaly and event detection visual and report-based analysis   We considered a few technologies for the purpose of improving ThingWorx time series analysis. Here are a few reasons we chose InfluxData: high compression of data ~45x ability to handle millions of writes per second* ability to read around thousands of rows in milliseconds* supports the standard time series functions of sampling, interpolation, time bucketing, aggregation, selector, transformation, predictor, etc.  * Query and write times will vary based on an individual ThingWorx application’s implementation with Influx. For example, as the number of concurrent reads increases, the query speed decreases. With the upcoming 8.4 release, the ThingWorx Sizing Guide will be updated to reflect representative performance for ThingWorx developers.   In addition to improved query capability, ThingWorx time series with Influx can now use less memory and CPU, giving your platform servers a bit of a break.   To start strategizing on how InfluxData can help you in your ThingWorx journey, here is a sneak preview of what it will look like:   New Features The new ThingWorx Influx Persistence Provider will make query services like ValueStream Thing QueryPropertyHistory, Stream Thing QueryStreamData and QueryStreamEntries even better. Simply create a new instance of the persistence provider, configure it to use your InfluxDB instance, create a new value stream (or stream) from the new persistence provider, and you’ll be writing, reading and analyzing your time series data like never before.   We’re also introducing a new enhancement to improve InfoTable support with time series data, including providing the ability to use a driver property. The driver property can be specified with the QueryPropertyHistoryWithDriverProperty service for time alignment and filling backward/forward in your stream queries.   Let’s walk through a driver property example where you have the properties of temperature, speed and battery level. Timestamp Temperature Speed Battery Level 1480589076592000000 80.003 5012 79 1480589077537000000 80.010 5011 79 1480589077550000000 80.010 5009 79 1480589077562000000 80.030 5011 78   Let’s say temperature is the key driver for your analysis. In other words, you are not concerned if speed or battery level changes—you only care about when temperature changes. We can specify temperature to be the driver property for that particular time and only return stream values for temperature, speed and battery level when temperature changes. If speed or battery level changes (but temperature does not change), the rows associated with those changes would not be included in the results set because neither speed nor battery level is a driver property. See chart below.  Timestamp Temperature (driver) Speed Battery Level 1480589076592000000 80.003 5012 79 1480589077537000000 80.010 5011 79 1480589077562000000 80.030 5011 78  Note that only three of the four rows are returned above because one entry in the original table did not have a change in temperature.    Stay Tuned Look out for these time series improvements and InfluxData integration in our upcoming 8.4 release. I’ll be sure to keep you updated on additional new features coming in our next release (like Orchestration and Mashup Builder 2.0), so check back shortly or subscribe to this Community so we can stay in touch. As always, if you have any questions, just ask Kaya!   Stay connected, Kaya

Fresh look at getting started with ThingWorx in a relevant context that outlines the DEVOPS needed to kick-start your programming.     For full-sized viewing, click on the YouTube link in the player controls. Visit the Online Success Guide to access our Expert Session videos at any time as well as additional information about ThingWorx training and services.

Is your team operating an effective DevOps pipeline? DevOps is an important part of a mature, enterprise ready application, but the process isn’t simple.   This expert session will focus on how a full DevOps pipeline looks like and how PTC can help to build a seamless pipeline. Join us for our upcoming Expert Session to learn how to create a Docker image, integrate Azure with Docker and Git, and set up a seamless DevOps pipeline.   When? Thursday, September 30th 2021 | 11 AM EST Host: Tori FIrewind, Senior Engineer in PTC IOT Enterprise Deployment Center Registration link: https://www.ptc.com/en/resources/iiot/webcast/devops-pipeline-thingworx 

  Hello everyone,   If you’re like me, you’re always looking for the optimal or most efficient way to do something. Today, I’ll share a quick trick and two tips to help you develop your awesome IoT solutions with ThingWorx.   #1. Trick: Finding Dependency References We are targeting a new “Where Used” Composer feature in an upcoming release of the platform to help you find your references of bindings, properties, mashups, and services. In the meantime, did you know you can get some of that information yourself today with a quick service call?   As of ThingWorx 8.5, a new service is present on Project entities; the service crawls the contents of your project and highlights the full external dependency list to help you find references. On any Project Entity, ListExternalDependencies() shows output like this in 9.0:  ListExternalDependencies() output   For each entity (“A”) in the project, the service calls out any entities (“B”) that it is referencing and the referenced dependency’s extension package if present. It will only find external dependencies to the project and will not currently list dependencies within the project. Notice also in the infotable output, the last column, “where used,” even lists the type of reference (e.g. coded in JavaScript, Mashup Data, Resource, Property binding, etc.). Pretty handy!   Code reference from “Where Used” service output   Click this link for additional help content that explains the service output and usage. Again, it only searches for entity references outside of your current project scope. Also, this service will stop crawling the dependency hierarchy when it finds items in a project, since its current purpose is packaging.  Consider if you have Thing T1 in Project P1, which uses ThingTemplate TT2 and it’s not in a Project. TT2, in turn, uses ThingShape TS3 which is also not in a Project.  Calling ListExternalDependencies()  on Project P1 will find both TT2 and TS3. If, however, we then put TT2 in a Project P2, then call the List() service on Project P1, the scan will stop at TT2 and NOT identify TS3.  The reason for this is that the service assumes that when you package P2, it will find the orphan TS3.     We know this doesn’t cover all “where used” type use cases, so there is still a planned feature to really complete this concept on the platform. But even in the 8.5 or 9.0 releases, if you wanted to see entity references (inside and outside of its project) for a single Thing A, you could quickly assign Thing A to a new project and run the ListExternalDependencies() service to find all of its references and then assign Thing A back to its original project once you’ve found what you are looking for. Moving entities into projects just for searching is not something I would recommend doing often, but it can work in a pinch!   #2. Tip: JavaScript looping When iterating through data from infotables, use a .forEach() loop! Consider these four code options and their average performance on the Rhino engine:  Infotable looping performance   Very clearly, the .forEach() syntax is the most performant and, in my opinion, the cleanest to read. Try it out in your app! We plan to update our help documentation with more of these ThingWorx JavaScript best practices in 9.1. We also plan to provide some updates to our Code Snippets features in an upcoming Composer release so we can recommend these good practices right from the start.   #3. Tip: Code optimizations As with many performance bottlenecks, it is those pesky loops that can really amplify degradation. Here are two ThingWorx patterns for your consideration:   Wrong Way:   In this block of code, we setup the property names we are looking for, and then loop through to make a logger message. While creating each logger message, we are making an API call for querying all things for a Thing named me.name and executing a service call GetMetadataAsJSON() on that Thing which walks the hierarchy to build a JSON representation of itself. In this trivial example, we are making these same API 2 calls for each item in the propertyNames list, though the Thing reference and JSON definitions are never changing. Pretty expensive.   Correct Way:   Notice in this example, we are not only declaring the propertyNames outside of the loop, but also the propertyDefinitions. This will significantly improve performance and reduce the number of API calls and round trips to the application server. Again, this is a trivial example, but can pay off in larger and more complex code areas.   If you like these quick tips, check out more best practices here! Got a tip of your own? Have a question on how to tackle something? As always, just Ask Kaya!   Stay connected! Kaya

Hi all, Here is the recording of the expert session hosted in September 3rd. For full-sized viewing, click on the YouTube link in the player controls Your feedback is very important to us! After watching the recording, please take 2 min to complete this survey  

  Hello, IIoT Developers!   Today, I’m going to provide you an overview of a new SDK we’re offering for developers to build custom web content. It’s called our Visual SDK. We released 9.0 in June, so it’s time to start getting excited! In case you missed it, check out these other 9.0 posts on active-active clustering, Composer and Mashup Builder, and the 9.0 release overall.   In 8.4, we introduced a new visualization architecture and set of web components based on Polymer. We’ve continually updated and added new widgets and features with this architecture each release, including new Chart Components in 9.0. This SDK was previously available mostly as a style guidance, informing designers and developers what elements and behaviors were available in the new PTC web component-based widgets. You could use this SDK to style in CSS custom elements of the web components in a mashup-based application. You weren’t, in 8.4, really able to use this SDK yet to actually create your own Polymer web components and have them be as robust in the Mashup Builder in ThingWorx.   In 9.0, this SDK has been expanded so you not only have style and behavioral descriptions of PTC components, but you also have tutorials and utilities that let you create your own components and import them into the Mashup Builder. The possibilities are endless here for custom content, so let’s look at what’s inside.   The SDK guide has a quick outline of some pre-requisites you should know about as you enter custom web development with ThingWorx. Things like knowledge of Polymer 3, downloading common tools like NPM and Gulp CLI, Aurelia, etc. Much of this info is also included in markdown documents within the SDK files, but the SDK web content makes it easier to follow and search.   From there, the guide walks you through more setup of your SDK directories, NPM install of PTC components, and basics around dependencies, styling, and demo pages. Each PTC-developed web component is also available in the SDK pages as well with more information on what they offer and their basic designs. This is useful if you would like to reference the PTC components as imports into your own web component. This technique is very useful for re-use and upgrade safety when developing custom components on top of ThingWorx. Sample overview pages in the SDK for ptcs-chart The SDK also includes a getting started tutorial and a sample Polymer component and a widget called simple-el , which are helpful as you want to reference during development and familiarizing yourself with concepts. The component is functional and offers a theme dropdown so you can see how the theming engine and events work. Sample Polymer component included in the SDK called simple-elOnce you have created your web component, there is also a new utility called mub, which scans your component project and wraps it in a shell for the Mashup Builder. If you run the mub utility on your component, you’ll find it produces a zip file with the relevant design and runtime wrappers for the mashup environment already mapped to your component. You can also use it to define properties for your new component in the mashup environment, include custom code for defining the widgets at design and runtime behaviors, and to add icons, categories and other standard platform features. Running the mub utility on a web component project Once you have run the utility, you just import the artifact into a ThingWorx platform and it will be available for your application developers to use in their mashups as a widget. Again, how it appears in the design experience, what properties are exposed, how it responds to platform binding and theming events are all customizable in the SDK. Sample Polymer Component wrapped as a widget for use in a Mashup Once you get the hang of things with the sample code and understand the ins and outs, you can then use those same patterns to develop your own content! These are the same techniques that the PTC R&D team uses when they make each of the new widgets that you see in our product, like the 9.0 charts! Uber cool stuff!   Like what you see? Have a question? Drop us a line in the comments!   Stay connected! Kaya