IoT & Connectivity Tips

  The IoT Enterprise Deployment Center’s goal is to create and share knowledge around the best practices for architecting, designing, and deploying successful, enterprise-scale Thingworx IoT Solutions.    To accomplish this goal, the EDC team takes a “real world” approach, using simulated IoT assets and users to benchmark the capabilities of different Thingworx deployment configurations. First, each implementation is pushed to its limit in an effort to establish real-world baselines, metrics which can be used to help customers determine which architecture choices will work for their custom needs. Then, each implementation is pushed beyond its limits, providing useful insight into where and why things fail, and illuminating potential implementation changes which could push the boundaries further.   Through the simulations testing to come, the EDC will be publishing the resulting benchmarks for all to see! These benchmarks will include details on implementation goals and performance metrics for different stages of deployment. Additionally, best-practice articles which illustrate how to deploy the different architectural components (those referenced within the benchmarks) will also be posted, highlighting the optimal approach to integrating everything into the Thingworx platform.   Stay tuned to see more about just how versatile the ThingWorx Platform can be! We look forward to discussing these findings as they are published right here on the PTC Community. 

In the summer heat, keep your operators cool with Operator Advisor. Sit by the pool and relax to the tunes of Episode 05 of “ThingWorx on Air.”     High five! We’re back with Episode 05 of “ThingWorx on Air,” our developer-focused IoT podcast.   In today’s episode Jordan Chaisson, a super talented product manager, joins me to share even more about Operator Advisor (OA). You may remember that we introduced OA in our very first episode of “ThingWorx on Air”. Today, we dive deeper into its business value and reveal what’s on its roadmap. Plus, hear the coolest use case she’s seen yet with Operator Advisor!   OA is an accelerator application built on the ThingWorx platform that enables manufacturing operators through digital work instructions and a comprehensive user experience to receive the right data at the right time to minimize scrap and maximize efficiency.   Looking for more? Check out the Operator Advisor Guide or discover where to download Operator Advisor today.   Reach out with any questions, and, as always, stay connected!   -  Kaya

  Energy. Innovation. Passion. That’s how I would describe LiveWorx19 in three words.   From beginning to end, LiveWorx truly was a one-of-a-kind digital transformation event. Whether it was the top-notch demos, the delicious lunch from food trucks, or the unique integration of our products with our partners’ technologies, everywhere I turned, inspiration and excitement were booming.   Here are my top three highlights:   1. Delivering a breakout session Chris Baldwin and I shared exciting new functionality that we’re releasing in ThingWorx to simplify and accelerate deployment of applications across your enterprise. We introduced Solution Central, a new portal in the cloud for application management and deployment (more info to come in a future post, but in the meantime, reach out with any questions).   2. Assisting with Xtropolis, our Demo Floor Speaking directly with you and hearing how you’re using ThingWorx to transform industries like medical devices, beer brewing or furniture manufacturing was pretty impressive. It was also quite rewarding and inspiring to hear your clever questions about your unique IoT use cases while being surrounded by folks performing service on a giant John Deere tractor, attendees having virtual rides on a Polaris vehicle and a team of brewers using ThingWorx and Vuforia to brew some tasty Trillium beer.   3. Participating in Usability Tests Thank you to everyone who participated in the UX Lab. Hearing your direct feedback in usability tests on what you like and what you want to see changed or added to the product helps make it feel like you are right there with us designing ThingWorx—because, well, you are. We take the feedback you share at LiveWorx right back to the office and begin iterating. So, quick shoutout to everyone who participated—thanks for your feedback.     If you weren’t able to attend LiveWorx, check out our CEO, Jim Heppelmann, deliver the opening keynote above and hear him discuss how PTC is interweaving AR, IoT, generative design, PLM and AI to create a digital thread that transforms industrial enterprises across the globe.     After hearing about our overall strategy, you’re looking for more details on our product roadmap, watch the above roadmap session delivered by our EVP of Products, Kathleen Mitford.   One of the most rewarding parts of the show for me was meeting some of you—ThingWorx developers and Ask Kaya readers—in person!   Stay connected, Kaya   P.S. If you want to hear what others thought of LiveWorx, check out what Forbes, Automation World or Diginomica had to say.   P.P.S. If you had as much fun as I did or you’re looking to attend for the first time, tickets for LiveWorx 2020 are already available—check them out here!

  DevOps. It’s not just a buzzword. It’s a true development methodology that can make all the difference in your application quality and release time. Today, I’ll walk you through how you can continuously integrate and deploy your ThingWorx applications to achieve CI/CD objectives as part of a DevOps-focused culture. At the end, I’ll provide you a sneak peek of what you can expect in a future release (hint: we’re working on some awesome new CI/CD functionality). Overview of ThingWorx DevOps and Common Tools I’ll start by providing an overview of the DevOps cycle, and then I’ll provide more details around each step of the cycle. Before we can start, you’ll need to define your high-level architecture and functional requirements as part of the “Plan” phase.   Now, let’s build your ThingWorx app. Ready? Here we go!   Code As with any software platform, developers can start working in any number of areas of the IoT application—from edge, to visualization, rules authoring to data modelling. For the purpose of this article, we’ll start with the UI, but much of the same steps can be applied in any order. Also, we’ll just call out high level steps of development, but for more info on building out each aspect of your application, please visit developer.thingworx.com.   In ThingWorx Composer, build out your user interface with Mashups. Starting with UI can help you think about the types of data you want to collect from devices and systems and how you want to solve your unique requirements for the business. Starting at this point can also help you show live POCs and functional mockups to stakeholders. Once you’ve built some starter screens and a skeleton of app navigation, you can start adding in data through configuration in Composer by creating your Things, Templates, properties and services. [Optional] We offer 65 out-of-the-box widgets for the UI in the ThingWorx platform. There are times when you have specific visualization requirements for your application and the out-of-the-box widgets don’t quite satisfy them. We have a path for that, through our custom widget extensions. If you choose to develop your own widget extensions, you can do so through other IDEs like Eclipse or WebStorm. Custom development and extensions are not just for UI. We also allow you to define Thing entities and their custom services in Java. If you are developing extensions in this way, we’d recommend you do so using Eclipse to code and Gradle to build and drive tests. For instructions on how to create your own extension, see “Creating Customized ThingWorx Widgets” on page 42 of the ThingWorx Application Development Guide posted on Ask Kaya. With a good start on the data model, business logic and UI, some quick testing and validation is in order. You’ll probably also want to save all of this work also to share with colleagues or move to other integration environments. Capture all of your entity and code artifacts (Mashups, style definitions, Thing shapes, Thing templates, JavaScript, etc.) by using the “Export to Source Control” feature from ThingWorx Composer to write entities to the file system. You can use Git or other source systems to monitor the file system and push to the remote repository of your choice (e.g. GitHub, Bitbucket, etc.). Again, if you are developing extensions outside of Composer, you’ll want to source control those items, too, from Eclipse or the file system directly. Build [Optional] You can build an application package as an extension with all entities and code from Eclipse using the ThingWorx Eclipse plugin. When you build the project, it will create an extension zip file. Again, more info in the Application Development Guide. Make your life easy by using tools like Gradle or Maven. ThingWorx is very similar to other Java development systems, so Gradle and Maven track your dependencies and create a package with all of the referenced extensions you may be using and put them into one single zip file package. Once you have a package built, you can import it into test or integration environments. For added automation, create repeatable tasks like a job in Jenkins so that every time your code is changed in the source repository (e.g. Git), it triggers a job to increment the version, build the project and create the package deliverables. Consider also configuring the Jenkins jobs to push artifacts to a central repository like Artifactory. Test Once your code has been built, we can’t forget about testing! Automation is king for DevOps! For ThingWorx apps, you should still design a test strategy for your application, and then define and create your tests. These can run in your local developer environment, as well as be triggered via build tasks/changes in the source repository. Tools like JUnit for your entities and Java-backed services or Selenium for testing the Mashup UIs can be used. You can create separate jobs in Jenkins along with the build to run the integration and unit tests against an instance of ThingWorx that has the latest artifacts deployed into it. You can also do static code analysis using tools like PMD to find bugs, check style issues or identify inefficient code paths. To round out your app also with performance and load testing, JMeter is one tool that you can leverage. Release Releasing is the culmination of the team’s great work! If the test results pass and the builds are green, you are good to go, and it’s time to establish your release build. Make sure that you consider a versioning scheme for your application and its artifacts. Semantic versioning is a pattern that can be implemented for your ThingWorx application. Correct versioning of ThingWorx packages affects your upgrade plans and is a signal to your users on the intent and content of the release. Again, see the Application Development Guide. Once a release milestone is met, you can create a source branch in Git for that milestone, which will have all the changes encompassed in that release. Configure a Jenkins job to create builds from that milestone branch for maintenance purposes. Deploy + Operate + Monitor   If you’ve tested and released your application, it’s time for production and real users! Using the build and testing infrastructure you’ve set up earlier in the development process, you can also deploy your release builds to your target staging and production ThingWorx environments with Jenkins jobs, Artifactory and automated steps. Finally, as with anything, it is important to measure success and monitor performance via KPIs, trends and logs. You can also extract application insights and recommendations from the PTC System Monitor (PSM) tool, which uses Dynatrace; here is a guide on how to install and deploy PSM.  There are many different paths through the platform and options for developers to match your local team processes and tools—this was simply a quick overview. Congrats! You’re now equipped to build ThingWorx apps while leveraging software best practices and incorporating a DevOps culture!   What can I expect in a future release of ThingWorx? Coming in a near-term release of ThingWorx, we’ll make it easier for you to continuously integrate and deploy your ThingWorx applications. How? Through new functionality that bolsters our packaging concepts, new cloud services to assist in deployment to environments and an error-proof way to integrate applications with an automated dependency awareness.   Stay tuned for more info about this exciting new deployment and application management functionality targeted for Fall 2019!   Reach out with any questions and stay connected.   -Kaya

  We are counting down the days for you—developers, technologists, futurists—to witness the unparalleled power of PTC’s technology. Hosted by PTC, LiveWorx is the world’s leading digital transformation event to equip you with the knowledge, power and tools you need to begin or accelerate your company’s digital transformation.   I’m excited to share that I will be presenting a breakout session on June 11 th , at 1:15pm EST, around a brand-new functionality we’re offering to improve your ability to manage and deploy your ThingWorx applications.   Want to learn more? Attend LiveWorx 2019 or learn about our livestream options. However you choose to attend, it’s an event that I’m pretty amped for and I can’t wait for you to be, too.   Hope to see you there—it’d be great to meet you in person!   Stay connected, Kaya  

Reminder (and for some, announcement!) that the new ThingWorx 8 sizing guide is available here  https://www.ptc.com/en/support/refdoc/ThingWorx_Platform/8.0/ThingWorx_Platform_8_x_Sizing_Guide

Since the marketplace extension is no longer supported and the drivers may be outdated, you may build your own jdbc package/extension: Download the Extension Metadata file Here Download the appropriate JDBC driver Build the extension structure by creating the directory lib/common Place the JAR file in this directory location: lib/common/<JDBC driver jar file> Modify the name attribute of the ExtensionPackage entity in the metadata.xml file as needed Point the file attribute of the FileResource entity to the name of the JDBC JAR file The metadata also contains a ThingTemplate the name is set to MySqlServer, but can be modified as needed Select the lib folder and metadata.xml file and send to a zip archive Tip: The name of the zip archive should match the name given in the name attribute of the ExtensionPackage entity in the metadata.xml file Import the newly created extension as usual To the JDBC extension, simply create a new thing and assign it the new ThingTemplate that was imported with the JDBC extension Configuration Field Explanation: JDBC Driver Class Name ​Depends on the driver being used Refer to documentation JDBC Connection String ​Defines the information needed to establish a connection with the database Connection string examples can be found in the ThingWorx Help Center ConnectionValidationString ​A simple query that will work regardless of table names to be executed to verify return values from the database   Alternatively, you may download the jdbc connector creator from the marketplace here https://marketplace.thingworx.com/Items/jdbc-connector-extension Then you may just view the mashup and use it to package your jdbc jar into an extension (which can be later imported into ThingWorx).  

  The scenario: Your company has settled on Azure as a cloud platform and you are currently using Azure IoT Edge as your connectivity strategy. You need a quick way to build IoT applications with your Azure devices. You’re looking for industry-proven and time-saving features like Mashup visualization, built-in connectivity to enterprise systems (like SAP or Oracle) with ThingWorx Flow, secure and scalable file transfer to your Azure-connected devices and the ability to create augmented reality (AR) experiences with Vuforia Studio. All of these options are available to you thanks to the ThingWorx-Azure IoT Hub Connector; it’s like the ice cream truck driving by on a hot summer day.   (If you’re wondering why we selected Azure as our preferred infrastructure, check out my previous interview with Neal, a Worldwide ThingWorx Center of Excellence Principal Lead here at PTC.)   I sat down with Ankit, a ThingWorx Product Manager, this week to learn more about the ThingWorx-Azure IoT Hub Connector. When Ankit’s not learning new hobbies like how to surf, snowboard or bike, he’s supporting our Microsoft partnership by enhancing and implementing ThingWorx-Azure functionality. Here’s how our conversation went:   Kaya: What is the Azure IoT Hub? Ankit: The Azure IoT Hub acts as a central message router for bi-directional communication between the cloud (and your ThingWorx applications) and your connected devices. The Azure IoT Hub securely connects, monitors and manages billions of devices. It is an open and flexible cloud platform as a service that supports open-source SDKs and multiple protocols. With ThingWorx, we enable you to authenticate user access per device to ensure your IoT solutions remain secure.   Kaya: I understand your team has created the ThingWorx-Azure IoT Hub Connector. Can you explain what it is and what it does? Ankit: The Azure IoT Hub Connector is an extension that is imported into ThingWorx for a developer to connect the Azure IoT Hub to ThingWorx. This helps ThingWorx to leverage the security and scalability of Azure while retaining the ThingWorx domain expertise to provide fast time to value.   The Connector is built on the ThingWorx Connection Server core. What it essentially does is convert JSON objects from Azure IoT Hub into ThingWorx property types (and vice versa) so that the digital twin data of an Azure device can be native to ThingWorx.   Since the Connector is built on the ThingWorx Connection Server, it is horizontally scalable and leverages features such as health check, metrics (message count and size, property writes) and logging.   Kaya: What was the challenge developers were facing that led us to create the Azure IoT Hub Connector? Ankit: There was no easy way for a developer to use ThingWorx to represent an Azure IoT device. Users weren’t easily able to take advantage of ThingWorx services and functionality on their Azure IoT devices, which were inherently connected to the Azure IoT Hub. Similarly, ThingWorx users were not able to take advantage of Azure services in a “configure-not-code” fashion in ThingWorx.   Kaya: How does the Connector solve this problem to enable you to integrate the two platforms and device models for a better combined solution? Ankit: Once you have an Azure device represented as a “Thing” in ThingWorx, you can use all the features and capabilities of ThingWorx Composer, Mashup Builder, etc. to build applications using the data from that Azure device.   Kaya: That’s pretty great. Ankit: Thanks, agreed. In the next version of the Connector, we’ll integrate more closely with Azure, such that our developers can leverage Azure services as well via ThingWorx, instead of building those services from scratch on Azure all on their own. For example, developers will be able to send software content, like firmware updates, to an Azure device without writing any code on Azure. All of this can be done on ThingWorx using Azure components like Azure IoT Edge Runtime.   Kaya: Awesome. In the meantime, what are the top two or three things a developer can do with the Azure IoT Hub Connector today? Ankit: Today, developers can take advantage of ingress and egress processing as well as file transfer. I’ll explain what these mean. Ingress Processing: Azure IoT devices (i.e. devices that are running Azure SDKs) send messages to the Azure IoT Hub. These messages are typically values of device properties (e.g. temperature). The Azure IoT Hub Connector “listens” for these messages, translates them and passes them to the ThingWorx platform. Egress Processing: Egress messages are messages that arrive from ThingWorx and are pushed to the Azure IoT Hub; an example might be pushing property updates to an Azure IoT device. File Transfer: The Azure IoT Hub Connector supports transferring files between Azure IoT devices and an Azure storage container (i.e. Blob store). An Azure storage container is represented by a ‘FileRepository’ Thing within ThingWorx. This enables developers to transfer files from an Azure storage container to ThingWorx and vice versa.   Kaya: What are two exciting features planned for a future release of the Connector? Ankit: Two exciting features planned for July include software content management (or SCM) and compatibility with ThingWorx Asset Advisor. Software Content Management (SCM): In our next release, we plan to have support for SCM from ThingWorx to an Azure IoT Edge device (an Azure IoT device with IoT Edge Runtime) via Azure IoT Hub. SCM allows users to transfer a variety of content like configuration settings, operating system patches and software updates and/or patches to a software agent on your Azure devices. SCM also allows you to manage your remote assets and keep them patched, secure and up-to-date with the latest features without having to dispatch a technician. This helps to reduce cost and complexity of software distribution and installation. Compatibility with ThingWorx Asset Advisor: Also planned for our next release, you will be able to readily manage Azure IoT devices directly through Asset Advisor to see key device alerts and warnings. This makes it even easier for you to leverage Asset Advisor to rapidly enable remote monitoring of your Azure devices.   Kaya: Exciting stuff. For our readers not familiar with Asset Advisor, check out this episode of my “ThingWorx on Air” podcast to understand what Asset Advisor is and how it works. Okay, next question. Do you have an example of a customer using Azure IoT Hub? Ankit: Absolutely. Colfax, an industrial manufacturing company, is using Azure IoT Hub to improve the efficiency of its IoT efforts across the enterprise. You should check out our case study on Colfax if you haven’t seen it yet.   Kaya: Where should I as a developer go if I want to learn more about the Azure IoT Hub Connector or Azure in general? Ankit: Depending on what you’re looking for, I’d recommend you check out the Help Center for technical guidance or the ThingWorx Azure IoT Hub Connector Release Notes, v. 2.0.0 for release updates.   Kaya: Finally, where can I go to download the ThingWorx Azure IoT Hub? Ankit: You can download it from the PTC Marketplace. Enjoy! Readers, let me know what you think about the Azure IoT Hub Connector in the comments below and reach out with any questions. While we’re excited to deliver what we have planned, our release content may change. In the meantime, for updates, tips and tricks and relevant info, stay connected!

In case it's useful for anyone, I successfully used the Thingworx Importer to import an Entity using version 8.4.1. The import command was in a CURL script (can also be run using e.g. Cygwin on Windows) and the entity data was contained in an XML file. The XML file is attached to this post, and the CURL script is copied into the bottom of the post body.   Notes on using the script: Copy CURL code into import.sh You might need to change the line endings to UNIX, e.g. in Notepad++ menu option Edit > EOL Conversion > Unix Give permissions to run import.sh (chmod +x import.sh) ./import.sh >>>>>>>>>>>> #!/bin/bash APPKEY="2e6704c0-XXXX-XXXX-XXXX-a1589e387d1a" TWX_HTTP_PORT="8018" FILE_PATH_TWX="Things_testImport.xml" PROTOCOL="http://" IP="localhost" URL="$PROTOCOL""$IP"":""$TWX_HTTP_PORT""/Thingworx/Importer?purpose=import" curl -X POST -H 'appKey: '$APPKEY \ -H 'Content-Type: multipart/form-data' \ -H 'Accept: application/json' \ -H 'x-thingworx-session: true' \ -H 'X-XSRF-TOKEN: TWX-XSRF-TOKEN-VALUE' \ -F 'upload=@'$FILE_PATH_TWX \ $URL <<<<<<<<<<<<   Also TWX Importer is explained in this support article.

Please find here an Labview implementation to connect to Thingworx via RestCalls. Have Fun using it. Any Feedback is appreciated. https://github.com/Seppel1985/LabVIEW_TWX_RestAPI

  Helloooooo ThingWorx users,   Ever wanted to see the coolest technology in action? Ever wished you could surround yourself with awesome ThingWorx developers? Maybe you’ve even wished you could meet the ThingWorx product management team!   If that’s the case, you’re in luck! Join me for LiveWorx 2019 from June 10 – 13 in Boston this summer to discover how you can make digital transformation a reality for your organization. See what all the hype’s about here!   I’ll be presenting a rockin’ session on some exciting new functionality coming in ThingWorx to help with enterprise-wide app deployment. See me present with Chris Baldwin on Tues, June 11, @ 1:15pm in our session titled Introducing Solution Central: Your Gateway to Accelerated IIoT Value Across the Enterprise!   For a sneak peek of what’s to come at LiveWorx, here are seven sensational sessions our developers can’t miss! (Note: Dates and times are subject to change.) It's Electric: HowCaterpillar Develops Compelling IIoT Apps That Resonate With Customers & Dealers Mon, June 10, @ 4:30 (45 min) ThingWorx and Microsoft Azure from A to Z Tues, June 11, @ 4:00pm (45 min) It’s All About The Apps: Introducing ThingWorxMashupBuilder 2.0 and More! Wed, June 12, @ 9:00am (45 min) Connecting Asset Advisor to Azure Wed, June 12, @ 3:00pm (45 min) ThingWorx for Scalability with InfluxDB and Beyond! Wed, June 12, @ 3:00pm (45 min) From Pilot to Production: Tips & Tricks for Vuforia Studio Thurs, June 13, @ 12:00pm (45 min)   Hope to see you all there and meet you in person!   Stay connected, Kaya

  Hi everyone,   This week, Anthony Moffa returns to Ask Kaya in a different form from his original appearance explaining the benefits of Thing Presence in ThingWorx. As much as we enjoy reading Anthony in print, you can now hear from the man himself in the “Moffa Monitoring Minute!”   Listen to Episode 04 of “ThingWorx on Air” as he explains what Asset Advisor is and how you can use it to remotely monitor assets, shorten service cycles, and improve visibility of your device fleet.   Want to learn even more about Asset Advisor? Check out this video or read through our website!   Reach out with any questions and just Ask Kaya!   Stay connected, Kaya

  You’ve seen him before. You’ve heard him before. Fans around the globe can’t get enough of him. He’s…   ...Joe Biron—our CTO of IoT!   Hear Joe share his thoughts on the future of the industrial IoT with ThingWorx in Episode 03 of our “ThingWorx on Air” podcast!   Any questions? Just Ask Kaya.   Stay connected!

  Question: What are some best practices around building IIoT solutions with ThingWorx?   Meet Ward. Ward works on the product management team for our Manufacturing Apps (i.e. Asset Advisor, Operator Advisor, Production Advisor, etc.). He’s a super cool and smart guy, and he always has an answer to my ThingWorx questions. He has so many answers, in fact, that he worked closely with other ThingWorx experts like Sangeeta to create the ThingWorx Application Development Guide.   I sat down with him to hear his top few tips from the guide. And, just in case we don’t have enough fun around here on “Ask Kaya,” we decided to list his top tips not by “1”-“2”-“3”, but by “W”-“A”-“R”-“D”.   Without further to do, here are Ward’s top tips from the ThingWorx Application Development Guide.   Whitelist your IPs for application keys. (See page 67.) Auto Refresh widget vs. GetProperties service? How should I update live data to my mashup? (See page 25.) Reuse components to increase efficiency and improve your application design. (See page 69.) Don’t use a Thing Template when you really should use a Thing Shape. (See page 10.)   To see more, check out the full ThingWorx Application Development Guide here!   Look out for our next release of the App Development Guide in July! It’ll feature our Manufacturing Apps to share even more ThingWorx best practices!   Reach out with any questions and stay connected! - Kaya

When predicting a Boolean goal such as Failure in the next hour or any other goal that has a yes or no answer, Thingworx Analytics(TWXA) models will output a 'risk' of the event occurring. TWXA will intelligently pick a threshold beyond which that risk warrants attention. 1. In Analytics Builder, click on the export button 2. This will export a PMML model and download it for you 3. Open up the PMML model, in the output section, you will find a condition that explains the threshold that was selected by TWX Analytics.   In this example case, TWXA chose 0.5 as the best Threshold.   Note: The export button will only be available in Builder for TWXA 8.4+.

  Hello, ThingWorx Users!   As promised, we are back with Episode 02 of ThingWorx on Air. Listen to our PM Milan share the secrets of Operator Advisor and how we built the solution with an eye for IIoT developers.   Learn how Operator Advisor provides you with pre-built snippets of code for widgets, services, etc. targeted specifically for shop floor operators. No more starting from scratch!   Reach out if you have any questions or topic requests!   Stay connected, Kaya   P.S. Keep your ears peeled for the “Wowza Widget of the Week!”

This is a lessons learned write up that I proposed to present at Liveworx but it didn't make the cut, but I did want to share it with all the developer folks. Please note that this is before we added Influx and Micro Services, which help improve the landscape. Oh and it's long 🙂 ------------------------------------------ This is written as of Thingworx 8.2   Different ways to scale Data and Processing with Thingworx Two main issues are targeted Data Storage Platform processing Data Storage in Thingworx Background Issues around storage is that due to the limited indexing in the Persistence Provider with then the actual values according to the datashape being in a JSON Blob So when you look in the Persistence Provider you’ll see Source sourceType Location entityID Datetime Tags ValueJSONBlob   The first six carry an index, the JSON Blob which holds the values according to the datashape is not, that can read something like {value1:firstvalue,value2:secondvalue,value3:[ …. ]} etc. This means that any queries beyond the standard keys – date/time, entityID (name of Stream or DataTable), source, sourcetype, tags, location become very inefficient because it will query the records and then apply the datashape query server side. Potentially this can cause you to pull way more records over from Persistence Provider to Platform than intended. Ie: a Query on Temperature in my data, that should return 25 records for a given month, will perhaps first return 250K records and then filter own to 25. The second issue with storage is that all Streams are stored in one table in the Persistence Provider using entityID as an additional key to figure out which stream the record is for. This means that your record count per table goes up much faster than you’d expect. Ie: If I have defined 5 ValueStreams for 5 different asset types, ultimately all that data is still in one table in the Persistence Provder. So if each has 250K records, a query against the valuestream will then in actuality be a query against 1.25 million records. I think both of these issues are well known and documented? By now and Dev is working on it. Solution approaches So if you are expecting to store a lot of records what can you do? Archive The easiest solution is to keep a limited set and archive off the rest of the data, preferably into a client’s datalake that is not part of the persistence provider, remember archiving from one stream to another stream is not a solution! Unless … you use Multiple Persistence Providers Multiple Persistence Providers Thingworx does support multiple persistence providers for storing data. So you can spin up extra schemas (potentially even in the same DataBase Server) to be the store for additional Persistence Providers which then are mapped to a specific Stream/ValueStream/DataTable/Blog/Wiki. You still have to deal with the query challenge, but you now have less records per data store to query through. Direct queries in the Persistence Provider If you have full access to your persistence provider (NOTE: PTC Cloud Services does NOT provide this right now). You can create an additional JDBC connection to the Persistence Provider and query the stream directly, this allows you to query on the indexed records with in addition a text search through the JSON Blob all server side. With this approach a query that took several minutes at times Platform side using QueryStreamEntries took only a few seconds. Biggest savings was the fact that you didn’t have to transfer so many records back to the Platform server. Additional Schemas You can create your own schema (either within the persistence provider DB – again not supported by PTC Cloud Services) in a Database Server of your choice and connect to it with JDBC/REST. (NOTE: I believe PTC Cloud Service may/might offer a standalone server with actual root access) This does mean you have to create your own Getter/Setter services to retrieve and store information, plus you’ll need some event to store (like DataChange). This approach right now is probably a common if not best practice recommendation if historical information is required for the solution and the record count looks to go over 1 million records and can’t just be queried based on timestamp. Thingworx Event Processing Background Thingworx will consistently deal with many Things that have many Properties, and often times there will be Alerts/Rules that need to run based on value changes. When you are using straight up Alerts based on a limit value, this isn’t such a challenge, but what if you need to add some latch/lock/debounce logic or need to check against historical values or check multiple conditions? How can you design something that can handle evaluating these complex rules, holds some historical or derived values and avoid race conditions and be responsive? Potential Problems Race conditions Multiple Events may need to update the same Permanent or Temporary store for the determination of a condition. Duplicates If you don’t have some ‘central’ tracker, you may possibly trigger the same rule multiple times. Slow response You are potentially triggering thousands or more events at the same time, depending on how you’ve set up your logic, your response could become so slow that the next event will be firing before finish and you’ll overload the system. System queue overrun If your events trigger faster than you can handle the events, you will slowly build up and finally overrun the event queue. System Thread count overrun Based on the number of cores in your system, you can overrun the number of threads that can be handled. Connection Pool overrun Each read/write to a stream/datatable but also Property Persist is a usage of the connection pool to your persistence provider. If you fire a lot at once, you can stack up requests and cause deadlocks System out of memory Potentially in handling the events you are depending on in memory information, if that is something that grows over time, you could hit an ‘Out of Memory’ issue. Solution Approaches Batch processing Especially with Agents/Sources that write a set of property updates, you potentially trigger multiple threads that all may need the same source information or update the same target information. If you are able to process this as a batch, you can take all values in account and only process this as a single event and have just a single read from source or single write to target. This will be difficult to achieve when using something like Kepserver, unless it is transferring as something non-standard like MQTT. But if you can have the data come in as a single REST POST this approach becomes possible. In Memory vs. Table/Stream Storage To speed up response time, you can put necessary information into Memory vs. in a DataTable or Stream. For example, if you need the most current received record together with some historical values, you could: Use a Stream but carry the current value because the stream updates async. (ie adding the current value to the stream doesn’t guarantee that when you read from the stream it has already been committed) Use a DataTable because they are synchronous but it can make the execution slow, especially if you are reaching 100K records or more Use an InfoTable or JSON Property, now this information is in memory and runs the fastest and is synchronous. Note that in some speed testing JSON object was faster than InfoTable and way faster than DataTable. One challenge is that you would have to do a full overwrite if you need to persist this information. Doing a full write does open up the danger of a race condition, if this information is being updated by multiple threads at the same time. If it is ok to keep the information in memory than an InfoTable is nice because you can just add/delete rows in memory. I sadly haven’t figured out yet how to directly do this to a JSON object property :(. It is important to consider disaster recovery scenarios if you are only using this in memory Centralized Processing vs. Distributed Processing Think about how you can possibly execute some logic within the context of the Entity itself (logic within the ThingShape/ThingTemplate) vs. having it fire into a centralized Service (sync or async) on a separate Entity. Scheduler or Timer As much as Schedulers and Timers are often the culprit of too many threads at the same time, a well setup piece of logic that is triggered by a Scheduler or Timer can be the solution to avoid race conditions If you are working with multiple timers, you may want to consider multiple schedulers which will trigger at a specific time, which means you can eliminate concurrence (several timers firing at the same time) Think about staggering execution if necessary, by using the hated, looked down upon … but oft necessary … pause() function !!!! Synchronous vs. Asynchronous Asynchronous execution can give great savings on the processing speed of a thread, since it will kick off the asynch parts and continue on. The terrible draw back, you can’t tell when it is finished nor what the resulting output is. As you mix and match synch/asynch vs processing speed, you may need to consider other ways to pick up when an asynch process finishes, some Property elsewhere that will trigger into a DataChange for example. Interesting examples Batch Process With one client there was a batch process that would post several hundred results at once that all had to be evaluated. The evaluation also relied on historical information. So with some logic these properties were processed as a batch, related to each other and also compared to information held in memory besides historically storing the information that came in. This utilized several in memory objects and ultimately also an eval() statement to have the greatest flexibility and performance. Mix and Match With another client, they had a requirement to have logic to do latch/lock and escalation. This means that some information needs to be persisted, however because all the several hundred properties per asset are coming in through Kepware once a second, it also had to be very fast. The approach here was to have the DataChange place information into an in memory infotable that then was picked up by a separate latch/lock/escalation timer to move it over to the persistent side. This allowed for the instantaneous processing of DataChange and Alerts, but also a more persistent processing of latch/lock/escalation logic. In Conclusion Remember that PTC created its software for specific purposes. I don’t think there ever will be a perfect magical platform that will do everything we need and want. Thingworx started out on a specific path which was very high speed data ingest and event platform with agnostic all around connectivity, that provided a very nice holistic modeling approach and a simple way to build UI/UX. Our use cases will sometimes go right past everything and at times to the final frontier aka the bleeding edge and few are a carbon copy of another. This means we need to be innovative and creative. Hopefully all of you can use the expert knowledge you have about our products to create those, but then also be proactive and please share with everyone else!  

Key Functional Highlights See What’s New in ThingWorx Apps and ThingWorx Operator Advisor Guide     Compatibility - ThingWorx Manufacturing and Service Apps ThingWorx 8.4.x KEPServerEX 6.2 and later Earlier Version of KEPServerEX and 3rd party OPC will be supported via Aggregator All other TWX supported data sources but specifically: NI, EMS and Azure IOT Hub Upgrade Support 8.0.1 and later     Compatibility – ThingWorx Manufacturing Operator Advisor ThingWorx 8.3.4 and later ThingWorx 8.4.x and later MPMLink 11.0 M030+ with WRS 1.3     Documentation   What’s New in ThingWorx Apps ThingWorx Apps Setup and Configuration Guide ThingWorx Apps Customization Guide ThingWorx Operator Advisor Guide     Additional information The National Instruments Connector can be found on PTC Marketplace     Download   ThingWorx Manufacturing and Service Apps & Operator Advisor Extensions National Instruments TestStand Connector

  Some call him JB. Some call him Joe. Others call him @bironology. I call him an awesome guy—he’s our CTO of IoT here at PTC—Joe Biron!   Joe’s an architect at heart, a developer, an avid gamer and a technologist (check him out on Twitter @bironology).   Watch as he guest stars in the Microsoft Channel 9 “IoT Deep Dive Live” show! Listen to him explain how you can build end-to-end industrial solutions with ThingWorx and Azure. ­­ And, guess who’s running the demo behind the scenes: our Global ThingWorx COE Lead, Neal Hagermoser, who you may recognize from a previous Ask Kaya post where I interviewed him on why we chose to partner with Azure.   Enjoy the show and stay connected! Kaya

  Question: What should I know about using ThingWorx with InfluxDB to store my time series data? Hi, ThingWorx users!   It’s here! Thanks for waiting patiently since my previous post announcing ThingWorx’ new support of InfluxDB as a time series persistence provider.   As of our 8.4 release, you can now use InfluxDB to store your ThingWorx time series data with incredible power and ease.   Want to learn more? Check out the following FAQs:   1. What is InfluxDB? Who is InfluxData? InfluxDB is a time series database designed to handle high write and query loads. It is meant to be used as a backing store for any use case involving large amounts of timestamped data, like monitoring, application metrics, IoT sensor data, and real-time analytics that you’d find in ThingWorx.   InfluxDB is created by InfluxData, an awesome company that we are proud to call a PTC partner.   2. When would I want to use InfluxDB for IIoT? While the ThingWorx IIoT platform supports multiple databases to persist IIoT data and is agnostic when it comes to the storage layer, InfluxDB is the ideal choice for time series. When the number of connected devices increases, along with the amount of streaming data, the need to have a high-scale telemetry database choice is obvious.   For very high scale data ingestion, InfluxDB should be used as a persistent provider with the ThingWorx platform for multiple reasons. Its flexibility and ease of use provides native support for standard time series functions, including: sampling, interpolation, time bucketing, aggregation, selector, transformation, predictor, etc. It does all of this while supporting a high compression of data (~45x) with the ability to handle thousands of writes per second and read thousands of rows in milliseconds.   Check out this article by our Enterprise Deployment Center (EDC) explaining why InfluxDB is great for small ThingWorx applications.   3. What are the three different flavors of InfluxDB? InfluxDB Open Source (TICK Stack), InfluxDB Enterprise & InfluxDB Cloud. Here’s more info on each: InfluxDB Open Source (TICK Stack): This is the open-source version of the product available to download via the InfluxData website. Also included here are the other projects that comprise the TICK Stack, including: [T] Telegraf; open source collection agent [I] InfluxDB; open source time series database [C] Chronograf; open source visualization application [K] Kapacitor; open source streaming processing engine; side car to InfluxDB InfluxDB Enterprise: This is the commercial software version of InfluxDB for high availability clustering and the recommended time series database to be used for production with ThingWorx 8.4 and later. InfluxDB Enterprise works with the rest of the TICK stack interchangeably (Telegraf, Chronograf, Kapacitor). InfluxDB Cloud: This is the commercial service version of InfluxDB, hosted on AWS, managed by InfluxData, and delivered as a service to customers. InfluxDB Cloud works with the rest of the TICK stack interchangeably (Telegraf, Chronograf, Kapacitor). To learn more about the different modules of InfluxDB (Telegraf, Chronograf, Kapcitor), check out InfluxData Introduction for documentation or InfluxData Products for product info.   4. What is the difference between InfluxDB opensource and enterprise? InfluxDB Open Source is available in a single (1 only) data node configuration only, albeit with “n” number of vCPU or “cores” provisioned on that single node.  InfluxDB Enterprise is available in multiple (2 or more) data node configuration, also with “n” number of vCPU or “cores” provisioned to each node. The Enterprise edition is generally preferred for production deployments that require high availability, replication, and redundancy. Provisioned along with the data nodes are three (3) meta nodes and a load balancer to distribute data workload across the multiple nodes. Typical configurations are in even increments of data nodes (i.e. 2, 4, 6, 8, etc.).   5. Where can I find the pricing overview for buying enterprise licenses for InfluxDB? The PTC product and go-to-market team have defined commercial pricing for InfluxDB Enterprise. For help with pricing, reach out to Chris Wensley (cwensley@ptc.com) and Anders Hinrichsen (anders@influxdata.com).   6. How do I configure InfluxDB with ThingWorx? We’ve outlined the steps for you in the ThingWorx Help Center and created a quick video to instruct you on how to install InfluxDB with ThingWorx. (view in My Videos) To see the current version of InfluxDB that we support, read our ThingWorx 9.0 System Requirements guide.   7. How do I configure InfluxDB and ThingWorx in a high availability scenario? With the ability to leverage multiple data stores, we work to provide the flexibility to best meet the needs of your IT preferences and investments. InfluxDB helps us do that. To configure ThingWorx for High Availability, please refer to this section of the ThingWorx Platform 9 Help Center. To configure InfluxDB for High Availability at the database level, please refer to InfluxData’s documentation on how to Install and deploy InfluxDB Enterprise clusters.   8. Where can I learn more about how to monitor and manage InfluxDB? Monitoring info for InfluxDB can be found here: Monitoring Tools for TICK Stack.   9. How can I tune and optimize InfluxDB with ThingWorx? The best approach for running InfluxDB with PTC ThingWorx 8.4 (or later) is to treat the workload and configuration just as you would in a stand-alone deployment. We suggest to stick to the recommendations in the InfluxDB and TICK stack documentation.   10. How do I perform backup and recovery of ThingWorx with InfluxDB? Please see the ThingWorx Platform Backup and Recovery Planning Technical Brief to plan for back and recovery. You can also find more more details on taking backups and restoring data from InfluxDB in the Backing up and restoring in InfluxDB Enterprise overview.   11. Where can I learn more about sampling, interpolation, time bucketing, aggregation, pivot​ and other key features of InfluxDB? Features of InfluxDB can be found here: InfluxData Time Series Platform. Implementation of InfluxDB features can be found here: Getting Started with InfluxDB.   12. What are all the different persistence providers supported with ThingWorx? When should I use InfluxDB? ThingWorx supports the following model and data provider storage options: H2, PostgreSQL, MS SQL Server and AzureSQL ThingWorx supports the following data provider only storage options: InfluxDB Please refer to the model and data best practices section of the ThingWorx 9 Help Center for further information on options how to store your model and data with ThingWorx.   We have also updated the ThingWorx Platform 9.0 Sizing Guide to provide relevant information to estimate the amount of processing and memory that ThingWorx may need to meet your requirements. It also provides guidance on when to use InfluxDB for your scale needs.   13. When should I use InfluxDB over DataStax Enterprise (DSE)? Here is a good blog post that benchmarks time series data performance of InfluxDB vs. Cassandra, which is the core of DataStax Enterprise (DSE). In specific use cases, InfluxData Enterprise may be more cost effective when compared to similar telemetry use cases with DSE.   14. How can I migrate my data from PostgreSQL to InfluxDB? Migration from PostgreSQL or MSSQL is supported by the ThingWorx in-built data tools, which can export entities and data from PostgreSQL or MSSQL and then import them into InfluxDB.   Details on how to upgrade to ThingWorx 9.0 can be found in the Upgrading ThingWorx  section of the ThingWorx 9 Help Center.   15. Should I use InfluxDB as a time series store rather than OSI PI, IP21, or others? For ThingWorx 8.4 and later, InfluxDB is the recommended time series store. This can be implemented at the edge with ThingWorx (i.e. “front end”) using the open source edition and can also be implemented at the hub (i.e. “back end”) using either of the commercial editions designed for HA production workloads.   As always, ThingWorx can connect to most industrial software, including OSI PI, IP21, etc. with our integration toolset.   That’s a wrap—almost! We’ve added two extra questions for you.   16. What’s on the roadmap for ThingWorx with InfluxDB? Key development work to fully leverage built-in InfluxDB querying capabilities and support InfluxDB 2.0 in future ThingWorx releases Leveraging query operations capabilities from InfluxDB to further improve query performance Supporting additional native InfluxDB features (e.g. continuous queries)   17. What should I do if I need technical support with InfluxDB? If you select InfluxDB as your persistence provider, then all support requests related to configuring InfluxDB can be logged through PTC Technical Support at https://support.ptc.com or by calling 1-800-477-6435. You may also want to use the PTC Community to learn and collaborate with the growing PTC developer community. For all other requests related to database management, troubleshooting, monitoring, and administration, we encourage you to reach out to InfluxData directly based on your enterprise purchase contract made with InfluxData. PTC customers using InfluxDB can also email ptc-support@influxdata.com for support requests related to InfluxData.   If you’re as excited as I am about the ability to store your time series data with InfluxDB, let me know in the comments below!   Until next time, if you have any questions, just ask Kaya!