IoT Tips

In this video we show: - how to deploy the microservices via jar files - how to setup ThingWorx to use those microservices for anomaly detection   Updated Link for access to this video:  ThingWorx Analytics: Deploying Training and Result Microservices via jar files for Anomaly Detection

Key Functional Highlights ThingWorx 8.0 covers the following areas of the product portfolio:  ThingWorx Analytics, ThingWorx Utilities and ThingWorx Foundation which includes Core, Connection Server and Edge capabilities. Highlights of the release include: ThingWorx Foundation Native Industrial Connectivity: Enhancements to ThingWorx allow users to seamlessly map data from ThingWorx Industrial Connectivity to the ThingModel. With over 150 protocols supporting thousands of devices, ThingWorx Industrial Connectivity allows users to connect, monitor, and manage diverse automation devices from ThingWorx. With this new capability, users can quickly integrate industrial operations data in IoT solutions for smart, connected operations. Native AWS IoT and Azure IoT Cloud Support: ThingWorx 8 now has deeper, native integration with AWS IoT and Azure IoT Hub clouds so you can gain cost efficiencies and standardize on the device cloud provider of your choice.  This support strengthens the connection between leading cloud providers and ThingWorx. Next Generation Composer: Re-imagined Composer using modern browser concepts to improve developer efficiency including enhanced functionality, updated user interface and optimized workflows. Product Installers:  New, Docker-based product installers for Foundation and Analytics make it easy and fast for customers to get the core platform and analytics server running. Single Sign On (SSO): Provides the ability to login once and access all PTC apps and enterprise systems. License Management: Simple, automated, licensing system for collection, storage, reporting, management and auditing of licensing entitlements. Integration Connectors: Integration Connectors allow Thingworx developers and administrators quick and easy access to the data stored on external ERP, PLM, Manufacturing and other systems to quickly develop applications providing improved Contextualization and Analysis. Thingworx 8.0 delivers ‘OData’ and ‘SAP OData’ connectors plus the ability to connect to generic web services to supplement the ‘Swagger’ and ‘Windchill Swagger’ Connectors released in Thingworx 7.4. An improved mapping tool allows Business Administrators to quickly and easily transform retrieved data into a standard Thingworx format for easy consumption. Includes single sign on support for improved user experience. ThingWorx Analytics Native Anomaly Detection: ThingWorx 8 features more tightly integrated analytics capabilities, including the ability to configure anomaly alerts on properties directly from the ThingWorx Composer. ThingWatcher technology is utilized to increase machine monitoring capabilities by automatically learning normal behavior, continuously monitoring data streams and raising alerts when abnormal conditions are identified. ThingWorx Utilities Software Content Management (SCM) – Auto Retry: Provides the ability to automatically retry delivery of patches to devices if interrupted.  This ensures the ability to successfully update devices.  ThingWorx Trial Edition ThingWorx Trial Edition will be available to internal PTC resources at launch and will be made available externally on the Developer Portal shortly after launch. Developer Enablement: Enhancements have been made to the Trial Edition installation tool, providing a native installation process of the ThingWorx platform including: ThingWorx Foundation ThingWorx Utilities ThingWorx Analytics ThingWorx Industrial Connectivity Documentation ThingWorx 8.0 Reference Documents ThingWorx Analytics 8.0 Reference Documents ThingWorx Core 8.0 Release Notes ThingWorx Core Help Center ThingWorx Edge SDKs and WebSocket-based Edge MicroServer Help Center ThingWorx Connection Services Help Center ThingWorx Industrial Connectivity Help Center ThingWorx Utilities Help Center ThingWorx Utilities Installation Guide ThingWorx Analytics Help Center ThingWorx Trial Edition User Guide Additional information ThingWorx eSupport Portal ThingWorx Developer Portal ThingWorx Marketplace Download The following items are available for download from the PTC Software Download site. ThingWorx Platform – Select Release 8.0 ThingWorx Utilities – Select Release 8.0 ThingWorx Analytics – Select Release 8.0 You can also read this post in the Developer Community from Jeremy Little about the technical changes in ThingWorx 8.0.

There are a lot of new and exciting changes included in ​ThingWorx 8.0.0, which is due out today, and no doubt, you'll want to get a test environment set up to try them out.  But there are a few changes that could impact your ability to get up and running with a new server that I wanted to share with everyone. IMPORTANT – Changes to Licensing in ThingWorx 8.0.0 In ThingWorx 7.4.0, a new Licensing Subsystem was introduced, and the license file required was provided as part of the ThingWorx 7.4.0 platform download.  As of ThingWorx 8.0.0, the license.bin file will no longer be provided with the platform download, but will instead need to be downloaded from the PTC Licensing Tool on the PTC eSupport Portal. As part of the operating system specific Installing ThingWorx (OS) sections in the Installing ThingWorx 8.0 guide, additional steps have been added that outline how to use the PTC Licensing Tool to download and install the ThingWorx license file for your organization. The license file downloaded from the ThingWorx Licensing Tool must be renamed to license.bin and copied to the /ThingworxPlatform directory prior to starting ThingWorx for the first time.  The server will fail to start if it cannot find this license file. For more information:      KCS Article 264374 - ThingWorx 8.0.0 Licensing IMPORTANT – Default Administrator Password for ThingWorx Composer is Changing in 8.0.0 In order to help encourage the use of secure passwords for the Administrator account on ThingWorx servers, the default Administrator account password will be changing in version 8.0 and above.  The new Administrator password is a complex password containing mixed case and special characters. This will encourage administrators to change their default, fully-privileged account password to one that is more secure and conforms to their organizational security standards. Information about the new default password can be found in each of the operating system specific Installing ThingWorx (OS) sections of the Installing ThingWorx 8.0 guide. PTC strongly advises against the use of any default password on any product, and encourages administrators to immediately change any default password to a proper, complex password. For more information: KCS Article 264270​ - ​Unable to log into ThingWorx 8.0 Composer using the default Administrator login Application Key Usage is Changing in 8.0.0 As part of an effort to better secure the ThingWorx Platform, the use of Application Keys as URL parameters (through the ‘appkey’ parameter) is being deprecated in ThingWorx 8.0.0. For example, the following URL uses the now deprecated ‘appkey’ parameter to pass in an application key:               https://thingworx.server.com/ThingWorx/Things/MyThing/Services/MyService?appkey=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx When included in the URL, the ‘appkey’ parameter becomes a browser-cacheable, clear-text rendition of a usable application key.  A knowledgeable user could retrieve this application key from their browser history and use it to perform unauthorized actions against a ThingWorx server. For full security, GET and POST requests that are run against a ThingWorx server should be performed over an HTTPS connection and should include the required application key in the request’s headers.  Query string parameters are visible in both HTTP and HTTPS contexts, and by moving the application key into the request headers, the application key itself is encrypted as part of the HTTPS request. (Note that the application key in a header is still visible for plaintext HTTP connections though!) By default, new installations of ThingWorx 8.0.0 will have the Allow Application Key as URL Parameter option disabled.  Upgrades from previous versions of ThingWorx will retain the ability to use the ‘appkey’ query string parameter, but PTC strongly encourages customers to promptly update any solutions that are dependent on sending the appkey using a URL parameter to move to request headers instead.  Please note that a future release of ThingWorx may completely disable the use of application keys as URL parameters. The Allow Application Key as URL Parameter option can be enabled or disabled on the ThingWorx PlatformSubsystem’s configuration page, accessible at System > Subsystems > PlatformSubsystem > Configuration. For more information: KCS Article 264349 - ThingWorx Appkey URL Parameter is Deprecated as of ThingWorx 8.0.0 Changes to Default Visibility (Organizations) in ThingWorx 8.0.0 Starting in ThingWorx 8.0.0, the Everyone organization will no longer be granted default visibility across all entity collections.  Only users who are a member of the Administrator group will be able to see ThingWorx entities on a newly installed server; any additional visibility permissions will need to be explicitly granted by the Administrator. This is a change in behavior from the previous ThingWorx releases where the Everyone organization (of which the all-encompassing Users group was a member) was granted visibility access by default to all entity collections. Visibility permissions had to explicitly be removed by the Administrator during solution development, which could be overlooked. For more information: KCS Article 264351 - Changes to Default Visibility (Organizations) in ThingWorx 8.0.0

Error: Exception: JavaException: java.lang.Exception: Import Failed: License is not available for Product: null Feature: twx_things Possible root cause: editing web.xml without further restart of the platform. In result, the product name does not pick up and the license path drops while the composer is still running, Fix: Go to LicensingSubsystem -> Services and run the AcquireLicense service. IF the issue does not get resolved, please contact the Support team, attaching your license.bin to the ticket.

All of the entities and data in storage entities(Stream, ValueStream, DataTable, Wiki, Blog) are kept in Persistence Provider, to enhance the storage volume of Thingworx, it is advised to use bigger Persistence Provider (PostgreSQL or Cassandra rather than Neo4j and H2), move and keep certain part of data in external database or set up a second Persistence Provider Data Storage entities (Stream, ValueStream, DataTable) and Collaboration entities (Wiki, Blog) can be assigned to the new Persistence Provider to keep the Business Data The original Persistence Provider (upon installation) will be freed from large data input and respond faster, and only holds the Model Data (entity information) Small Persistence Provider like H2 or Neo4j, which does not require extra steps to setup in the Thingworx installation process, cannot set up a second Persistence Provider (they don't have .bat files to configure the database) Here is an example for setting up a second PostgreSQL (may also applicable for Cassandra) Install a new PostgreSQL in the server Should give different folder names for the files, or two PostgreSQL will affect each other PostgreSQL 9.5 could work with Thingworx 7.2.x, but cannot be used with 7.3 and above. Version 9.5 is not officially supported or advised by now, and the user should take their own risk if doing so. The new PostgreSQL will have a new port number (e.g. first port is 5432, the new port is 5433) Open PostgreSQL using pgAdmin 3 Create a new user role: a. Right click PostgreSQL9.4 (localhost:5433). b. Select NewObject>New Login Role. On the Properties tab, in the Role name field, enter the user role name. c. On the Definition tab, in the Password field, enter a unique password (you will be prompted to enter it twice). Add the new <postgres-installation>/bin folder to system path variable Edit the thingworxPostgresDBSetup.bat and thingworxPostgresSchemaSetup.bat from the Thingworx software download package, and change the port property to 5433 Execute the two scripts, and new database and tablesapce will be created platform-settings.json file does not need extra configuration Restart the Tomcat server, a new folder will be created under ThingworxPostgresqlStorage folder Go to Thingworx composer, create a new Persistence Provider in the Home list Give it a new name, and select Persistence Provider Package(only one choice) After selection, a Configuration field will show up in the ENTITY INFORMATION field ( such field will not appear in Neo4j based Thingworx) Open Configuration and change the JDBC URL to jdbc:postgresql://localhost:5433/thingworx, update Password, click Save The new Persistence Provider can be used in data storage entities now The new Persistence Provider can also be set as default provider

This is just a quick note/reminder that starting 7.4, the utilities installation process has slightly changed. Several extensions are no longer bundled with ThingWorx Utilities and must be downloaded from the ThingWorx Marketplace and installed separately. For more information, see the “Prerequisites” topic in the ThingWorx Utilities Installation Guide: http://support.ptc.com/WCMS/files/172616/en/ThingWorxUtilitiesInstall.pdf The following extensions must be installed prior to installing ThingWorx Utilities available at ThingWorx IoT Marketplace ○ Google Maps Widget ○ Mail Extension ○ Web Sockets Tunnel Widget and Library

This script is provided as is and is not officially supported by PTC. Users are welcome to edit or change this script as desired. This script automates the installation steps outlined in the ThingWorx Installation Guide. If you are not familiar with these processes or are installing ThingWorx for the first time, it is recommended to refer to the guide for additional information. Requirements:   64-bit Linux: Ubuntu (all flavors) 14/15/16, Red Hat Enterprise Linux 6/7, or CentOS 6/7   Root privileges   Internet connection   A ThingWorx platform installation file for either H2 or PostgreSQL (available in the PTC downloads center https://support.ptc.com/appserver/cs/software_update/swupdate.jsp )   *NOTE: This script does not require a GUI and can be run in headless mode Minimal download option: The script will look in the working directory for pre-downloaded Java, Tomcat, and (for RHEL and CentOS) PostgreSQL installation files. If the installer detects one of the supported files, the user will be given the option to install with that file and skip the automated download. This feature is limited to specific versions and requires one (or more) of the following files:     jdk-8u92-linux-x64.tar.gz     apache-tomcat-8.0.33.tar.gz     pgdg-redhat94-9.4-3.noarch.rpm     NOTE: This reduces the amount of data downloaded by the script, but does not eliminate it entirely. An active internet connection is still required. Support for existing SSL certificates: If an existing SSL certificate exists and the user wishes to use it instead of automatically generating one with the script, the user can copy the '.keystore' file to the working directory before running the script. During the installation process the user will be prompted with the option to use this keystore file, and will need to provide the keystore password created with the certificate.     NOTE: this script does not support keystore passwords containing the following characters: '!/@\"#$%^&*()_+]    *Note for Red Hat Enterprise Linux users*     This script was designed to work on servers with an active Red Hat subscription. It has been tested and appears to work on unsubscribed servers if the following applications are installed: gcc make epel repository (see This Red Hat Announcement for more information) Download the file attached to this document Steps to use this script: 1. Create a new folder on your Linux server. (This will be referred to as the 'working folder') 2. Download the ThingWorx platform installer file from the PTC Software downloads page. (For example, MED-61111-CD-072_SP2_ThingWorx-Platform-Postgres-7-2-2.zip) 3. Copy the installer file to the working folder. Do not unzip. 4. Copy the installer.sh file to your working folder. 5. In the command prompt, cd to your working directory. 6. Make the installer.sh file excutable:   > chmod +x installer.sh 7. Run the installer.sh file either using sudo or logging in as root   > sudo bash installer.sh   OR   > su   > bash installer.sh   *NOTE: The 'bash' command is mandatory, running the installer with the 'sh' command will not work

If the ThingWorx 7.4 installation with MSSQL db doesn't start with a license error on the splash screen, despite taking the necessary steps for specifying the license path, the error might be misleading and the problem is actually lying in the database connection. Going to THingworxStorage/logs and opening ApplicationLog.log might reveal the following (or similar ) errors: 2017-04-13 10:26:17.993-0400 [L: INFO] [O: c.t.s.ThingWorxServer] [I: ] [U: SuperUser] [S: ] [T: localhost-startStop-1] Sending Post Start Notifications... 2017-04-13 10:26:17.999-0400 [L: ERROR] [O: c.t.p.m.MssqlModelExceptionTranslator] [I: ] [U: SuperUser] [S: ] [T: localhost-startStop-1] [message: Could not create a transaction for ThingworxPersistenceProvider] 2017-04-13 10:26:18.001-0400 [L: ERROR] [O: E.c.t.p.d.FileTransferDocumentModelProvider] [I: ] [U: SuperUser] [S: ] [T: localhost-startStop-1] [context: Could not create a transaction for ThingworxPersistenceProvider][message: Could not create a transaction for ThingworxPersistenceProvider] 2017-04-13 10:26:18.004-0400 [L: ERROR] [O: c.t.p.m.MssqlModelExceptionTranslator] [I: ] [U: SuperUser] [S: ] [T: localhost-startStop-1] [message: Could not create a transaction for ThingworxPersistenceProvider] 2017-04-13 10:26:18.005-0400 [L: ERROR] [O: c.t.s.s.f.FileTransferSubsystem] [I: ] [U: SuperUser] [S: ] [T: localhost-startStop-1] Error loading queued transfer jobs from persistence provider 2017-04-13 10:26:18.006-0400 [L: ERROR] [O: c.t.p.m.MssqlModelExceptionTranslator] [I: ] [U: SuperUser] [S: ] [T: localhost-startStop-1] [message: Could not create a transaction for ThingworxPersistenceProvider] 2017-04-13 10:26:18.006-0400 [L: ERROR] [O: E.c.t.p.d.FileTransferDocumentModelProvider] [I: ] [U: SuperUser] [S: ] [T: localhost-startStop-1] [context: Could not create a transaction for ThingworxPersistenceProvider][message: Could not create a transaction for ThingworxPersistenceProvider] 2017-04-13 10:26:18.007-0400 [L: ERROR] [O: c.t.p.m.MssqlModelExceptionTranslator] [I: ] [U: SuperUser] [S: ] [T: localhost-startStop-1] [message: Could not create a transaction for ThingworxPersistenceProvider] 2017-04-13 10:26:18.008-0400 [L: ERROR] [O: c.t.s.s.f.FileTransferSubsystem] [I: ] [U: SuperUser] [S: ] [T: localhost-startStop-1] Error loading queued transfer jobs from persistence provider 2017-04-13 10:26:18.008-0400 [L: INFO] [O: c.t.s.ThingWorxServer] [I: ] [U: SuperUser] [S: ] [T: localhost-startStop-1] Thingworx Server Application...ON A few things to check here. First, how the database was set up. The standard expected port if 1433, however, if you choose a different port - ensure that the port is available. If it isn't - update the port setting in SQL configuration manager. Another possible root cause -- the database scripts did not run properly upon the setup. Currently, there is an active Jira PSPT-3587 for resolving the script issue (NOTE, this thread to be updated once the Jira is fixed). There are two options here, either to run the sql files manually (found in the "install" folder of the downloadable), or edit the bat scripts manually. It's recommended to edit the script files instead because running the sql commands allows more room for a mistake. The following line in the bat scripts needs to be edited to have the ".\" removed: sqlcmd.exe -S %server%\%serverinstance%,%port% -U %adminusername% -v loginname=%loginname% -v database=%database% -v thingworxusername=%thingworxusername% -v schema=%schema%  -i .\thingworx-database-setup.sql Note, that the scripts need to be run from the same directory ("/install")

Please open your ApplicationLog located in ThingworxStorage/logs and inspect for errors. Something like the following might be observed: **********LICENSING ERROR ANALYSIS 2017-03-31 16:29:19.591+0300 [L: ERROR] [O: ] [I: ] [U: SuperUser] [S: ] [T: localhost-startStop-1] C:\WINDOWS\Sun\Java\bin is listed as a java.library.path but it does not exist 2017-04-12 13:51:53.515+0200 [L: ERROR] [O: c.t.s.s.l.LicensingSubsystem] [I: ] [U: SuperUser] [S: ] [T: localhost-startStop-1] Failed to load FlxCore library. Ensure it's in PATH (Windows) or LD_LIBRARY_PATH(other platforms) or set with the VM arg, -Djava.library.path. Error message : com.flexnet.licensing.DllEntryPoint.entry([B) Typically, if the license file has been downloaded and placed correcrtly, according to the 7.4 installation guide, the error in the log will specify where the file was found. If the license path was specified per the installation guide in the tomcat java path, you may try to clear it from the Tomcat java settings and keep these parameters: -Dserver -Dd64 -XX:+UseNUMA -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8 And then set up the license path in the environment variable path: Go to explorer, right click on "my computer" -> Properties -> Advanced settings -> Environment variables -> edit "PATH", add ; and then path to your tomcat extensions folder, “ ;<path to extensions folder of tomcat> “ or, for example ";C:\ptc\Thingworx\webapps\Thingworx\WEB-INF\extensions"

In the following scenario (for redhat in this case), running the dbsetup script results in the error: ./thingworxPostgresDBSetup.sh psql:./thingworx-database-setup.sql:1: ERROR:  syntax error at or near ":" LINE 1: CREATE TABLESPACE :"tablespace" OWNER :"username" location :... ^ psql:./thingworx-database-setup.sql:3: ERROR:  syntax error at or near ":" LINE 1: GRANT ALL PRIVILEGES ON TABLESPACE :"tablespace" to :"userna... ^ psql:./thingworx-database-setup.sql:5: ERROR:  syntax error at or near ":" LINE 1: GRANT CREATE ON TABLESPACE :"tablespace" to public; ^ psql:./thingworx-database-setup.sql:14: ERROR:  syntax error at or near ":" LINE 1: CREATE DATABASE :"database" WITH ^ psql:./thingworx-database-setup.sql:16: ERROR:  syntax error at or near ":" LINE 1: GRANT ALL PRIVILEGES ON DATABASE :"database" to :"username"; Given that the installed components match the requirements guide (tomcat 8, Postgresql 9.4.5+ for Thingworx 7.x), run the following command: Run this directly from bin directory of postgres deployment – psql -q -h localhost -U twadmin -p 5432 -v database=thingworx -v tablespace=thingworx -v tablespace_location=/app/navigate/ThingworxPostgresqlStorage -v username=twadmin That must get into command line interface. From there  run the following with full qualified path to the sql file on disk (replace FULLPATH with the path to sql file ) \i ./FULLPATH/thingworx-database-setup.sql If you are experiencing the above-mentioned syntax error, then likely the output will be: psql: FATAL:  database "twadmin" does not exist. Then from postgres bin directory, run the following: ./psql postgres \set Then the second command; \q psql -q -h localhost -U twadmin -p 5432 -v database=thingworx -v tablespace=thingworx -v tablespace_location=/app/navigate/ThingworxPostgresqlStorage -v username=twadmin \set   We see the following outputs: ./psql postgres Password: psql.bin (9.4.11) Type "help" for help. postgres=# \set AUTOCOMMIT = 'on' PROMPT1 = '%/%R%# ' PROMPT2 = '%/%R%# ' PROMPT3 = '>> ' VERBOSITY = 'default' VERSION = 'PostgreSQL 9.4.11 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-55), 64-bit' DBNAME = 'postgres' USER = 'postgres' PORT = '5432' ENCODING = 'UTF8' postgres=# \q -bash-4.1$ psql -q -h localhost -U twadmin -p 5432 -v database=thingworx -v tablespace=thingworx -v tablespace_location=/ThingworxPostgresqlStorage -v username=twadmin Password for user twadmin: twadmin=# \set AUTOCOMMIT = 'on' QUIET = 'on' PROMPT1 = '%/%R%# ' PROMPT2 = '%/%R%# ' PROMPT3 = '>> ' VERBOSITY = 'default' VERSION = 'PostgreSQL 8.4.20 on x86_64-redhat-linux-gnu, compiled by GCC gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-17), 64-bit' database = 'thingworx' tablespace = 'thingworx' tablespace_location = '/ThingworxPostgresqlStorage' username = 'twadmin' DBNAME = 'twadmin' USER = 'twadmin' HOST = 'localhost' PORT = '5432' ENCODING = 'UTF8' Note, even though Postgresql 9.4.5 has been installed by the system administrator, there are still traces of Postgresql 8.4.20 present in the system that cause the syntax error issue (possibly as part of  the default OS packaging). Removing the 8.4.20 rpms will resolve the problem.

This post will cover the challenges I've had while going through the setup of .NET SDK based ADO Service for SQL Server DB Connection. I'll be starting from the scratch on setting up the service for this to present full picture on the setup. Pre-requisite 1. Download and install Microsoft SQL Server Express or Enterprise edition, for testing I worked with Express edition : https://www.microsoft.com/en-us/sql-server/sql-server-editions-express 2. Once installed, it's imperative that the TCP/IP Protocol is enabled in the SQL Server Configuration Manager for the SQL Server 3. Download ThingWorx Edge ADO Service from PTC Software download page What is ThingWorx ADO Service? An ActiveX Data Object service allowing connection to a Microsoft database source e.g. MS SQL Server, MS Excel or MS .NET application to the ThingWorx platform. It is based on the ThingWorx .NET SDK. Installing ADO Service Let me begin by saying this is just a summary, in a crude way of course, of ThingWorx Edge ADO Service Configuration Guide. So when in doubt it's strongly recommended to go through the guide,also provided together with the downloaded package. I'll be using the ThingWorx ADO Service v5.6.1, most recent release, for the purpose of this blog. Depending if you are on x86 or x64 Windows navigate to the C:\Windows\Microsoft.NET for accessing the InstallUtil.exe You'll find the above specified file under following two locations, use the one that applies to your use case. i) For x64 : C:\Windows\Microsoft.NET\Framework64\v4.0.30319 ii) For x86 : C:\Windows\Microsoft.NET\Framework\v4.0.30319 1. Copy over the desired InstallUtil.exe to the location where you have unzipped the ADO Service package, the one downloaded above. e.g. I've put mine at C:\Software\ThingWorxSoftware\ADOService\ 2. Start a command prompt (Windows Start Menu > Command Prompt) and execute the InstallUtil.exe ThingWorxADOService.exe 3. This should create a service and some additional info in the \\ADOService folder in the form of InstallUtil.InstallLog 4. Check the log for confirmation, you should see something similar Running a transacted installation. ...     .... The Commit phase completed successfully. The transacted install has completed. ​​5. In Windows Explorer navigate to the folder containing all the unzipped files, and edit the AdoThing.config 6. For this blog I've security disabled, though obviously in production you'd definitely want to enable it 7. Configure the ConnectionSettings as per your requirement (refer to the guide for more detail on settings), below I'm noting the settings that will require configuration in its most minimum form (I've also attached my complete AdoThing.config file for reference) "rows": [       {         "Address": "localhost",         "Port": 8080,         "Resource": "/Thingworx/WS",         "IsSecure": false,         "ThingName": "AdoThing",         "AppKey": "f7e230ac-3ce9-4d91-8560-ad035b09fc70",         "AllowSelfSignedCertificates": false,         "DisableCertValidation": true,           "DisableEncryption": true       }     ] 8. Configure the connection string for the SQL Server in following section, in the same file opened above     "rows": [       {         "ConnectionType": "OleDb",         "ConnectionString": "Provider=SQLNCLI11;Server=localhosts\\SQLEXPRESS;Database=TWXDB;Uid=sa;Pwd=login123;",         "AlwaysConnected": true,         "QueryEnabled": true,         "CommandEnabled": true,         "CommandTimeout": 60       }     ] 9. Just to highlight what's what in ConnectionString above: "ConnectionString": "Provider=SQLNCLI11;Server=<Machine/ClientName>\\SQLServerInstanceName;Database=<databaseName>;Uid=<userName>;Pwd=<password>;" 10. To get correct connection string syntax for different source refer to the ConnectionStrings.com 11. Save the file 12. Navigate to the windows services by opening Windows Start > Run > services.msc 13. Check for the service ThingWorx .NET ADO Client as you'll have to start it if it's set to Manual, like so in my case Following message will be logged on successful connection  in the DotNETSDK -X-X-X.log : [Critical] twWs_Connect: Websocket connected! At the end of the blog I'll share some of the errors that I came across while working on this and how to go about addressing them. Creating and connecting to Remote Database Thing Now, let's navigate to the ThingWorx Composer and create a Thing with RemoteDatabase Template to consume the resource created above in the form of ADO Service. I've named my thing as AdoThing while creating it in ThingWorx Composer, which matches with the ThingName used in the AdoThing.json file. If everything went through as needed you should see the isConnected = true in the AdoThing's Properties section. Since, this is a Database thing I can now go about creating all the required services concerning the Create, Update, Delete (CRUD) operations, just like for any database for created using the RDBMS Connector. Handling errors while setting up the ADO Service Here are some of the errors that I encountered while setting up the ADO service for this blog: Error 1: com.thingworx.ado.AdoThing Cannot connect to database. : System.Data.OleDb.OleDbException: Login timeout expired Note: Logged in DotNetSDK-X-X-X.log Cause & Resolution: - Service is not able to successfully reach or authenticate against the SQL Server Express DB instance - Ensure that the TCP/IP is enabled for the Protocols for the SQL Express, as I have shared in the screenshot above - Make sure that the username / password used for authenticating with the database is correctly provided while configuring the settings for the OLEDB section in    AdoThing.config Error 2: com.thingworx.ado.AdoThing GetTables OleDbException error : System.Data.OleDb.OleDbException Note: Logged in Application.log from ThingWorx platform Cause & Resolution - This exception is thrown when user attempts to check for the available tables, while creating the service in the ThingWorx Composer - Resolution to this is similar to that mentioned above for Error 1 Error 3: [U: SYSTEM] [O: com.thingworx.ado.AdoThing] OleDbException [code = -2147217865, message = Invalid object name 'TWXDB.DemoTable'.] executing SQL query Note: Logged in Application.log from ThingWorx platform while testing/executing the SQL service created in the ThingWorx Composer Cause & Resolution - The error is due to the usage of DB name in front of the table name, it's not required since the DB name is already selected in the connection String Error 4: [O: com.thingworx.Configuration] Could not read configuration file. : Newtonsoft.Json.JsonReaderException: Bad JSON escape sequence: \S. Path 'Settings.rows[0].ConnectionString', line 656, position 71. Note: Logged in DotNetSDK-X-X-X.log Cause & Resolution - This is caused due to the "ConnectionString": "Provider=SQLNCLI11;Server=<machineNameOrIP>\SQLEXPRESS;Database=TWXDB;Uid=sa;Pwd=login123;", - Json requires this to be escaped thus switching to "ConnectionString": "Provider=SQLNCLI11;Server=<machineNameOrIP>\\SQLEXPRESS;Database=TWXDB;Uid=sa;Pwd=login123;", resolved the issue - Among many other, https://jsonformatter.curiousconcept.com/​ is quite helpful in weeding out the issues from the JSON syntax Error 4: [O: com.thingworx.ado.AdoClient] Error while initializing new AdoThing, or opening connection to Platform. : System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.     at com.thingworx.communications.client.TwApiWrapper.twApi_Connect(UInt32 timeout, Int32 retries)     at com.thingworx.communications.client.TwApiWrapper.Connect(UInt32 timeout, Int16 retries)     at com.thingworx.communications.client.BaseClient.start()     at com.thingworx.ado.AdoClient.run() Note: Logged in DotNetSDK-X-X-X.log Cause & Resolution - This error is observed when using FIPS version of the  ADO Service, esp. when downloaded from the ThingWorx Marketplace - Make sure to recheck the SSL configuration - When not using SSL check that the x64 and x86 directories only contain twApi.dll as by default FIPS version contain two additional dlls i.e. libeay32.dll & ssleay32.dll in both x64 & x86 directories

ThingWorx 7.4 covers the following areas of the product portfolio:  ThingWorx Analytics, ThingWorx Utilities and ThingWorx Foundation which includes Core, Connection Server and Edge capabilities. Key Functional Highlights Highlights of the release include: Source Integration: Improved integration framework making it easy to connect with external systems, including a standard Windchill connector, to retrieve data on demand. Industrial Connectivity: New Industrial Gateway and Discover navigation tool simplifying the mapping of tags to properties, including performance enhancements for data updates. Edge/CSDK: Build process improvements, Subscribed Property Manager (SPM) enhancements, asynchronous service requests and TLS updates to increase developer productivity, improve application performance and strengthen security. AWS IoT Connector: The latest version 1.2 of the Connector allows customers to more fully leverage their investment in AWS IoT. It features improved deployment automation via CloudFormation and automatic extension installation, ThingWorx Edge JavaScript SDK for use with the Connector with support for properties, services and events, and just-in-time certificate registrations. Contextualize Next Generation Composer: Re-imagined Composer using modern browser concepts to improve developer efficiency including enhanced functionality, updated user interface and optimized workflows. Engage These features will be available at the end of March 2017. Advanced Grid: New grid widget with improved design, context menu, multi-column sorting, global search and many more common grid-related features. Tree Grid: New tree-grid widget with same features as advanced grid plus ability to pre-load tree levels, dynamically load child data and auto expand all nodes to build more powerful mashups. Administer & Manage MS SQL Server: New persistence provider for model and run-time data providing customers an alternative to PostgreSQL. Security: ThingWorx worked with industry standard security scanning and auditing tools to identify and correct all non-trivial, and many trivial, vulnerabilities to ensure secure software best practices. Licensing: Link ThingWorx to PTC systems of record to manage user entitlement and provide usage information and auditing capability critical to TWX, PTC and its partners.  Documentation ThingWorx 7.4 Reference Documents ThingWorx Core 7.4 Release Notes ThingWorx Core Help Center ThingWorx Edge SDKs and WebSocket-based Edge MicroServer Help Center ThingWorx Connection Services Help Center ThingWorx Utilities Help Center Additional information ThingWorx eSupport Portal ThingWorx Developer Portal ThingWorx Marketplace Download ThingWorx Platform – Select Release 7.4 ThingWorx Edge C SDK 1.4 – Select Most Recent Datecode, C-SDK-1-4-0 ThingWorx AWS IoT Connector 1.2 – Select Release 7.4 ThingWorx Utilities – Select Release 7.4

ThingWorx 7.4 introduces a new licensing system. A license file (license.bin) needs to be placed in the ThingworxPlatform folder. A new license file is also required if you upgrade from 7.4  to a major or minor release (not service pack-level releases). For example: • If you are using version 7.3, a license is not required. • If you upgrade from version 7.4.1 to version 7.4.2, a license upgrade is not required. • If you upgrade from version 7.4.3 to version 7.5.2, a license upgrade is required. Refer to the Installing ThingWorx 7.4 guide or Upgrading ThingWorx 7.4 guide for detailed process steps. Paid customers would have unlimited use of entities for 7.4.0. As currently a license file is locked to  version rather than SCN/host and is part of download package on  PTC Support, customers can use the same downloadable for multiple instances. Developer Trial Edition provides a constrained license file (5 users, 100 things, 120 days), and the license file is part of on premise download package on Dev Portal. Developer Trial Edition for Manufacturing (Kinex) provides a constrained license file (5 users, 100 things, no Composer access), and license file is part of download package on Kepware Portal. A new Licensing Subsystem is now available. Licensing subsystem services include: -AcquireLicense– service allows for retrieval of feature entitlements in license.bin, used when new license dropped in folder (no need to instance restart) –GetCurrentLicenseInfo – returns info on current license file –GetRemainingDaysInLicense –used for trial editions –GetLicenseUsageData – returns nformation about user’s license usage –PurgeLicenseUsageData –deletes the license usage data that is two years and older

This is just a quick reference on how to install pgadmin 3 if the autoinstall with yum command (sudo yum install pgadmin3) fails. Two routes would be available. Try running yum list pgadmin* ​If you see something like this: that means the package is available in the highlighted repository, you'd just need to add it. rpm -Uvh http://yum.postgresql.org/9.4/redhat/rhel-7-x86_64/pgdg-redhat94-9.4-3.noarch.rpm After that, try sudo yum install pgadmin3_94 (insert your actual version) 2. If you would like a different version (or the latest one) of pgadmin, what you could do is grab the .tar.gz file from here https://www.postgresql.org/ftp/pgadmin3/release/ Then manually install it. For example for version 1.22.2 (the version is just for demoing purposes – I grabbed the top one available in the list): mv pgadmin3-1.22.2.tar.gz /usr/local/src cd /usr/local/src tar –zxvf pgadmin3-1.22.2.tar.gz cd pgadmin3-1.22.2 ./configure make make install Then you would need to configure your server to allow remote user access of the database using pgadmin. Two config files would need to be modified: Open up the postgresql.conf configuration file.  Do a search or find for the phrase ‘listen_addresses’ without quotes.  In order to open the access up to all IP addresses change the value to a *.  The default is set to ‘localhost’ which does not allow connection from remote computers. Next open the pg_hba.conf configuration file. Scroll down to the bottom of the file to the section marked # IPv4 local connections.  Add in the following code on its own line, just underneath the 127.0.0.1/32 line necessary for ‘localhost’. host all all youripaddress/32          trust This will allow for local access of the database server to the computer with the IP address you specified.  To add additional remote computers simply add a new line with their appropriate IP address. Now that your configuration is complete restart the PostgreSQL database server for the changes to take effect. su – su postgres pg_ctl restart –D /usr/local/pgsql/data

Embedded databases come with the installation of the ThingWorx Platform No additional installation or configuration is required for embedded databases Read about the various benefits and pitfalls of embedded versus external below Database Options H2 RDBMS (relational database management system), written in Java Has a small memory footprint Embedded into ThingWorx for easy installation Not as robust as other database options Not scalable in production environments (unless used alongside a separate, external database for stream, value stream, and other data) ​ See KCS Article CS243975 for further reading on the use of external databases Meant to be used for quick deployments and testing environments PostgreSQL ORDBMS (object-relational database management system), written in C PostgreSQL is the ThingWorx recommended database for production systems More Robust External database installed separately from ThingWorx Beneficial because external databases can be specifically configured for use in production, while embedded databases cannot Able to efficiently handle larger amounts of data and store more data without affecting ThingWorx system performance Greater Stability Recover from data corruptions more easily by accessing the database from an external application (separate from ThingWorx) using simple SQL statements Easier to back-up the database in case of issues (further reading in KCS Article CS246598) Less risky and simpler upgrade procedure, which occurs "in-place" Instead of exporting and importing data and entities, a simple schema update allows these to automatically persist into the new version If ThingworxStorage folder is accidentally deleted, entities and data are secure in the external database More Secure HA (High Availability) allows for multiple server instances at different locations in the network Assists in time of failover, i.e. if one server fails, the other can immediately take over Secures the data and prevents further data loss in the event of a failure Customizable security settings and complex password requirements Fewer security vulnerabilities than other databases Because Postgres is an external database, it can be harder to install Follow the steps in the installation guide closely See KCS Articles CS235937 and CS230085 for troubleshooting and help with installation and configuration Hana RDBMS (relational database management system) In-memory, column based data storage For more information on this database, please see the Getting Started with SAP HANA Guide Neo4J GDBMS (graph database management system), written in Java Data is not easily accessed by external applications, and CQL must be used instead of SQL, making recovery from corruptions very difficult Embedded database with limited configuration options Known to have issues with deadlocks Deprecated in version 7.0 (related KCS Article: CS228537) For full installation steps for H2 and PostgreSQL, see the ThingWorx Installation Guide

Hi everybody, In this blogpost I want to share with you my local ThingWorx installation, with some optimizations that I did for local development. -use the -XX:+UseConcMarkSweepGC . This uses the older Garbage Collector from the JVM, instead of the newer G1GC recommended by the ThingWorx Installation guide since version 7.2. The advantage of ConcMarkSweepGC is that the startup time is faster and the total memory footprint of the Tomcat is far lower than G1GC. -use -agentlib:jdwp=transport=dt_socket,address=1049,server=y,suspend=n. This allows using your Java IDE of choice to connect directly to the Tomcat server, then debugging your Extension code, or even the ThingWorx code using the Eclipse Class Decompilers for example. Please modify the 1049 to your port of choice for exposing the server debugging port. -use -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=60000 -Dcom.sun.management.jmxremote.ssl=false                  -Dcom.sun.management.jmxremote.authenticate=false           This sets up the server to allow JMX monitoring. I usually use VisualVM from the JDK bin folder, but you can use any JMX monitoring tool.           This uses no Authentication, no SSL and uses port 6000 - modify if you need. I usually startup Tomcat manually from a folder via startup.bat, and the setenv.bat looks like: set JAVA_HOME=C:\Program Files\Java\jdk1.8.0_102 set JRE_HOME=C:\Program Files\Java\jdk1.8.0_102 set THINGWORX_PLATFORM_SETTINGS=D:\Work\servers\apache-tomcat-8.0.33 // this is where the platform-settings.json file is located set CATALINA_OPTS=-d64 -XX:+UseNUMA -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8 -agentlib:jdwp=transport=dt_socket,address=1049,server=y,suspend=n -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=60000 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false In this mode I can look at any errors in almost real time from the console and it makes killing the server for Java Extension reload a breeze -> Ctrl+C Please don't hesitate to provide feedback on this document, I certainly welcome it. Be warned: THESE ARE NOT PRODUCTION SETTINGS. Best regards, Vladimir

This Blog presents a simple Java utility to validate the deployment of ThingWatcher. It is important to note that the utility used is not a real life situation, the intent was to keep it as simple as possible in order to achieve its aim: validation of the deployment. An understanding of Java IDE (such as Eclipse) is necessary in order to run the utility with relevant dependency and classpath setup. Those are beyond the scope of this posting. We will cover the following points: Pre-Requisites Using the sample utility Code walk through Validate training job creation Validate model job creation Update for ThingWorx Analytics 8.0 Pre-requisites A strict adherence to the ThingWatcher deployment guide is recommended in order to first deploy training and model microservices as well as to familiarize yourself with ThingWatcher APIs. Prior to testing ThingWatcher, both the training and model microservices should be up and running The media for ThingWatcher (including model and training micro-service) should be downloaded from PTC Software Download page . The commands to deploy the micro-services will vary depending on the platform used and are presented in the ThingWatcher deployment guide. As a reference example, on Windows the command will be similar to the following: Start Docker: Start > Program > Docker > Docker Quick Start Terminal Load model micro service tar $ docker load < "D:\PTC\MED-61147-CD-522_F000_ThingWorx-Analytics-ThingWatcher-52-2\components\ModelService\ModelService\model-service.tar"     3. Install model service: $ docker run -d -p 8080:8080 -v '/d/TWatcherStorage/model:/data/models' -v '/d/TWatcherStorage/db:/tmp/' twxml/model-service:1.0 -Dfile.storage.path=/data/models -jar maven/model-1.0.jar server maven/standalone-evaluator.yml     4. Load training micro service tar file                         $ docker load < "D:\PTC\MED-61147-CD-522_F000_ThingWorx-Analytics-ThingWatcher-52-2\components\TrainingService\TrainingService\training-service.tar"     5. Install training service                         $ docker run -d -p 8090:8080  twxml/training-service:1.0.0  -Dmodel.destination.uri=model://192.168.99.100:8080/models -jar maven/training-standalone-1.0.0-bin.jar server /maven/training-standalone-single.yml Note: the -Dmodel.destination.uri points here to the model micro-service host. To find the ip address, enter docker-machine ip on the model micro-service docker machine.     6. Validate micro-services deployment: Execute docker ps  and confirmed that both services are up, as in the following example: CONTAINER ID        IMAGE                          COMMAND                      CREATED            STATUS              PORTS NAMES 5b6a29b95611        twxml/training-service:1.0.0  "java -Dmodel.destina"  13 days ago        Up 44 minutes      8081/tcp, 0.0.0.0:8090->8080/tcp  modest_albattani 8c13c0bc910e        twxml/model-service:1.0        "java -Dfile.storage."      2 weeks ago        Up 44 minutes      0.0.0.0:8080->8080/tcp, 8081/tcp  thirsty_ptolemy   Using the sample utility Download the attachment Main.java Import Main.java into Eclipse (or IDE of choice) with the ThingWatcher dependencies added in classpath. Update the trainingBaseURI (see below) to points to the training micro-services. The utility should be ready to execute. Code walk through The code declares a thingwatcher in the following snippet: ThingWatcher thingwatcher = new ThingWatcherBuilder() .certainty(90.0) .trainingDataDuration(60) .trainingDataDurationUnit(DurationUnit.SECOND) .trainingBaseURI("http://192.168.99.100:8090/training") .getThingWatcher(); In the above code it is important to update the trainingBaseURI argument with the correct ip address and port for the training micro-service host. The code then loops 10000 times and sends a new value, which simulates the sensor data, at a simulated 100 ms interval. The value is computed as Math.sin(i) for the whole calibrating phase and most of the monitoring phase too. We artificially introduce an anomaly by sending a value of Math.incremetExact(i) between the 9000 th and 9900 th iterations. During the Monitoring phase, the code logs the value, the anomalous status and the thingwatcher state. It is advised to save the output to a file in order to review the logging once the utility has run. In Eclipse this can be done by selecting the Main.java with right mouse button > Run As… > Run Configuration > Common and tick Output File under the Standard Input and Output, and specify a location for the output file. A review of the output log file will shows that somewhere between timestamp 900000 and 990000, the isAnomalousValue is true. Note that this does not starts and ends exactly at 900000 and 990000, as ThingWatcher needs a few occurrences before reporting it as anomaly. Sample output indicating an anomalous state: [main] INFO com.thingworx.analytics.demo.Main - Value = 901700,9017.0,-9016.403802019577 [main] INFO com.thingworx.analytics.demo.Main - isAnomalousValue = true [main] INFO com.thingworx.analytics.demo.Main - ThingWatcherStat = MONITORING As part of validating the successful deployment of ThingWatcher, it is recommended to validate the correct creation of a training and model job. Validate training job creation In order to validate the successful creation of a training job, execute a GET request to the training micro service : http://192.168.99.100:8090/training (update the ip address to the one on your system) This should return a COMPLETED job whose body starts with something similar to: Validate model job creation In order to validate the successful creation of a model job, execute a GET request to http://192.168.99.100:8080/models (update the ip address to the one on your system) to see all the models that have been created. For example: Alternatively, click (or use) the URI reported in the training job output, here http://192.168.99.100:8080/models/6/pmml.xml, to see the complete model definition. The output will be similar to: When this sample test runs correctly, the ThingWatcher deployment has been validated. Update for ThingWorx Analytics 8.0 Deploying the microservices, see Video Link : 1937 Updated Java code: see Does anyone know how to use java api to achieve anomaly detection with Thingwatcher8.0? To Note: The utility provided is for testing purpose only. The code does not represent any kind of best practice and is not meant to be a perfect java coding example. It is provided as is with no guarantee.

In this video we cover the process of installing ThingWorx Analytics Server 52.1. Make sure to have reviewed the part 1 video about pre requisite   Updated Link for access to this video:  Installing ThingWorx Analytics Server: Part 2 of 2

In this video we review the prequisite needed prior of installing ThingWorx analytics server 52.1   Updated Link for access to this video:  Installing ThingWorx Analytics Server: Part 1 - Prerequisites

Let's consider that we have two Streams Stream1 and Stream2 with same DataShape StreamDS. DataShape StreamDS has two fields Id (number) and Name (string). We want to copy all the entries from Stream1 to Stream2. Steps: 1. Open Stream1 Stream in Composer and run GetStreamEntriesWithData service. 2. In the popup click on Create DataShape from Result option to create a new DataShape GetStreamEntriesDS. 3. Create a Service and use JavaScript like below (Added Comments for Details): // Create Temporary Infotable to hold output of GetStreamEntriesWithData Service var paramsForInfotable = {   infoTableName: "InfoTable" /* STRING */,   dataShapeName: "GetStreamEntriesDS" /* DATASHAPENAME */ }; // result: INFOTABLE var InfotableForCopy = Resources["InfoTableFunctions"].CreateInfoTableFromDataShape(paramsForInfotable); //Save output of GetStreamEntriesWithData Service to Temporary Infotable InfotableForCopy var paramsForGetStreamEntriesWithDataService = {   oldestFirst: false /* BOOLEAN */,   maxItems: 10000 /* NUMBER */ }; // result: INFOTABLE dataShape: "GetStreamEntriesDS" InfotableForCopy = Things["Stream1"].GetStreamEntriesWithData(paramsForGetStreamEntriesWithDataService); // Read the data from Infotable row by row and add it to new Stream var tableLength = InfotableForCopy.rows.length; for (var x = 0; x < tableLength; x++) {   var row = InfotableForCopy.rows ; // values:INFOTABLE(Datashape: StreamDS) var values = Things["Stream2"].CreateValues(); values.Id = row.Id; //NUMBER values.Name = row.Name; //STRING var paramsForAddStreamEntryService = {   sourceType: row.sourceType /* STRING */,   values: values /* INFOTABLE*/,   location: row.location /* LOCATION */,   source: row.source /* STRING */,   timestamp: row.timestamp /* DATETIME */,   tags: row.tags /* TAGS */ }; // AddStreamEntry(tags:TAGS, timestamp:DATETIME, source:STRING, values:INFOTABLE(StreamDS), location:LOCATION):NOTHING Things["Stream2"].AddStreamEntry(paramsForAddStreamEntryService); } var result = InfotableForCopy;

In this blog we will have a look at the installation of the Thingworx Analytics Builder extension. This is used as guideline but make sure to check the Help Center for your release as steps do vary with versions. The installation has been divided in 3 parts: Introduction and import of the extension into Thingworx Platform Video Link : 1568 Configuration of the extension Note: For release 8.1, the Settings menu differs from previous versions, seeWhat's New in ThingWorx Analytics Builder 8.1 between times 00:12 sec to 00:40 sec for up to date menu selection. Video Link : 1572 Installation of the UploadThing module Note: this step no longer applies as of ThingWorx Analytics 8.1 Video Link : 1573 Useful links: PTC Download page for Thingworx Analytics PTC Reference Document page for Thingworx Analytics How to copy files from Windows to Linux ?