Load Testing through Remote Device Simulation
Designing an enterprise-ready application requires extensive testing and quality assurance. This includes all sorts of tests, of course, from examining the user interface for flaws to verifying there is correct logic in all background services. However, no area of testing is more important than scalability. Load testing is how to test the application to ensure it still functions as desired when remote things are connected and streaming information to the Platform.
Load testing is considered a critical component of the change management process. It is mentioned numerous times throughout PTC best practice documentation. This tutorial will step you through designing a load test using Kepware as a simulator. Kepware is free to download and use in short demos, making it the perfect tool for this type of test.
Start by acquiring the latest version of Kepware from the download site. Click “Download Free Demo” if a license was not included in your PTC product package. The installation of Kepware is simple, and for details, see the Kepware Installation Guide. The tutorial shown here uses Kepware version 6.7 and ThingWorx version 8.4.4. Given that we are testing a ThingWorx application, this tutorial assumes ThingWorx is already installed and configured correctly.
Once Kepware is installed, follow these steps:
(This tutorial was developed by Desheng Xu and edited by Victoria Tielebein. Exact specifications of the equipment used in both large scale and local tests are given in step VI, which discusses the size of the simulation)
Understand how to configure Kepware as a simulator
Go to the Help menu within Kepware, and click on “Driver Help”
Select “Simulator” in the pop-up window, and click “OK”
Expand “Address Descriptions” and then “Simulation Functions”
Select “Ramp Function” to review details about the function needed for this tutorial, as well as information about function syntax
Close the window once this information has been reviewed
Create a new project in Kepware
Click “File” > “New”
In case you are connected to runtime, Kepware will allow you to choose to edit this project offline
Add a channel in Kepware
Channels represent threads which Kepware will use to contact ThingWorx
Under “Connectivity”, click “Click to add a channel.”
From the drop-down list, select “Simulator”
Use all the default settings, selecting “Next” all the way down to “Finish”
Next, add one device to the channel
Highlight the new channel and click “Click to add a device” (which will appear in the center of the screen)
Once again, use the default settings, selecting “Next” all the way down to “Finish”
Add a tag to this device Within Kepware, tags represent properties which bind to remote things on the Platform and update with new information over time. Each device will need several tags to simulate remote property updates. The easiest way to add many tags for testing is to create one, and then copy and paste it.
Highlight the device created in the previous step and click “Click to add static tag”, which appears in the center of the screen
For “name” type “tag1”
For Address, enter the Ramp function: RAMP(1000,1,2000000,1)
The first parameter is the update rate given in milliseconds
The next two parameters are the range of values which can be sent
The last parameter is the increment or step
Together this means that every 1 second, this tag will send a new value that is 1 higher than the previous value to the Platform, starting at 1 and ending at 2 million
Ensure the Data Type is given as “DWord” or any type which will be read as a “Number” (and NOT an “Integer”) on the Platform
Change the Scan Rate to 250
Then click “OK”
Add more devices to the test The most basic set-up is now done: if this project connected to the Platform, one remote thing with one remote property could be used to simulate property updates. That is not very useful for load testing, however. We need many more things than this, and many more properties. The number of tags on each device should match the expected number of remote properties in the application itself. The number of devices in each channel should be large enough that when more channels are created, the number of total devices is close to the target for the application. For example, to simulate 10,000 things, each with 25 remote properties, we need 25 tags per device, 200 devices per channel, and 50 channels. This would require a lot of memory to run and should not be attempted on a local machine.
A full test of 40 channels each with 10 devices was performed as shown in the screenshots here. This simulates 10,000 writes per second to the Platform total, or about 400 remote device connections. This test used the following hardware specifications:
Kepware machine running Windows 2016 64-bit, 2 cores, 8G
ThingWorx Platform machine running Ubuntu 16.04, 4 cores, 16G
PostgreSQL 9.6 machine running Ubuntu 16.04, 4 cores, 16G
Influx 1.6.3 machine running Ubuntu 16.04, 4 cores, 16G
A local test was also run on Windows 10 (64-bit), using the H2 database, with Kepware and ThingWorx running side by side on the machine, 4 cores, 16G. This test made use of only 2 channels, with 10 devices each. For local tests to see how the simulation works, this is fine, but a more robust set-up like the above will be needed in a true load test.
If there is not enough memory on the machine hosting Kepware, errors like this will appear in the Kepware logs: One or more value change updates lost due to insufficient space in the connection buffer. Once you decide on the number of tags and devices needed, follow the steps below to add them.
To add more tags, copy and paste the existing tag (ctrl+c and ctrl+v work in Kepware for convenience) until there as many tags as desired
To add more devices, highlight the device in Kepware and copy and paste it as well (click on the channel before pasting)
Then, copy and paste the entire channel until the number of channels, devices, and tags totals the desired load (be sure to click on “Connectivity” before hitting paste this time)
Configure the ThingWorx connection
Right click on Project in the left-hand navigation bar and in the pop-up window that appears, highlight ThingWorx
Change the “Enable” field to “Yes” to activate the other fields
Fill in the details for “Host”, “Port”, “Application Key”, and “Thing name”
Note that the application key will need to be created in ThingWorx and then the value copied in here
The certificate and encryption settings may also need to be adjusted to match your environment
For local set-ups, it is likely that self-signed and all certificates will need to be accepted, so both of those fields will likely need to be set to “Yes” (Encryption may need to be disabled as well). In production systems, this should not be the case
Save the project It doesn’t matter too much if this project is saved as encrypted or not, so either enter a password to encrypt the save or select “No encryption”
Connect to ThingWorx
Click “Runtime” > “Connect…”
A pop-up will appear asking if you want to load this project, click “Yes”
The connection status should then appear in the bottom portion of the window where the logs are displayed
Configure in ThingWorx
Login to the ThingWorx Platform
Under “Industrial Connections” a thing should appear which is named as indicated in the Kepware configuration step above
Click to open this thing and save it
Also create a new thing, a value stream for ingesting data from Kepware
Create remote things in ThingWorx
Import the provided entity into ThingWorx (should appear as a downloadable attachment to this post)
Open the KepwareUtil thing and go to the services tab
Run the AutoKepwareCreate service to generate remote things on the Platform
Give the name of the stream created above so each thing has a place to store property information
The IgnoreTemplate flag should be set to false. This allows for the service to create a thing template first, which is then passed to the remote devices. The only reason this would be set to true is if the devices need to be deleted and recreated, but the template does not (then set the flag to true). To delete the devices, use the AutoKepwareDelete service also provided on the KepwareUtil thing
Note that the AutoKepwareCreate service is asynchronous, so once it is executed, close the window and check the script logs to see when it completes. The logs will look like: KepwareUtil AutoKepwareCreate task finished!!!
Check status of remote things
Once the things are created, they should automatically connect to the Platform
Run the TotalDeviceByTemplateWithTemplate service to see if the things are connected
The template given here could be the one created by the AutoKepwareCreate service, or just give it RemoteThings if this is a small local set-up without many remote things on it
The number of devices will equal the number of devices per channel times the total number of channels, which in the test shown here, is 400
isConnected will be checked if all of the devices are connected without issue
If some of them are not connected, verify in the logs if there are any errors and resolve those before moving on
View Ingestion Rate
Once the devices are created, their tags should show as numbers (NOT integers), and they should already be updating with new values every second
To view the ingestion rate, run the KepwareUtil service AutoKepwareRateSummary
Give the thing template name that is created by the AutoKepwareCreate service, which will look like the name of the Kepware thing itself with a “T-“ in the front
The start time should be close to the current time, and the periodInMinutes should be large enough to include some of the test (periodInMinutes is used to calculate the end time within the service)
Note in the results here that the Average Write Per Second is only 9975 wps, which is close but not exactly what we would expect. This means that there are properties not updating correctly, which requires us to look at the logs and restart some things. If nothing shows up here, despite the Total Connected Things showing correctly, then look at the type of the tags on one of the remote devices. The type must be NUMBER for the query within this service to work, and not INTEGER. If the type of the tags is incorrect, then the type of the tags within Kepware was probably given as something which is not interpreted as a number in ThingWorx. Ensure DWord is used for the tags in Kepware
Within the script log, look for any devices which show errors as seen in the image below and restart them to get their properties updating correctly
Once the ingestion rate equals what is expected (in the case of the test here, 10,000 wps), use the AutoKepwareIngestionStat service on the KepwareUtil thing to see details about each remote device
The TimeGapAvg in this service represents the gap between two ingestions in milliseconds, showing any lag that may be present between Kepware and ThingWorx
The TimeGapSTD shows the standard deviation of the time gap between two ingestions on any given thing, also indicating lag (the lower this number, the better)
The StartTime and EndTime show the first and last timestamp observed in the ThingWorx database during the given duration
The totalCount shows the total number of ingested records during the sampling cycle
The StartValue and EndValue fields show the first and last value ingested into the tag during the given duration
If the ingestion rate is working as expected, and the ramp function is actually sending an update on time (in this case, once each second), then the difference between the EndValue and StartValue should always be equal to the totalCount plus 1. If this doesn’t match up, then there may be data loss or something else wrong with the property updates, which will show as a checked box in the valueException column. It is not enough to ensure that the ingestion rate is correct, as sometimes the rate may fluctuate only by 1 or 2 wps and appear perfect, even while some data is lost. That is why it is important to ensure that there are no valueException boxes showing as checked in the test of the application. If none of these are marked as having failed, then the test was successful and this ingestion rate is acceptable for the application
This tutorial is a very basic way to simulate many remote devices ingesting data into the Platform. For this to be a true test of the application, the remote things created in this test will need to be given business logic tasks as well. The AutoKepwareCreate service can be modified to give any template (and not just RemoteThing) to the thing template which is created and subsequently passed into the demo devices. Likewise, the template itself can be created, and then manually modified to look like the actual remote device template in the application, before the rest of the things are created (using the IgnoreTemplate flag in the creation and deletion services, as discussed above).
Ensure that events are triggered as expected and that subscriptions to property updates are in place on the thing template before creating the demo things. Make use of the subsystem monitor to ensure that the event, value stream, and stream queues do not grow so large that the Platform cannot keep up with the requests (for details about tuning the stream and value stream processing subsystems, see PTC’s best practice documentation). Also be sure to load some of the mashups to see how they perform while the ingestion test is happening. This will test whether or not the ingestion rate and business logic of the application can function side by side without errors, data loss, or performance issues.
View full tip
