Best Strategy to Log High-Frequency Properties Periodically in ThingWorx?

[leekhil_kataray]

Hi Community,

I’m using ThingWorx 10.0.0 and working with high-frequency properties named property1 and property2 that update very often.
If these are marked as Logged = TRUE, they generate a large amount of history data and can impact system performance.

Proposed Approach

Before I implement anything, this is the strategy I’m considering to log values only every X minutes:

1. Keep original properties unlogged (Logged = FALSE).
2. Create separate periodic properties (Logged = TRUE, DataChange = ALWAYS).
3. Use a Timer to write values to these periodic properties every X minutes:

me.property1_periodic = me.property1;
me.property2 = me.property2;

This would store history only at fixed intervals instead of every update.

Looking for Suggestions

Is this a good approach?
Are there any built-in features or alternative strategies for periodic logging that I should consider before implementing this?

I’d appreciate any recommendations or best practices from the community.

[slangley · Accepted solution]

Hi @leekhil_kataray 

Your strategy for managing high-frequency properties in ThingWorx 10.0.0 by logging values at fixed intervals is a solid approach to minimize the impact on system performance. Here are the key points to consider:

Unlogged Original Properties: Keeping your original properties (property1 and property2) unlogged (Logged = FALSE) prevents excessive data generation from frequent updates.

Periodic Properties: Creating separate periodic properties that are logged (Logged = TRUE) and configured with DataChange = ALWAYS allows you to capture the state of these properties at defined intervals, effectively controlling the volume of historical data.

Timer Implementation: Using a Timer to update these periodic properties every X minutes is a practical solution. This method aligns with best practices for data-centric modeling, where controlled data logging is encouraged.

Performance Considerations: Logging high-frequency data can indeed impact system performance. Your approach of logging at fixed intervals helps mitigate this risk. However, be aware of potential minor data loss due to batch processing for frequently changing properties.

Monitoring and Adjustments: After implementing your strategy, monitor the system's performance and adjust the logging frequency as needed based on observed performance and data requirements.

Alternative Strategies: While the context does not specify built-in features for periodic logging, consider using streams for logging time-driven events if applicable. Streams are optimized for high-speed writing and can help manage data more effectively.

Overall, your approach is well thought out, and testing it in a controlled environment is advisable to ensure it meets your performance and data retention needs. For further insights, you may want to refer to the following resources:

• Best Practices for Developing Applications
• Performance Report

Regards.

--Sharon

[leekhil_kataray]

Thanks Slangley
Will test before implementing this on production environment.

[slangley]

Hi @leekhil_kataray 

Did you get a chance to test?  If so, please remember to mark the solution in the post, or let us know if you have other questions.

Regards.

--Sharon

[leekhil_kataray]

Hi Slangley,

We are still in discussion with the customer, and the logging strategy has not yet been finalized. If we finalize the strategy as periodic logging, I will test this approach.

Thanks.

[VladimirRosu]

Hi @leekhil_kataray 

What exactly is the update rate of these properties that resulted in the fact "they generate a large amount of history data and can impact system performance"?

The property logging mechanism is quite efficient, so I'm curious to understand the details around this.

The feasibility of your approach depends on whether your use-case allows loosing property values between the timer ticks, which is up to you to assess.

[leekhil_kataray]

Hi Vladimir,

The frequency is less than one second. In addition, I have 10 properties that are being logged at this frequency. Altogether, there are around 200 entities, each with high-frequency properties like these.

We are currently in discussions with the customer regarding logging strategies. If the customer asks for periodic logging, Thingworx do not have this type of  logging. Therefore, I wanted to check your thoughts on the above strategy.

[VladimirRosu]

Hi @leekhil_kataray ,

Whenever estimating the load of the system, you should calculate the Property Writes per Second (aka WPS or Writes per Second in our terminology), which in this case would be 200 Entities * 10 properties * 2 Hz = 4000 WPS.

If you look at the performance tests available in the Help Center, here and here, you will observe that we tested the platform with values far higher.

While in reality ThingWorx's performance depends not only on data ingested, but also by the user count and their activity on the platform, and the backend data processing done, I characterize 4000 WPS as a low value, for which I would not want to optimize anything. Note that, however, you should never approach a new solution with the idea that you will store all data forever in either DB or Influx - you must have a hot and cold data strategy. Influx, just as an example, has a retention period for its buckets which I usually recommend to be as low as possible, so that we don't end up storing 6 years worth of data in essentially a hot system.

In this situation what were the exact concerns about the amount of data or system performance?

Note: In the case you know this value and it's not good enough by whatever reason, there are a few strategies to optimize without resorting to any complex logic: 

1. If you set the Property's DataChange as Value (which is the default setting), just below it you have something called "Data Change Threshold". Set this to a (absolute) value good enough so that it does not log new values when they arrive in the platform. If a Data Change won't be triggered, also the property won't be logged.

2. If you have these properties as Kepware you can configure the property at point 1 directly in the Kepware server, by setting the Push Threshold (so that they won't even arrive in the platform) and additionally you can even set the ScanRate, which means how often ThingWorx will retrieve these values from Kepware - essentially achieving this periodical scanning.

As you see, there are a few ways in which you can optimize even in the case when 4000 WPS are too much.