it's never been properly explained what the value/purpose of the ThingBackupStorage folder is/does, or how to manage the size of both storage folders. Obviously the Neo4j database grows in size as more Thing persistent properties, valuestreams etc are saved, but there is no guidance on how to maintain the core database (that I can see). For example, is there a way to 'purge' a test ThingWorx database of it's values easily? Is anyone from the ThingWorx team able to enlighten us on this thanks?
The ThingworxBackupStorage holds a backup of the Thingworx database and is updated on each Backup that you run, this is a service within the platform and by default that should be running (Look for something called SystemBackup which is a Scheduler Thing)
There is no auto-archiving/purging within the platform, so for each Data Store (Stream, DataTable, ValueStream, Blog, Wiki) you will need to build purge functionality.
thanks - so hence it's probably very important to locate your Thingworx storage folders on a non-OS partition then(!). Luckily they can be moved relatively straightforwardly. I was aware of how the backup process worked thanks, but no backup is useful without a restore capability, and that is what is missing information with regards to the ThingworxBackupStorage folder. Is there actually a process to restore from it?
No real examples, if you look at any Thing you can see it comes with a Purge for the history stored in a ValueStream
Please note you can only purge a single property at a time and I recommend using the immediate flag if you are doing purging of multiple properties. (Not using the flag would be an asynchronous action and you would not know when it finishes so you can go into the next purge)
So you could do this based on a Schedule, a Timer or some programmatic Counter that you maintain, even a DataChange event.
Hi Pai, a quick test of this didn't seem to decrease the overall (Neo4j) database size. Are the values just being 'nulled'? E.g. is the graph DB entity immutable? (I could understand this if it was an architectural decision based on speed reasons with Neo4J). I haven't tested it with the DataStax DB for property storage.
I purged all the properties of my value stream by looping through the properties. QueryPropertyHistory now returns only the data created now. However I noticed that neostore.propertystore.db and neostore.relationshipstore.db significantly. Also it created too many nioneo_logical.log.v* files which I deleted to free up the space issue. The question I have is why those *.db files grow after deleting the data. How do reduce the size. I tried restarting the tomcat couple of times. It did not help.
I would appreciate any suggestions.
The Neo4J database's size, which can be found in the ThingworxStorage/database/data directory, will clean up files that have met ThingWorx's default 7 day retention period for the logs. Please review this article for further information on how to tune your retention period.
I did follow the documentation and updated retention policy to one day last week. I believe this growth is related to purging of Value Stream. Instead of shrinking the data store it increased significantly. I am not sure how to compact the files now. My understanding is there are not much configuration/tools related neo4j exposed in Thingworx.