ThingWorx auto restarts due to Out of Memory when a script is run

[AP_10132067]

Hi,

When we run the below script, we are facing an out of memory issue and Thingworx tomcat service restarts automatically. we have run the below script manually from composer with execute option in a batch of 25 & 50 vin's in input array, the scripts executes and prints the final array in the script logs but internally it is causing out of memory issue, Can any one please let us know what is the issue in this script and why this is creating out of memory issue becuse we do not see any errors in the logs also.

Script:

serviceName = "updateDeviceNum";
//sample input format :{ "vin": ["123456786", "456782354"]}

var inputArrayVin = inputVIN.vin;
//logger.info(myName + " inputVIN " + inputVIN);
var inputLen = inputArrayVin.length;
logger.info(myName + " inputLen " + inputLen);
var arValueUpdated = [];
var arSkipped = [];

for (var i = 0; i < inputLen; i++) {
//calling the getAsset using VIN to get deviceID
var getAssetByVINURL = "https://**********" + inputArrayVin[i];
var apiParams = {
headers: ******,
url: getAssetByVINURL,
ignoreSSLErrors: true
};

var resultFromGetAsset = Resources["ContentLoaderFunctions"].GetJSON(apiParams);
logger.debug(myName + " resultFromGetAsset response is: " + JSON.stringify(resultFromGetAsset));
//Getting deviceID from asset response
if (resultFromGetAsset.array) {
var deviceID = resultFromGetAsset.array[0].deviceId;
logger.info(myName + " deviceID: " + deviceID);
//calling getDeviceAPI to get DeviceNo
var params = {
deviceID: deviceID
};
var resultFromGetDevice = Things["TestThing"].GetDevice(params);
logger.debug(myName + " resultFromGetDevice response is: " + JSON.stringify(resultFromGetDevice));
if(resultFromGetDevice){
var serialNo = resultFromGetDevice.sn;
var thingName = "Telematic_" + deviceID;
if (Things[thingName] && Things[thingName].PCMSerialNo == "undefined") {
Things[thingName].PCMSerialNo = serialNo;
//Things[thingName].PCMSerialNo = '5678943';
updatedDevices = {
VIN: inputArrayVin[i],
deviceID: deviceID,
sn: serialNo
};
arValueUpdated.push(updatedDevices);
} else {
updatedDevices = {
VIN: inputArrayVin[i],
deviceID: deviceID,
sn: serialNo
};
arSkipped.push(updatedDevices);
}

}
else {
updatedDevices = {
VIN: inputArrayVin[i],
deviceID: deviceID
};
arSkipped.push(updatedDevices);
}

}
else {
updatedDevices = {
VIN: inputArrayVin[i]
};
arSkipped.push(updatedDevices);
}

}

logger.info(myName + " arValueUpdated " + JSON.stringify(arValueUpdated));
logger.info(myName + " arSkipped " + JSON.stringify(arSkipped));

[PaiChung]

How much of a memory spike are you getting?

Besides whatever payload you are retrieving from the device, none of these look like they would be very large nor take up much memory.

[AP_10132067]

Hi @PaiChung ,

Thank you for the quick response. 

We have allocated 106 GB of memory for tomcat where ThingWorx is installed on Linux VM. Out of 106 GB, free memory was approximately 10 GB and we tried to run this script three times with a batch of 50 values in InputArray for first time, 25 in second & third batch. So, all three times the TWX tomcat got auto restarted with in 10 or 15 min after the execution completion of this script and that time free memory was ~500 MB. we captured the finalArray from logs and we have not seen any errors during that time. But still we observed all the three times when this script is executed, it auto restarted ThingWorx tomcat and also saw an out of Memory/ zero swap space error in the Linux VM Kernel logs.

We are not retrieving payload from Device but calling two API's to fetch data from DB and its a simple json payload which has only 15 key-value pair in it.

 

Also, when we tested this script in ThingWorx 8.5 where the number of connected devices and data flow is very less, we have not seen these issues with a batch of 500 values in inputArray but when we executed in ThingWorx 8.1 where the average connected devices are >3000 and with high data flow, tomcat got auto restarted all the times only when this script is run.

We would like to know if there is anything wrong in the way script is implemented?

[PaiChung]

Are you sure it is the memory that is used up, aside from threads being used up.

From what you are describing it does seem to grab a lot of memory at once.

Do you know exactly what you are retrieving from the device, and how much data it is?

That would be the information going into memory.

500MB left before crash probably has to do with how the JVM is set, to only use up to ~20% of max.