Hello,
KEPServerEX 6.11 V6.11.718.0
On 3/28 around 10:44pm a Dell Windows server experienced some communication failures with the Kepware OPC server software installed on said server. This ultimately appears to have resulted in a possible memory leak on the Dell Windows server or something else consuming all of the virtual memory causing Windows services and processes to crash and become unrecoverable by automated methods. Manually restarting the services restored operational stability and there has been no reoccurrences since.
I am trying to investigate the event logs and identify the root cause of this comms fault. Leading up to the Windows server communication log events, the Kepware server has log events for R/W requests being stopped, which appears to have resulted in the "badnocommunication" errors on the Windows server which preceded the out of system memory error events. The R/W stop events in Kepware appear to be the initial trigger leading up to the server spiraling out of control..
I found this knowledgebase article (https://www.ptc.com/en/support/article/CS337204) that describes the only possible cause of these specific Kepware log events to be some type of manual human intervention committing changes or "edits". However I am being told there was no one making any changes or online edits that would have caused these events at 10:40pm.
So my question is very simple, is it at all possible that these events could be produced by anything other than a human manually making edits. For example, could network congestion or the server hardware being over utilized cause the R/W requests to be stopped? The event logs are posted below. As you can see there were plenty of log events for online edits around 11am but that was 12 hours prior to the failure event timeframe and during the failure there are no logs indicating any edits were being made at the time of failure. There is a full 3min of time between the R/W being stopped, update to the logical addresses being started and the update being completed. Typically in the past this logical address update process takes a few seconds, we have plenty of history of these events going back 12 months. So what else could be the cause?
Date Time Level Source Event
3/26/2024 7:21:03 PM Warning Allen-Bradley ControlLogix Ethernet CP2.CP2 | Project online edit detected. Currently using Symbolic addressing.
3/26/2024 7:21:19 PM Warning Allen-Bradley ControlLogix Ethernet CP2.CP2 | Project online edit detected. Currently using Symbolic addressing.
3/26/2024 7:21:20 PM Warning Allen-Bradley ControlLogix Ethernet CP2.CP2 | Project online edit detected. Currently using Symbolic addressing.
3/28/2024 11:11:16 AM Warning Allen-Bradley ControlLogix Ethernet MLC.MLC | Project online edit detected. Currently using Symbolic addressing.
3/28/2024 11:11:16 AM Warning Allen-Bradley ControlLogix Ethernet MLC.MLC | Project online edit detected. Currently using Symbolic addressing.
3/28/2024 11:11:41 AM Warning Allen-Bradley ControlLogix Ethernet MLC.MLC | Project online edit detected. Currently using Symbolic addressing.
3/28/2024 11:11:50 AM Warning Allen-Bradley ControlLogix Ethernet MLC.MLC | Project online edit detected. Currently using Symbolic addressing.
3/28/2024 11:11:51 AM Warning Allen-Bradley ControlLogix Ethernet MLC.MLC | Project online edit detected. Currently using Symbolic addressing.
3/28/2024 11:11:51 AM Warning Allen-Bradley ControlLogix Ethernet MLC.MLC | Project online edit detected. Currently using Symbolic addressing.
3/28/2024 11:12:21 AM Warning Allen-Bradley ControlLogix Ethernet MLC.MLC | Read/Write requests to device stopped. Updating Logical Addresses from device project.
3/28/2024 11:12:29 AM Warning Allen-Bradley ControlLogix Ethernet MLC.MLC | Read/Write requests to device resumed. Updating Logical Addresses from device complete. Currently using Logical addressing.
3/28/2024 10:39:46 PM Warning Allen-Bradley ControlLogix Ethernet DCS.DCS | Read/Write requests to device stopped. Updating Logical Addresses from device project.
3/28/2024 10:40:06 PM Warning Allen-Bradley ControlLogix Ethernet MLC.MLC | Read/Write requests to device stopped. Updating Logical Addresses from device project.
3/28/2024 10:40:23 PM Warning Allen-Bradley ControlLogix Ethernet CP2.CP2 | Read/Write requests to device stopped. Updating Logical Addresses from device project.
3/28/2024 10:42:47 PM Warning Allen-Bradley ControlLogix Ethernet DCS.DCS | Read/Write requests to device resumed. Updating Logical Addresses from device complete. Currently using Logical addressing.
3/28/2024 10:43:00 PM Warning Allen-Bradley ControlLogix Ethernet MLC.MLC | Read/Write requests to device resumed. Updating Logical Addresses from device complete. Currently using Logical addressing.
3/28/2024 10:43:09 PM Warning Allen-Bradley ControlLogix Ethernet CP2.CP2 | Read/Write requests to device resumed. Updating Logical Addresses from device complete. Currently using Logical addressing.
3/28/2024 10:57:31 PM Warning Allen-Bradley ControlLogix Ethernet MLC.MLC | Read/Write requests to device stopped. Updating Logical Addresses from device project.
3/28/2024 10:57:43 PM Warning Allen-Bradley ControlLogix Ethernet MLC.MLC | Read/Write requests to device resumed. Updating Logical Addresses from device complete. Currently using Logical addressing.
