Community Tip - You can change your system assigned username to something more personal in your community settings. X
Hi,
We are using a ThingWorx Edge .NET microserver based on a modified version of the included steam sensor example.
We are noticing that it seems to sometimes crash randomly after running for a while, but have been having trouble determining the cause.
Below are the errors are in the Windows event viewer after it crashes. Are there any known issues with the .NET Edge server crashing? Let me know if there's any other info we can send that would be helpful.
Log Name: Application
Source: .NET Runtime
Date: 1/8/2016 3:23:48 AM
Event ID: 1026
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: hqqatest50
Description:
Application: TestMonitor.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: exception code c0000005, exception address 000007FEF8FFF73C
Stack:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name=".NET Runtime" />
<EventID Qualifiers="0">1026</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2016-01-08T08:23:48.000Z" />
<EventRecordID>1046621</EventRecordID>
<Channel>Application</Channel>
<Computer>hqqatest50</Computer>
<Security />
</System>
<EventData>
<Data>Application: TestMonitor.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: exception code c0000005, exception address 000007FEF8FFF73C
Stack:
</Data>
</EventData>
</Event>
Log Name: Application
Source: Application Error
Date: 1/8/2016 3:23:32 AM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: hqqatest50
Description:
Faulting application TestMonitor.exe, version 1.0.0.0, time stamp 0x568d8e13, faulting module twApi.dll, version 0.0.0.0, time stamp 0x551a50f6, exception code 0xc0000005, fault offset 0x000000000000f73c, process id 0x%9, application start time 0x%10.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2016-01-08T08:23:32.000Z" />
<EventRecordID>1046620</EventRecordID>
<Channel>Application</Channel>
<Computer>hqqatest50</Computer>
<Security />
</System>
<EventData>
<Data>TestMonitor.exe</Data>
<Data>1.0.0.0</Data>
<Data>568d8e13</Data>
<Data>twApi.dll</Data>
<Data>0.0.0.0</Data>
<Data>551a50f6</Data>
<Data>c0000005</Data>
<Data>000000000000f73c</Data>
</EventData>
</Event>
Message was edited by: Eric Freure Attached crash dump
From this link http://stackoverflow.com/questions/17168982/exception-error-c0000005-in-vc which you probably when to its a memory access issue.
Assuming its time based, are you running out of memory on the device?
What kind of device is the code running on?
What is its basic activity?
Can you share code snippet to provide insight to the changes you made.
It's running on a PC running Windows 2008. I don't think it's running out of memory, although the machines we're running it on don't have much, only 2 gigs of RAM. I don't see any other programs that are running on it having memory errors.
I don't know if I would say it is time based. It seems pretty random. We have it run on 14 different machines that are restarted each evening and it seems like there will usually be one or two instances of the Edge server that crash during the day.
Its basic activity is to communicate with a Python program that we run tests with, then update a Thingworx mashup with the test's status.
Here's a snippet with most of the changes that were made to the steam sensor example. I don't think it does anything too different from the steam sensor example:
TestMonitorThing.cs:
using com.thingworx.common;
using com.thingworx.common.exceptions;
using com.thingworx.communications.client;
using com.thingworx.communications.client.things;
using com.thingworx.metadata;
using com.thingworx.metadata.annotations;
using com.thingworx.metadata.collections;
using com.thingworx.types;
using com.thingworx.types.collections;
using com.thingworx.types.constants;
using com.thingworx.types.primitives;
using System;
using System.Collections.Specialized;
using System.Linq;
using System.Text;
using System.Threading;
using System.Globalization;
using Microsoft.Win32.TaskScheduler;
using System.Collections;
using log4net;
namespace TestMonitor
{
// Property Definitions
[ThingworxPropertyDefinition(name = "TestBaselineTask", description = "Baseline Task", baseType = "STRING", category = "Status", aspects = new string[] { "isReadOnly:true" })]
[ThingworxPropertyDefinition(name = "TestTask", description = "Test Task", baseType = "STRING", category = "Status", aspects = new string[] { "isReadOnly:true" })]
class TestMonitorThing : VirtualThing
{
private static readonly ILog logger = LogManager.GetLogger(typeof(TestMonitorThing));
private static String TEST_NAME = "TestName";
private static String TEST_STATUS = "TestStatus";
// Lock and bool field used to keep the shutdown logic from occuring multiple times before the actual shutdown
private object _shutdownLock = new object();
private bool _shuttingDown = false;
private TestMonitor.TestCommunicator _testCommunicator;
public TestMonitorThing(string name, string description, string identifier, ConnectedThingClient client, TestMonitor.TestCommunicator testCommunicator)
: base(name, description, identifier, client)
{
//Data shape definition that is used by the GetSteamSensorReadings service
FieldDefinitionCollection readingfields = new FieldDefinitionCollection();
readingfields.addFieldDefinition(new FieldDefinition(TEST_NAME, BaseTypes.STRING));
readingfields.addFieldDefinition(new FieldDefinition(TEST_STATUS, BaseTypes.STRING));
defineDataShapeDefinition("SteamSensorReadings", readingfields);
// Populate the thing shape with the properties, services, and events that are annotated in this code
base.initializeFromAnnotations();
_testCommunicator = testCommunicator;
}
// From the VirtualThing class
// This method will get called when a connect or reconnect happens
// Need to send the values when this happens
// This is more important for a solution that does not send its properties on a regular basis
public override void synchronizeState()
{
// Be sure to call the base class
base.synchronizeState();
// Send the property values to Thingworx when a synchronization is required
base.syncProperties();
}
// The processScanRequest is called by the SteamSensorClient every scan cycle
public override void processScanRequest()
{
// Be sure to call the base classes scan request
base.processScanRequest();
// Execute the code for this simulation every scan
this.scanDevice();
}
private Task getTestTask()
{
Task kickoffTask;
using (TaskService ts = new TaskService())
{
kickoffTask = ts.GetTask("Kickoff");
}
return kickoffTask;
}
private Task getTestBaselineTask()
{
Task kickoffBaselineTask;
using (TaskService ts = new TaskService())
{
kickoffBaselineTask = ts.GetTask("Kickoffbaseline");
}
return kickoffBaselineTask;
}
// Performs the logic for the steam sensor, occurs every scan cycle
public void scanDevice()
{
Random random = new Random();
Task testTask = getTestTask();
Task baselineTask = getTestBaselineTask();
string testTaskStatus = testTask.Name + " - " + testTask.State.ToString();
string testBaselineTaskStatus = baselineTask.Name + " - " + baselineTask.State.ToString();
// Set the property values
base.setProperty("TestTask", testTaskStatus);
base.setProperty("TestBaselineTask", testBaselineTaskStatus);
try
{
// Update the subscribed properties and events to send any updates to Thingworx
// Without calling these methods, the property and event updates will not be sent
// The numbers are timeouts in milliseconds.
base.updateSubscribedProperties(15000);
base.updateSubscribedEvents(60000);
}
catch (Exception ex)
{
// handle exception as appropriate
logger.Error("Error in scanDevice!");
}
}
[method: ThingworxServiceDefinition(name = "LaunchTest", description = "Launch test.")]
[return: ThingworxServiceResult(name = CommonPropertyNames.PROP_RESULT, description = "Result", baseType = "STRING")]
public string LaunchTest()
{
getTestTask().Run();
return "Success";
}
[method: ThingworxServiceDefinition(name = "StopTest", description = "Stop test.")]
[return: ThingworxServiceResult(name = CommonPropertyNames.PROP_RESULT, description = "Result", baseType = "STRING")]
public string StopTest()
{
getTestTask().Stop();
return "Success";
}
[method: ThingworxServiceDefinition(name = "LaunchBaselineTest", description = "Launch baseline test.")]
[return: ThingworxServiceResult(name = CommonPropertyNames.PROP_RESULT, description = "Result", baseType = "STRING")]
public string LaunchBaselineTest()
{
getTestBaselineTask().Run();
return "Success";
}
[method: ThingworxServiceDefinition(name = "GetTestState", description = "Get the state of tests that run on this machine.")]
[return: ThingworxServiceResult(name = CommonPropertyNames.PROP_RESULT, description = "Result", baseType = "INFOTABLE", aspects = new string[] { "dataShape:SteamSensorReadings" })]
public InfoTable GetTestState()
{
var table = new InfoTable(getDataShapeDefinition("SteamSensorReadings"));
try
{
OrderedDictionary testStatuses = _testCommunicator.GetTestStatuses();
foreach (DictionaryEntry de in testStatuses)
{
string currentTestName = (string)de.Key;
string currentTestStatus = (string)de.Value;
var entry = new ValueCollection();
entry.SetStringValue(TEST_NAME, currentTestName);
entry.SetStringValue(TEST_STATUS, currentTestStatus);
table.addRow(entry);
}
}
catch (Exception e)
{
// handle exception as appropriate
logger.Error("Error in GetTestState!");
}
return table;
}
[method: ThingworxServiceDefinition(name = "Shutdown", description = "Shutdown the client")]
[return: ThingworxServiceResult(name = CommonPropertyNames.PROP_RESULT, description = "", baseType = "NOTHING")]
public void Shutdown()
{
// Highly unlikely that this service could be called more than once, but guard against it anyway
lock (this._shutdownLock)
{
if (!this._shuttingDown)
{
// Start a thread to begin the shutdown or the shutdown could happen before the service returns
ThreadPool.QueueUserWorkItem(new WaitCallback(this.shutdownThread));
}
this._shuttingDown = true;
}
}
private void shutdownThread(object state)
{
try
{
// Delay for a period to verify that the Shutdown service will return
Thread.Sleep(1000);
// Shutdown the client
this.getClient().shutdown();
}
catch
{
logger.Error("Error in shutdownThread!");
// Not much can be done if there is an exception here
// In the case of production code should at least log the error
}
}
}
}
Eric,
Could the Python be causing an issue? I have run Python on my PI's and sometimes they stop. How are you communicating with the Python ?
I was leaning away from the Python communication causing the crashing since the exception specifically mentions "faulting module twApi.dll", but I guess I it's still possible that the communication with Python could be causing the problem.
The Python process communicates with the .NET Edge server using zeromq. Python uses the pyzmq library and the .NET Edge server uses NETMQ. A thread on the .NET Edge server runs that listens for updates from the Python process through ZeroMQ. When an update comes in, a single string variable is updated. Another thread that runs the .NET Edge server sensor updates calls a method which parses the string and turns it in to an InfoTable that gets sent to the Thingworx mashup.
It's looking like it may have been a thread safety issue. I put in some locks where a shared variable was being used between threads in code that communicated with the Python program and we haven't seen the crash again yet.
Oops, spoke too soon, it crashed again on one of our machines last night with similar errors in the event log (see below). So, still not sure what could be causing this crash.
Log Name: Application
Source: Application Error
Date: 1/20/2016 1:45:52 AM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: hqqatest9
Description:
Faulting application TestMonitor.exe, version 1.0.0.0, time stamp 0x5696c6b9, faulting module twApi.dll, version 0.0.0.0, time stamp 0x551a50f6, exception code 0xc0000005, fault offset 0x000000000000f73c, process id 0x%9, application start time 0x%10.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2016-01-20T06:45:52.000Z" />
<EventRecordID>501767</EventRecordID>
<Channel>Application</Channel>
<Computer>hqqatest9</Computer>
<Security />
</System>
<EventData>
<Data>TestMonitor.exe</Data>
<Data>1.0.0.0</Data>
<Data>5696c6b9</Data>
<Data>twApi.dll</Data>
<Data>0.0.0.0</Data>
<Data>551a50f6</Data>
<Data>c0000005</Data>
<Data>000000000000f73c</Data>
</EventData>
</Event>
Log Name: Application
Source: .NET Runtime
Date: 1/20/2016 1:46:08 AM
Event ID: 1026
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: hqqatest9
Description:
Application: TestMonitor.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: exception code c0000005, exception address 000007FEF8B6F73C
Stack:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name=".NET Runtime" />
<EventID Qualifiers="0">1026</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2016-01-20T06:46:08.000Z" />
<EventRecordID>501768</EventRecordID>
<Channel>Application</Channel>
<Computer>hqqatest9</Computer>
<Security />
</System>
<EventData>
<Data>Application: TestMonitor.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: exception code c0000005, exception address 000007FEF8B6F73C
Stack:
</Data>
</EventData>
</Event>
I found that a crash dump was generated. I've attached it to the first post in case anyone is interested in looking at it.
I've managed to open up the crash dumps in Visual Studio, and I'm seeing that it seems to have crashed on line 77 of the C Edge SDK file src\messaging\twMessaging.c:
if (msgHandlerSingleton && msgHandlerSingleton->incomingRequestCallbacks) {
I'm not sure, but it looks like the msgHandlerSingleton variable may be invalid when it is referenced on that line.